Image processing method and related device

Abstract

The invention provides an image processing method and a related device. The image processing method comprises the following steps: acquiring a first image; determining blur information indicating a blur region in the first image; and using the first neural network to perform deblurring processing on the first image based on the first image and the blurring information of the first image to obtain a second image. In the method, a first neural network is trained to deblurry an image. Specifically, the first neural network is based on an image and blurring information for indicating a blurring region of the image during training. Therefore, compared with a neural network obtained only based on image training, when the first neural network is used for deblurring the image, the definition of the deblurred image can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of autonomous driving, and in particular to an image processing method and related devices. Background Art

[0002] In the field of autonomous driving or video surveillance, image quality is very important. However, in real life, the captured images are usually blurred due to some factors, such as camera shake or rapid movement of the target object. Therefore, the image needs to be deblurred.

[0003] At present, the method for deblurring an image is as follows: construct a training sample, which includes a blurred image and a label, and the label indicates a clear image corresponding to the blurred image; train a neural network based on the training sample to obtain a trained neural network; input a new blurred image into the trained neural network to obtain an image after deblurring the new blurred image.

[0004] However, the clarity of the image obtained by the above deblurring cannot meet the clarity requirements in many scenarios. Summary of the invention

[0005] The present application provides an image processing method and related devices, in order to improve the clarity of a deblurred image.

[0006] In a first aspect, the present application provides an image processing method, comprising: acquiring a first image; determining blur information of the first image, the blur information indicating a blurry area in the first image; using a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, the first neural network being used to deblur the image based on the image and the blur information of the image.

[0007] The blur information of the first image may be considered as some information that can indicate a blurry area in the first image.

[0008] It should be noted that this embodiment does not limit the method of determining the blur information of the first image.

[0009] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the image-level blur level sorting between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network is an image, and then after the first image is input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0010] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the pixel-level blur level sorting between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network is an image, and then after the first image is input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0011] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the image-level blur level sorting and the pixel-level blur level sorting between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network is an image, and then after the first image is input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0012] In this embodiment, after the blur information of the first image is determined, the first neural network is used to perform deblurring processing on the first image based on the first image and the blur information of the first image, wherein the first neural network is trained for image deblurring. In other words, in this embodiment, after the first image and the blur information of the first image are input into the first neural network, the image output by the first neural network is used as the image after the first image is deblurred.

[0013] Specifically, the first neural network in the present application is trained based on the image and the blur information of the acquired image, rather than trained based only on the image, wherein the blur information of the image can be considered as some information that can indicate the blurry area in the image. Therefore, compared with the neural network trained based only on the image, when the first neural network is used to deblur the image, it helps to improve the clarity of the deblurred image.

[0014] In combination with the first aspect, in a possible implementation, the method also includes: obtaining first event data generated by an event camera within a first time length, the first time length being an exposure time length used by a first camera for capturing a first image; determining blur information of the first image, including: determining the blur information of the first image based on the first image and the first event data.

[0015] In this embodiment, when acquiring the blur information of the first image, since the event data generated by the event camera, namely the first event data, is also referred to, it helps to improve the accuracy of the blur information of the acquired first image, thereby helping to further improve the deblurring effect.

[0016] In one implementation, determining the fuzzy information of the first image based on the first image and the first event data includes: inputting the first image and the first event data into a fuzzy feature extractor and determining the features output by the fuzzy feature extractor as the fuzzy information of the first image;

[0017] Among them, the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data. The first training data includes a pair of images with different blur levels, event data corresponding to the pair of images with different blur levels, and labels. The event data corresponding to the pair of images with different blur levels are the same, and the label is used to indicate the image-level blur level sorting and / or pixel-level blur level sorting between a pair of images with different blur levels.

[0018] For example, in one implementation, multiple training samples are constructed, each of which includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the pair of images with different blur levels, and a label. The event data corresponding to the pair of images with different blur levels are the same, and the label indicates the image-level blur ranking between the pair of images with different blur levels; a neural network is trained using multiple training samples, the input of the neural network is the image and the event data, and then after the first image and the first event data are input to the trained neural network, the feature output by a convolutional layer in the trained neural network is determined as the blur information of the first image.

[0019] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the pair of images with different blur levels, and labels, wherein the event data corresponding to the pair of images with different blur levels are the same, and the label indicates the pixel-level blur level sorting between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network being the image and the event data, and then after the first image and the first event data are input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0020] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also referred to as a sorted sample pair), event data and labels corresponding to the pair of images with different blur levels, wherein the event data corresponding to the pair of images with different blur levels are the same, and the label indicates the image-level blur level sorting and the pixel-level blur level sorting between the pair of images with different blur levels, and a neural network is trained using the plurality of training samples, the input of the neural network being the image and the event data, and then after the first image and the first event data are input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0021] Optionally, a pair of images with different blur levels have different exposure times, or a pair of images with different blur levels have different resolutions.

[0022] In combination with the first aspect, in a possible implementation, the method further includes: obtaining j images of different resolutions based on the first image, the j images of different resolutions including a first image of a first resolution and j-1 images of different resolutions after reducing the first image; using a fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images of different resolutions and first event data; wherein, using a first neural network to deblur the first image based on the first image and the fuzzy information of the first image to obtain a second image includes: using the first neural network to deblur the first image based on the first image, the fuzzy information of the first image, j-1 images of different resolutions, and the fuzzy information of each of the j-1 images of different resolutions to obtain a second image.

[0023] Optionally, j is 3, and j-1 images with different resolutions include an image obtained by downsampling the first image by a factor of 2 and an image obtained by further downsampling the image obtained by downsampling the first image by a factor of 2 by a factor of 2.

[0024] Specifically, in the method of obtaining the first neural network, second training data is obtained, the second training data including a third image of a first resolution and blur information of the third image, j-1 images of different resolutions after reducing the third image and their respective blur information, the third image and respective labels of the j-1 images of different resolutions after reducing the third image, the label of each image is used to indicate the image of each image after deblurring; the neural network is trained based on the second training data to obtain the first neural network.

[0025] With reference to the first aspect, in a possible implementation manner, the first camera is an RGB camera.

[0026] In combination with the first aspect, in a possible implementation, the first neural network is a convolutional neural network.

[0027] In a second aspect, the present application provides an image processing device, including: an acquisition module, used to acquire a first image; a processing module, used to determine blur information of the first image, the blur information indicating a blurred area in the first image; the processing module, also used to use a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, the first neural network being used to deblur the image based on the image and the blur information of the image.

[0028] In combination with the second aspect, in a possible implementation method, the acquisition module is also used to: acquire first event data generated by the event camera within a first time length, the first time length being the exposure time length used by the first camera for capturing the first image; the processing module is also used to: determine the blur information of the first image based on the first image and the first event data.

[0029] In combination with the second aspect, in a possible implementation, when the processing module determines the blur information of the first image based on the first image and the first event data, the processing module is specifically used to: input the first image and the first event data into a fuzzy feature extractor, and then determine the features output by the fuzzy feature extractor as the blur information of the first image; wherein the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on first training data, the first training data includes a pair of images with different blur degrees, event data corresponding to a pair of images with different blur degrees, and labels, the event data corresponding to a pair of images with different blur degrees are the same, and the label is used to indicate the image-level blur degree ranking and / or pixel-level blur degree ranking between a pair of images with different blur degrees.

[0030] In conjunction with the second aspect, in a possible implementation manner, a pair of images with different blur levels have different exposure times, or a pair of images with different blur levels have different resolutions.

[0031] In combination with the second aspect, in a possible implementation manner, the processing module is further used to: obtain j images with different resolutions based on the first image, where the j images with different resolutions include the first image with the first resolution and j-1 images with different resolutions that are obtained by reducing the first image;

[0032] A fuzzy feature extractor is used to obtain fuzzy information of each image based on each image in j-1 images with different resolutions and the first event data; and a first neural network is used to deblur the first image based on the first image, the fuzzy information of the first image, the j-1 images with different resolutions, and the fuzzy information of each of the j-1 images with different resolutions to obtain a second image.

[0033] Optionally, j is 3, and the j-1 images with different resolutions include an image obtained by downsampling the first image by 2 times, an image obtained by further downsampling the image obtained by downsampling the first image by 2 times, and an image obtained by further downsampling the image obtained by downsampling the first image by 2 times.

[0034] In combination with the second aspect, in a possible implementation, the acquisition module is also used to: acquire second training data, the second training data including a third image of a first resolution and blur information of the third image, j-1 images of different resolutions after reducing the third image and their respective blur information, the third image and respective labels of j-1 images of different resolutions after reducing the third image, the label of each image is used to indicate the image of each image after deblurring; the processing module is also used to train the neural network based on the second training data to obtain the first neural network.

[0035] In conjunction with the second aspect, in a possible implementation manner, the first camera is an RGB camera.

[0036] In combination with the second aspect, in a possible implementation, the first neural network is a convolutional neural network.

[0037] In a fourth aspect, the present application provides an automatic driving device for executing the image processing method described in the first aspect or any one of the implementation methods thereof.

[0038] In a fifth aspect, the present application provides an autonomous driving system, comprising an image processing device as described in the second aspect.

[0039] In a sixth aspect, an image processing device includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions in the memory to execute the method described in the first aspect.

[0040] In a seventh aspect, the present application provides a computer-readable medium, wherein the computer-readable medium stores a program code for computer execution, wherein the program code includes instructions for executing the method described in the first aspect or any possible implementation method thereof.

[0041] In an eighth aspect, the present application provides a computer program product, wherein the computer program product includes a computer program code, and when the computer program code runs on a computer, the computer implements the method described in the first aspect or any possible implementation method thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A structural schematic diagram of an image deblurring system provided in this application;

[0043] Figure 2A schematic diagram of a process of an image deblurring method provided by an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a neural network training based on sorted sample pairs provided in this application;

[0045] Figure 4 A schematic diagram of obtaining a first neural network provided in the present application;

[0046] Figure 5 A structural schematic diagram of another image deblurring system provided in this application;

[0047] Figure 6 A schematic diagram of a process of an image deblurring method provided by another embodiment of the present application;

[0048] Figure 7 A schematic diagram of a neural network training based on event data provided in this application;

[0049] Figure 8 A structural schematic diagram of an image deblurring method provided by another embodiment of the present application;

[0050] Fig. 9 A structural schematic diagram of an image deblurring device provided by an embodiment of the present application;

[0051] Fig.10 A structural schematic diagram of an image deblurring device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0053] For ease of understanding, the relevant terms involved in this application are first explained.

[0054] 1. Event Camera

[0055] Event camera, sometimes also called dynamic vision sensor (DVS), is a new type of camera that simulates the working mechanism of the human brain. It has the advantages of high dynamic range, low latency, high frame rate and low power consumption. It can capture high-precision time and space information of fast-moving targets. Each pixel of the camera can independently sense the change of light brightness. When the change exceeds the threshold, a specially designed circuit outputs an event. The working state of each pixel is independent of each other and can be output asynchronously. Therefore, it has an extremely high response speed (microsecond level), which can effectively improve the effects of tasks such as motion perception and image enhancement.

[0056] 2. Neural Networks

[0057] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0058] The key technology of artificial intelligence today is neural networks (NN). Neural networks simulate the connections between nerve cells in the human brain and widely interconnect a large number of simple processing units (called neurons) to form a complex network system.

[0059] A simple neural network consists of three layers: input layer, output layer and hidden layer (also called middle layer). The connection between each layer corresponds to a weight (its value is called weight, parameter). The reason why neural networks can perform well in computer vision, natural language processing and other fields is that the weights are adjusted through training algorithms to make the prediction results of the neural network optimal.

[0060] The training of neural networks generally includes two calculation steps, the first step is forward calculation, and the second step is reverse calculation. Among them, the forward calculation is: after the input value and the parameter are calculated, they are passed through a nonlinear function to generate the output value. The output value is either the final output of the network or the subsequent input value to continue to perform similar calculations. The deviation between the output value of the network and the actual label value of the corresponding sample is measured by the model loss function. The loss function is expressed as a function f(x,w) of the input sample x and the network parameter w. In order to minimize the loss function, it is necessary to continuously adjust the network parameter w, and the reverse calculation is to obtain the updated value of the parameter w. In the algorithm based on gradient descent, the reverse calculation starts from the last layer of the neural network, calculates the partial derivative of the loss function to each layer of parameters, and finally obtains the partial derivative of all parameters, which is called the gradient. In each iteration, the parameter w is updated in the opposite direction of the gradient with a certain step size η to obtain a new parameter w, that is, one step of training is completed.

[0061] The process of training a neural network is the process of learning the weights corresponding to neurons. Its ultimate goal is to obtain the weights corresponding to each layer of neurons in the trained neural network model.

[0062] 3. Exposure duration

[0063] Exposure time refers to the time interval from the opening to the closing of the shutter, during which the target object can leave an image on the film.

[0064] Image quality is very important in the fields of autonomous driving, video surveillance or sports photography. For example, in an autonomous driving scenario, an autonomous vehicle needs to make further decisions on autonomous driving behavior based on the collected images during driving, such as whether to turn left, right or go straight. It should be understood that in this scenario, the quality of the image is a key factor affecting the decision-making of the autonomous vehicle, thereby affecting the safety of the autonomous vehicle during driving.

[0065] However, in real-world conditions, the captured images are usually blurred due to some factors. For example, when the target object moves quickly or the camera shakes, the captured image will be blurred. Therefore, how to deblur the image has become an urgent problem to be solved.

[0066] At present, a method for deblurring an image is as follows: construct a training sample, the training sample includes a blurred image and a label, the label indicates the clear image corresponding to the blurred image; train a neural network based on the training sample to obtain a trained neural network; input a new blurred image into the trained neural network to obtain an image after deblurring the new blurred image. In other words, in the deblurring method, the mapping relationship between the blurred image and the clear image is directly learned through the neural network, and then the blurred image to be deblurred is input into the learned neural network to obtain the deblurred image.

[0067] However, it is found through analysis that in the above deblurring methods, the image quality after deblurring is still not good enough.

[0068] In view of this, the present application provides an image deblurring method to improve the quality of the deblurred image.

[0069] Figure 1 This is a structural diagram of the image deblurring system provided in this application. Figure 1 As shown, in the deblurring system, a first camera 101 and a processing module 102 are included. The first camera 101 is used to collect images that can describe external environment information, and then input the image into the processing module 102. The processing module 102 is used to receive the image sent by the first camera 101 and use a preset processing algorithm to perform deblurring processing based on the image to obtain a deblurred image, thereby improving the quality of the video.

[0070] It should be noted that the embodiment of the present application does not limit the specific form of the first camera 101. For example, it can be a fisheye camera, an eagle eye camera, or a monocular or binocular camera, which does not constitute a limitation of the present application.

[0071] It is also noted here that this embodiment does not limit the specific scenario in which the image deblurring system is applied.

[0072] For example, the image deblurring system can be applied to the field of sports photography: when athletes are moving at high speed, the image will be blurred. Through the method of the present application, a deblurred image is provided.

[0073] For example, the image deblurring system can be applied to autonomous driving scenarios: when an autonomous driving vehicle is driving and in a dark environment, the image quality of the autonomous driving vehicle decreases, and the image is easily blurred, which affects the perception and decision-making of the autonomous driving. The deblurring method provided in this application provides a deblurred image for the autonomous driving vehicle.

[0074] For example, the image deblurring system can be applied to video surveillance scenarios: in a dark environment, the exposure time of the camera increases, and moving objects in the surveillance images and videos are prone to local blur, affecting the recognition of pedestrians and vehicles. Through the deblurring method provided by this application, a deblurred image is obtained.

[0075] Of course, the image deblurring system can also be applied to other fields that perform other processing based on images.

[0076] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0077] Figure 2 This is a schematic diagram of the process of an image deblurring method provided by the present application. The method of this embodiment may include S201, S202 and S203. The image processing method may be Figure 1 The processing modules shown are executed.

[0078] S201, acquiring a first image.

[0079] In this embodiment, acquiring the first image specifically refers to acquiring an image captured by the first camera within the exposure time for capturing information that can describe the external environment. As an example, the first camera is an RGB camera.

[0080] It should be noted that this embodiment does not limit the specific value of the exposure time of the first camera.

[0081] S202: Determine blur information of the first image, where the blur information indicates a blurry area in the first image.

[0082] In this embodiment, after the first image is acquired, blur information of the first image is first obtained. The blur information specifically indicates a blurry area in the first image. In other words, in this embodiment, the blur information of the image refers to some information that can describe the blurry area of ​​the image.

[0083] Several implementations for obtaining blur information indicating a blur region in the first image are described below.

[0084] In the first implementation, it includes:

[0085] 1) Construct multiple training samples.

[0086] Specifically, each of the multiple training samples includes a pair of images with different blur levels and a label, wherein the label indicates an image-level blur ranking between the pair of images with different blur levels.

[0087] In this application, a pair of images with different blur levels included in a training sample is also referred to as a sorted sample pair.

[0088] For example, a pair of sorted samples is called image 1 and image 2; the label value is a scalar, for example, when it is 1, it indicates that the blurriness of image 1 is greater than that of image 2, and when it is 0, it indicates that the blurriness of image 1 is less than or equal to that of image 2. Or the label is a two-dimensional data, for example, when it is 01, it indicates that the blurriness of image 1 is greater than that of image 2, and when it is 10, it indicates that the blurriness of image 1 is less than or equal to that of image 2.

[0089] 2) Train the neural network through multiple training samples.

[0090] For example, the neural network includes a feature extractor and a global sorter. The feature extractor is used to receive an image, extract image features and output the image features to the global sorter. The global sorter is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the image-level fuzzy degree ranking between the sorted sample pairs.

[0091] It should be understood that in this manner, when training the neural network, what the neural network learns is the blur degree at the image level, that is, the neural network can be considered as a neural network for learning the blur degree at the image level of an image.

[0092] 3) The feature extractor is determined as a fuzzy feature extractor for extracting fuzzy information of the image.

[0093] At this time, the blur information of the first image can be obtained by inputting the first image collected in S201 into the feature extractor.

[0094] The second implementation includes:

[0095] 1) Construct multiple training samples.

[0096] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the blur level sorting between the pair of images with different blur levels. For example, for two sorted sample pairs with a resolution of 8*8, the label is 64 values, each value indicating the blur level sorting between the same pixel points.

[0097] 2) Train the neural network through multiple training samples.

[0098] For example, the neural network includes a feature extractor and a local sorter, the feature extractor is used to receive an image, extract image features and output the image features to the local sorter, the local sorter is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the pixel-level blur degree ranking between the sorted sample pairs.

[0099] It should be understood that in this way, when training the neural network, the neural network learns the blur degree at the pixel level, that is, the neural network can be considered as a neural network for learning the blur degree at the pixel level of an image.

[0100] 3) The feature extractor is determined as a fuzzy feature extractor for extracting fuzzy information of the image.

[0101] At this time, the first image collected in S201 is input into the feature extractor to obtain blur information of the first image.

[0102] The third implementation is a combination of the first and second implementations. The implementation includes:

[0103] 1) Construct multiple training samples.

[0104] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the image-level blur ranking and the pixel-level blur ranking between the pair of images with different blur levels.

[0105] 2) Train the neural network through multiple training samples.

[0106] For example, Figure 3 As shown, the neural network includes a feature extractor, a global sorter and a local sorter. The feature extractor is used to receive images, extract image features and output the image features to the global sorter and the local sorter. The global sorter is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the image-level blur ranking between the sorted sample pairs. The local sorter is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the pixel-level blur ranking between the sorted sample pairs.

[0107] It should be understood that in this way, when training the neural network, the neural network learns both the blur degree at the image level and the blur degree at the pixel level, that is, the neural network can be considered as a neural network for learning the blur degree at the image level and the blur degree at the pixel level.

[0108] 3) The feature extractor is determined as a fuzzy feature extractor for extracting fuzzy information of the image.

[0109] At this time, the first image collected in S201 is input into the feature extractor to obtain blur information of the first image.

[0110] In the present application, the blur information of an image is also referred to as blur degree information of the image, for example.

[0111] S203, using a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, wherein the first neural network is used to deblur the image based on the image and the blur information of the image.

[0112] Specifically, in this embodiment, the first neural network is a trained neural network for obtaining a deblurred image based on an image and blur information of the image, wherein the blur information of the image is obtained based on the collected image.

[0113] As an example, combining Figure 4 , illustrating a way to obtain the first neural network.

[0114] 1) Construct multiple training samples.

[0115] Specifically, each of the multiple training samples includes a blurred image and a label, wherein the label indicates a clear image corresponding to the blurred image.

[0116] 2) Build a neural network model and train it through multiple training samples.

[0117] In this embodiment, Figure 4 As shown, when building a neural network model, the neural network includes an input layer, multiple feature extractors and an output layer, wherein the input layer is used to receive an input image, and the output layer is used to output a deblurred image based on the received features.

[0118] Specifically, for at least one of the multiple feature extractors, its input includes not only the image feature data input by the previous layer feature extractor, but also the fuzzy information obtained after the image input to the neural network model passes through the fuzzy feature extractor. That is, the fuzzy information indicating the fuzzy area of ​​the input image is embedded into the image feature.

[0119] For example, for Figure 4In the neural network shown, the input of the second feature extractor includes fuzzy information. At this time, when training the neural network, after the image is input into the neural network through the input layer, the neural network first extracts features through the first feature extractor to obtain feature data 1, and then inputs feature data 1 and fuzzy information into the second feature extractor to obtain further extracted features, and the further extracted features are input into the subsequent model until the output image is obtained, and then the output image is compared with the label of the input image to optimize the parameters of the neural network.

[0120] It should be noted that this application is for Figure 4 The specific model in the neural network shown is not limited. For example, the first feature extractor may include at least one convolution layer, or the fuzzy information may also be introduced into the first feature extractor, or may also be introduced into the last feature extractor, without constituting a limitation.

[0121] After completing the training through multiple training samples, a trained neural network, i.e., the first neural network, is obtained.

[0122] It should be understood that compared to a neural network trained only by images, since there is fuzzy information in the training process, that is, the neural network can also learn the image based on the fuzzy information of the image, when the trained first neural network is used to deblur the image to be tested, the deblurring effect is better than when the neural network trained only by images is used to deblur the image to be tested, that is, the quality of the deblurred image can be improved.

[0123] Furthermore, in this embodiment, when the first image is to be deblurred, after obtaining blur information of the first image, the first image and the blur information of the first image are input into the first neural network to obtain a second image after deblurring the first image.

[0124] It should be noted that the input of the first neural network described in this embodiment is different from the input of the input layer of the first neural network. Figure 4 In the example shown, the input layer in the neural network is only used to receive the input image, but not the fuzzy information, but the input of the first neural network includes both the image and the fuzzy information of the image. In other words, the input of the first neural network regards the first neural network as a whole.

[0125] Since event cameras can capture changes in brightness, have the characteristics of high temporal resolution, high dynamic range, low data redundancy and low power, and can perceive changes in image grayscale values, the present application also provides another image deblurring method.

[0126] In this method, when deblurring an image, the event data obtained by the event camera is also integrated to further improve the quality of the deblurred image. That is, in other words, when deblurring an image, it is based on data captured by two cameras, one is the data captured by the first camera, and the other is the data captured by the event camera. Or in other words, when deblurring an image, deblurring is performed together based on the data captured by these two cameras. Among them, the event camera and the first camera are two independent cameras.

[0127] Before introducing the method of image deblurring by fusing event data obtained by event cameras, we first combine Figure 5 , introduces a structural schematic diagram of another image deblurring system provided by the present application. Figure 5 As shown, the image deblurring system includes a first camera 501 , an event camera 502 and a processing module 503 .

[0128] Among them, the first camera 501 is used to collect images that can describe external environmental information, and the event camera 502 is used to obtain event data corresponding to the images collected by the first camera 501. Among them, the event data corresponding to the images collected by the first camera 501 specifically refers to the event data collected by the event camera within the exposure time of the image taken by the first camera.

[0129] The processing module 503 is used to perform deblurring processing on the image based on the image captured by the first camera and the event data captured by the event camera using a preset processing algorithm to obtain a deblurred image, thereby improving the deblurring effect.

[0130] It is also noted here that this embodiment does not limit the specific scenario in which the image deblurring system is applied.

[0131] For example, the image deblurring system can be applied to sports photography scenes, autonomous driving scenes, or video surveillance scenes. Of course, the image deblurring system can also be applied to other fields based on image processing.

[0132] Specifically, when the event data obtained by the event camera is fused to further improve the deblurring effect, blur information of the image is obtained based on the image and the event data corresponding to the image.

[0133] Combination Figure 6 , a schematic diagram illustrating the process of another image deblurring method provided by the present application, which can be performed by Figure 5 The processing modules shown are executed. Figure 6 As shown, the method includes:

[0134] S601: Acquire a first image captured by a first camera.

[0135] The description of this step can be found in Figure 2 The description of S201 in the illustrated embodiment will not be repeated here.

[0136] S602: Acquire first event data generated by an event camera within an exposure time of a first camera capturing a first image.

[0137] In this embodiment, the first event data generated by the event camera within the exposure time of the first camera shooting the first image is also referred to as the first event data corresponding to the first image.

[0138] Since the data generated by the event camera is usually in the form of a sequence, in this embodiment, the first event data is also referred to as first sequence data.

[0139] It should be noted that, in this embodiment, there is no sequential execution relationship between S601 and S602.

[0140] S603: Use a fuzzy feature extractor to obtain fuzzy information of the first image based on the first image and the first event data.

[0141] In this embodiment, the fuzzy feature extractor is used to obtain fuzzy information of the image based on the image and event data corresponding to the image.

[0142] In this embodiment, after the first image and the first event data are acquired, the fuzzy information of the first image is obtained through a fuzzy feature extractor.

[0143] Below, several methods for obtaining fuzzy feature extractors are described:

[0144] In the first implementation, it includes:

[0145] 1) Construct multiple training samples.

[0146] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the sorted sample pair, and a label.

[0147] Among them, the event data corresponding to the two images in the sorted sample pair are the same.

[0148] The label indicates the image-level blur ranking between the two images with different blur levels. For details, please refer to Figure 2 The description in the illustrated embodiment will not be repeated here.

[0149] 2) Train the neural network through multiple training samples.

[0150] For example, the neural network includes a feature extractor and a global sorter. The feature extractor is used to receive an image, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the global sorter. The global sorter is used to receive the cascaded features of each image in the sorted sample pair sent by the feature extractor and output the image-level fuzzy degree sorting between the sorted sample pairs.

[0151] 3) Use the feature extractor in the trained neural network as a fuzzy feature extractor.

[0152] At this time, the fuzzy information of the first image can be obtained by inputting the first image and the first event data into the fuzzy feature extractor.

[0153] The second implementation includes:

[0154] 1) Construct multiple training samples.

[0155] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the sorted sample pair, and a label.

[0156] The label indicates the pixel-level blur ranking between the two images with different blur levels. Figure 2 The description in the illustrated embodiment will not be repeated here.

[0157] 2) Train the neural network through multiple training samples.

[0158] For example, the neural network includes a feature extractor and a local sorter, the feature extractor is used to receive an image, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the local sorter, the local sorter is used to receive the cascaded features of each image in the sorted sample pair sent by the feature extractor and output the pixel-level fuzzy degree sorting between the sorted sample pairs.

[0159] 3) Use the feature extractor in the trained neural network as a fuzzy feature extractor.

[0160] At this time, the fuzzy information of the first image can be obtained by inputting the first image and the first event data into the fuzzy feature extractor.

[0161] In the third implementation, the first implementation and the second implementation are combined, including:

[0162] In implementation, a method for obtaining a fuzzy feature extractor for extracting image-level fuzzy information and pixel-level fuzzy information includes:

[0163] 1) Construct multiple training samples.

[0164] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the sorted sample pair, and a label.

[0165] The labels indicate the image-level blur ranking and the pixel-level blur ranking between the pair of images with different blur levels.

[0166] 2) Train the neural network through multiple training samples.

[0167] For example, Figure 7 As shown, the neural network includes a feature extractor, a global sorter and a local sorter. The feature extractor is used to receive images, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the global sorter and the local sorter. The global sorter is used to receive the cascaded features of each image in the sorted sample pairs sent by the feature extractor and output the image-level fuzzy degree ranking between the sorted sample pairs. The local sorter is used to receive the cascaded features of each image in the sorted sample pairs sent by the feature extractor and output the image-level fuzzy degree ranking between the sorted sample pairs and the pixel-level fuzzy program ranking.

[0168] 3) Use the feature extractor in the trained neural network as a fuzzy feature extractor.

[0169] At this time, the fuzzy information of the first image can be obtained by inputting the first image and the first event data into the feature extractor.

[0170] In this embodiment, no matter it is the first implementation manner, the second implementation manner or the third implementation manner, the neural network trained in these three manners is also referred to as a second neural network.

[0171] S604, using a first neural network to obtain a second image after deblurring the first image based on the first image and blur information of the first image, wherein the first neural network is used to obtain an image after deblurring the image based on the image and blur information of the image.

[0172] Specifically, in this embodiment, the first neural network is a trained neural network for obtaining a deblurred image based on an image and blur information of the image, wherein the blur information of the image is obtained based on the image and event data corresponding to the image.

[0173] As an example, a method of obtaining the first neural network is described.

[0174] 1) Construct multiple training samples.

[0175] Specifically, each of the multiple training samples includes an image, event data corresponding to the image, blur information of the image, and a label, wherein the label indicates a clear image corresponding to the image.

[0176] 2) Building a neural network model and training it through multiple training samples, and determining the trained neural network as the first neural network.

[0177] In this embodiment, when building a neural network model, for at least one feature extractor included in the neural network model, its input includes not only the image feature data input by the previous layer feature extractor, but also the fuzzy information obtained after the input image input to the neural network model passes through the fuzzy feature extractor. That is, the fuzzy information of the image is embedded in the image features.

[0178] It can be seen that in this embodiment, when the first image is to be deblurred, the blur information of the first image is first obtained based on the first image and the first event data corresponding to the first image. That is, the first event data collected by the event camera is integrated when obtaining the blur information of the first image. Therefore, the deblurring effect when deblurring the first image can be further improved, thereby improving the quality of the image after deblurring.

[0179] Optionally, it is possible that the event camera collects multiple sets of sequence data within the exposure time of the image. In this case, the Figure 6 The clear image corresponding to the label indication image in the embodiment shown in the figure can also be replaced by a clear image sequence (i.e., multiple clear images) corresponding to the indication. Accordingly, when the first event data is multiple sets of sequence data, after the first image, the first event data, and the fuzzy information of the first image are input into the trained first neural network, the first neural network can output a clear image sequence (i.e., output multiple clear images).

[0180] Furthermore, in this embodiment, before using the first neural network to deblur the image, it can also include: obtaining j images of different resolutions based on the first image, the j images of different resolutions include a first image of a first resolution and j-1 images of different resolutions after reducing the first image; using a fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images of different resolutions and the first event data; accordingly, using the first neural network to deblur the first image based on the first image and the fuzzy information of the first image to obtain the second image, including: the first neural network deblurring the first image based on the first image, the fuzzy information of the first image, the j-1 images of different resolutions and the fuzzy information of each of the j-1 images of different resolutions to obtain the second image.

[0181] Next, let j be 3, take the case where the event camera collects multiple sets of sequence data within the exposure time of the first camera, and when the first neural network is used for deblurring, the first neural network can output a clear image sequence as an example, combined with Figure 8 , introduce a complete image deblurring method.

[0182] Specifically, the method includes:

[0183] Step 1: Acquire a first image of a first resolution (for example, represented as B1) and first event data corresponding to the first image.

[0184] Step 2: down-sample the first image B1 by 2 times to obtain an image B2 with a second resolution, and down-sample the image B2 by 2 times to obtain an image B3 with a third resolution.

[0185] Through step 2, three images with different scales will be obtained.

[0186] Step 3: Use the feature extractor in the trained blur estimation network to obtain blur information corresponding to image B1, image B2, and image B3 respectively.

[0187] The feature extractor in the fuzzy estimation network is used to obtain the fuzzy information of the image, that is, the fuzzy feature extractor. Specifically, during implementation, it may include:

[0188] 1) Construct at least one training sample pair, each training sample includes a sorted sample pair, event data corresponding to the sorted sample pair, and a label.

[0189] More specifically, based on the assumption that the longer the exposure time, the greater the image blur, that is, the more clear frames are used to synthesize blurry frames, the blurrier the image, and image-based downsampling can reduce the blur of the image, when constructing the sorted sample pairs, the sorted sample pairs can include two types, namely, sorted sample pairs with different exposure times and sorted sample pairs with different resolutions.

[0190] For example, for sorted sample pairs with different exposure times, in specific implementation, one frame of image can be selected as the center frame, and then T frames are selected around it to synthesize a blurred frame (the value of T is not limited, for example, one of {3, 5, 7, 9, 11}), where the two blurred samples generated by selecting different T values ​​are sorted sample pairs with different exposure times. Optionally, the selected frame can be interpolated using the "SuperSlomo" method and then all frames can be averaged to avoid ghosting in the image.

[0191] For example, for sorted sample pairs with different resolutions, during implementation, T frames can be randomly selected to generate blurred frames, which are then downsampled S times using the bicubic interpolation method (the value of S is not restricted, for example, any value in {1, 2, 4}) to generate two blurred samples with different resolutions, which are sorted sample pairs with different resolutions.

[0192] The generation of event data corresponding to the sorted sample pairs is simulated using, for example, the V2E method.

[0193] 2) Training a fuzzy estimation network using the at least one training sample pair.

[0194] For example, Figure 8 As shown, the neural network includes a feature extractor, a global sorter and a local sorter. The feature extractor is used to receive images, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the global sorter and the local sorter. The global sorter is used to receive the cascade features of each image in the sorted sample pairs sent by the feature extractor and output the image-level blur degree ranking between the sorted sample pairs. The local sorter is used to receive the cascade features of each image in the blurred image pairs sent by the feature extractor and output the pixel-level blur degree ranking between the sorted sample pairs. Then, through the image-level blur degree ranking output by the global sorter and the pixel-level ranking output by the local sorter and the optimization neural network of the label, that is, using the global sorting loss and the local sorting loss as supervision, when the conditions are met, such as when all training sample pairs are trained or when the preset number of iterations is reached, a trained blur estimation network is obtained.

[0195] It is noted that this embodiment does not limit the specific forms of the feature extractor, global sorter, and local sorter in the fuzzy estimation network. For example, the feature extractor includes 4 residual modules, the global sorter includes 1 convolution layer, 1 global average pooling layer, and 1 fully connected layer. The local sorter includes 1 convolution layer and 1 Sigmoid activation function.

[0196] It should be understood that due to the mismatch in size between the sorted sample pairs with different resolutions, it is impossible to calculate the local loss sorting. At this time, during training, optimization can be performed only through the global sorting network without the local sorting network.

[0197] During testing, we obtain the information output by the feature extractor in the blur estimation network for subsequent deblurring tasks.

[0198] Step 4: Use the trained clear sequence recovery network (i.e., the first neural network) to obtain a clear sequence after deblurring the first image. The clear sequence recovery network is used for deblurring processing and outputs a clear image sequence after deblurring.

[0199] For example, Figure 8 As shown, the clear sequence recovery network in this embodiment includes 3 layers, the first layer includes a convolutional layer, a feature learning module 1 (i.e., the learning module 1 in the figure), a feature aggregation module 1 (i.e., the aggregation module 1 in the figure), a residual module and a convolutional layer, the second layer includes a convolutional layer, a feature learning module 2 (i.e., the learning module 2 in the figure), a feature aggregation module 2 (i.e., the aggregation module 2 in the figure), a residual module and a convolutional layer, and the third layer includes a convolutional layer, a feature learning module 1 (i.e., the learning module 3 in the figure), a residual module and a convolutional layer.

[0200] Specifically, it includes:

[0201] 1) The image B1 is subjected to a convolutional layer to extract image features, and then subjected to a feature learning module 1 to obtain the modulation features of the image B1 (i.e., the features output by the feature learning module 1). The input of the feature learning module 1 also includes the fuzzy information of the image B1.

[0202] 2) The image B2 is subjected to a convolutional layer to extract image features, which are then concatenated with the features output by the feature learning module 1 and passed through the feature learning module 2 to obtain the modulation features of the image B2 (i.e., the features output by the feature learning module 2). The input of the feature learning module 2 also includes the fuzzy information of the image B2.

[0203] 3) The image B3 is subjected to a convolutional layer to extract image features, which are then concatenated with the features output by the feature learning module 2 and then subjected to the feature learning module 3 to obtain the modulation features of the image B3 (i.e., the features output by the feature learning module 3). The input of the feature learning module 3 also includes the fuzzy information of the image B3.

[0204] The input of each feature learning module above includes not only the image of the corresponding resolution, but also the fuzzy information of the corresponding resolved image. That is, in this way, the fuzzy information can be embedded into the image features, and the fuzzy information can be used to guide the feature extraction process.

[0205] It should be noted that this embodiment does not limit the specific form of the feature learning module. For example, the feature learning module is composed of 2 convolutional layers and 1 residual module.

[0206] 4) The modulation features of the images B1, B2 and B3 are input into the feature aggregation module 1 to obtain the aggregated features of the image B1 (i.e., the features output by the feature aggregation module 1). The input of the feature aggregation module 1 also includes the fuzzy information of the image B1.

[0207] 5) The modulation features of images B1, B2 and B3 are input into the feature aggregation module 2 to obtain the aggregated features of image B2 (i.e., the features output by the feature aggregation module 2). The input of the feature aggregation module 2 also includes the fuzzy information of image B2.

[0208] 6) The modulation features of image B3 are used as the aggregate features of image B3.

[0209] The input of each of the above feature aggregation modules not only includes the modulation features output by the previous layer of feature aggregation module, but also includes the fuzzy information of the corresponding resolved image. In this way, the fuzzy information can be embedded into the feature aggregation module, and the fuzzy information can be used to guide the feature aggregation process.

[0210] It should be noted that this embodiment does not limit the specific form of the feature aggregation module. For example, the feature aggregation module includes 4 convolutional layers and 1 softmax function. The fuzzy feature information of image B1, image B2, and image B3 passes through the convolutional layer respectively, and then performs a position-by-position softmax operation to generate aggregation weights. The modulated features of image B1, image B2, and image B3 are cascaded and multiplied element-by-element with the aggregation weights through the convolutional layer to obtain the aggregated features.

[0211] 7) The aggregated features of image B3 are added to image B3 after passing through the residual module and convolution layer of the third layer to obtain a clear sequence with the same scale as image B3. Among them, the output of the residual module of the third layer is upsampled for use in the next step.

[0212] 8) The aggregated features of image B2 are concatenated with the upsampling result in step 7), and added to image B2 after passing through the residual modulus and convolutional layer of the second layer to obtain a clear sequence with the same scale as image B2. Among them, the output of the residual modulus of the second layer is upsampled for use in the next step.

[0213] 9) The aggregated features of image B1 are concatenated with the upsampling result in step 8), and added to image B1 after passing through the residual modulus and convolution layer of the first layer to obtain a clear sequence with the same scale as image B1.

[0214] Next, a method for training a clear sequence restoration network is described. For example, a second training sample is obtained, the second training sample includes 3 images, blur information of each of the 3 images, and a label of each of the 3 images, and the label of each image is a clear sequence image after each image is deblurred; based on the second training sample, Figure 7 The neural network of the structure shown is trained. For example, the clear sequences of each resolution use L1 loss as supervision, and the stochastic gradient descent method is used to update the model parameters in iteration to obtain a trained clear sequence restoration network.

[0215] Combined with the above Figures 2 to 8 , describes in detail the image processing method of the embodiment of the present application, and will now be combined with Fig. 9 and Fig.10 The image processing device provided by the present application is described in detail.

[0216] Fig. 9 The following is a schematic diagram of the structure of an image processing device provided by an embodiment of the present application. Fig. 9 As shown, the device 900 includes: an acquisition module 901 and a processing module 902 .

[0217] Specifically, 901, an acquisition module 901 is used to acquire a first image; a processing module 902 is used to determine blur information of the first image, where the blur information indicates a blurred area in the first image; the processing module 902 is also used to use a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, where the first neural network is used to deblur the image based on the image and the blur information of the image.

[0218] In one example, the acquisition module 901 is used to execute Figure 2 The step of acquiring a first image in S201 in the embodiment.

[0219] In another example, the processing module 902 is used to execute Figure 2 The step of using a first neural network to deblur the first image based on the first image and blur information of the first image in S203 of the embodiment to obtain a second image.

[0220] In a possible implementation, the acquisition module 901 is also used to: acquire first event data generated by the event camera within a first time length, where the first time length is the exposure time length used by the first camera for capturing the first image; the processing module 902 is also used to: determine blur information of the first image based on the first image and the first event data.

[0221] In a possible implementation, when the processing module 902 determines the blur information of the first image based on the first image and the first event data, the processing module 902 is specifically used to: input the first image and the first event data into a fuzzy feature extractor, and then determine the features output by the fuzzy feature extractor as the blur information of the first image; wherein the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on first training data, the first training data includes a pair of images with different blur degrees, event data corresponding to a pair of images with different blur degrees, and labels, the event data corresponding to a pair of images with different blur degrees are the same, and the label is used to indicate the image-level blur degree ranking and / or pixel-level blur degree ranking between a pair of images with different blur degrees.

[0222] In a possible implementation, the processing module 902 is further used to: obtain j images of different resolutions based on the first image, the j images of different resolutions including a first image of a first resolution and j-1 images of different resolutions that are obtained by reducing the first image; obtain fuzzy information of each image based on each image in the j-1 images of different resolutions and the first event data using a fuzzy feature extractor; and deblur the first image using a first neural network based on the first image, the fuzzy information of the first image, the j-1 images of different resolutions, and the fuzzy information of each of the j-1 images of different resolutions to obtain a second image.

[0223] In a possible implementation, the acquisition module 901 is further used to: acquire second training data, the second training data including a third image of a first resolution and blur information of the third image, j-1 images of different resolutions after reducing the third image and their respective blur information, the third image and respective labels of j-1 images of different resolutions after reducing the third image, the label of each image being used to indicate the image of each image after deblurring; the processing module 902 is further used to train a neural network based on the second training data to obtain a first neural network.

[0224] Optional, Fig. 9The image processing device can be deployed in a general-purpose computer, a server, or as an embedded module. Among them, the image processing device described in this application is installed in a general-purpose computer, and the collected RGB and event data can be processed to obtain a clear image or image sequence and saved locally. The image processing device described in this application is installed in the server and runs as a service in the server. Other devices can restore the image by sending requests to call the computing resources of the server and the image processing device described in this application. When used as an embedded module, it may exist in devices such as smartphones, tablet computers, vehicle-mounted mobile devices, monitoring systems, and VR / AR glasses.

[0225] Fig.10 A structural schematic diagram of an image processing device provided in another embodiment of the present application. Fig.10 The device shown can be used to execute the method described in any of the above embodiments.

[0226] like Fig.10 As shown, the apparatus 1000 of this embodiment includes: a memory 1001 and a processor 1002. Optionally, the apparatus 1000 further includes a communication interface 1003 and a bus 1004. The memory 1001, the processor 1002, and the communication interface 1003 are connected to each other through the bus 1004 for communication.

[0227] The memory 1001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1001 may store a program. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 is used to execute the program. Figures 2 to 8 The individual steps of the method are shown.

[0228] The processor 1002 may be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits for executing related programs to implement the present application. Figures 2 to 8 The method shown.

[0229] The processor 1002 may also be an integrated circuit chip having the ability to process signals. Figures 2 to 8 Each step of the method can be completed by an integrated logic circuit of hardware in the processor 1002 or an instruction in the form of software.

[0230] The processor 1002 may also be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0231] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001 and completes the functions required to be executed by the units included in the device of the present application in combination with its hardware. For example, it can execute Figures 2 to 8 The various steps / functions of the illustrated embodiment.

[0232] The communication interface 1003 may use, but is not limited to, a transceiver or other transceiver device to implement communication between the apparatus 1000 and other devices or a communication network.

[0233] The bus 1004 may include a path for transmitting information between various components of the device 1000 (eg, the memory 1001 , the processor 1002 , and the communication interface 1003 ).

[0234] It should be understood that the apparatus 1000 shown in the embodiment of the present application may be an electronic device, or may be a chip configured in an electronic device. The apparatus 1000 may be deployed in a terminal device, or may be deployed in a network device.

[0235] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0236] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0237] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0238] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0239] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0240] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0244] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0245] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: acquiring a first image; determining blur information of the first image, the blur information indicating a blurry area in the first image; A first neural network is used to deblur the first image based on the first image and blur information of the first image to obtain a second image, wherein the first neural network is used to deblur the image based on the image and blur information of the image.

2. The method according to claim 1, characterized in that: The method further comprises: Acquire first event data generated by an event camera within a first time period, where the first time period is an exposure time period used by a first camera to capture the first image; The determining the blur information of the first image includes: Blur information of the first image is determined based on the first image and the first event data.

3. The method according to claim 2, characterized in that The determining the blur information of the first image based on the first image and the first event data comprises: After the first image and the first event data are input into a fuzzy feature extractor, the feature output by the fuzzy feature extractor is determined as fuzzy information of the first image; Among them, the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data. The first training data includes a pair of images with different blur levels, event data corresponding to the pair of images with different blur levels, and labels. The event data corresponding to the pair of images with different blur levels are the same, and the label is used to indicate the image-level blur level sorting and / or pixel-level blur level sorting between the pair of images with different blur levels.

4. The method according to claim 3, characterized in that The pair of images with different blur levels have different exposure durations, or the pair of images with different blur levels have different resolutions.

5. The method according to claim 4, characterized in that The method further comprises: Obtain j images with different resolutions based on the first image, wherein the j images with different resolutions include the first image with a first resolution and j-1 images with different resolutions that are obtained by reducing the first image; Using the fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images with different resolutions and the first event data; The step of using a first neural network to deblur the first image based on the first image and blur information of the first image to obtain a second image includes: A first neural network is used to deblur the first image based on the first image, the blur information of the first image, the j-1 images with different resolutions, and the blur information of each of the j-1 images with different resolutions to obtain a second image.

6. The method according to claim 5, characterized in that j is 3, and the j-1 images with different resolutions include an image obtained by downsampling the first image by a factor of 2 and an image obtained by further downsampling the image obtained by downsampling the first image by a factor of 2 by a factor of 2.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: Acquire second training data, wherein the second training data includes a third image of a first resolution and blur information of the third image, j-1 images with different resolutions reduced from the third image and their respective blur information, the third image and labels of the j-1 images with different resolutions reduced from the third image, wherein the label of each image is used to indicate the image after each image is deblurred; A neural network is trained based on the second training data to obtain the first neural network.

8. The method according to any one of claims 1 to 7, characterized in that The first camera is an RGB camera.

9. The method according to any one of claims 1 to 8, characterized in that The first neural network is a convolutional neural network.

10. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image; a processing module, configured to determine blur information of the first image, wherein the blur information indicates a blurry area in the first image; The processing module is further used to use a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, and the first neural network is used to deblur the image based on the image and the blur information of the image.

11. The device according to claim 10, characterized in that: The acquisition module is also used for: Acquire first event data generated by an event camera within a first time period, where the first time period is an exposure time period used by a first camera to capture the first image; The processing module is also used for: Blur information of the first image is determined based on the first image and the first event data.

12. The device according to claim 11, characterized in that When the processing module determines the blur information of the first image based on the first image and the first event data, the processing module is specifically configured to: After the first image and the first event data are input into a fuzzy feature extractor, the feature output by the fuzzy feature extractor is determined as fuzzy information of the first image; Among them, the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data. The first training data includes a pair of images with different blur levels, event data corresponding to the pair of images with different blur levels, and labels. The event data corresponding to the pair of images with different blur levels are the same, and the label is used to indicate the image-level blur level sorting and / or pixel-level blur level sorting between the pair of images with different blur levels.

13. The device according to claim 12, characterized in that The pair of images with different blur levels have different exposure durations, or the pair of images with different blur levels have different resolutions.

14. The device according to claim 13, characterized in that The processing module is also used for: Obtain j images with different resolutions based on the first image, wherein the j images with different resolutions include the first image with a first resolution and j-1 images with different resolutions that are obtained by reducing the first image; Using the fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images with different resolutions and the first event data; A first neural network is used to deblur the first image based on the first image, the blur information of the first image, the j-1 images with different resolutions, and the blur information of each of the j-1 images with different resolutions to obtain a second image.

15. The device according to claim 14, characterized in that j is 3, and the j-1 images with different resolutions include an image obtained by downsampling the first image by a factor of 2 and an image obtained by further downsampling the image obtained by downsampling the first image by a factor of 2 by a factor of 2.

16. The device according to claim 14 or 15, characterized in that The acquisition module is also used for: Acquire second training data, wherein the second training data includes a third image of a first resolution and blur information of the third image, j-1 images with different resolutions reduced from the third image and their respective blur information, the third image and labels of the j-1 images with different resolutions reduced from the third image, wherein the label of each image is used to indicate the image after each image is deblurred; The processing module is further used to train a neural network based on the second training data to obtain the first neural network.

17. The device according to any one of claims 10 to 16, characterized in that The first camera is an RGB camera.

18. The device according to any one of claims 10 to 16, characterized in that The first neural network is a convolutional neural network.

19. An automatic driving device, characterized in that: Used to execute the image processing method according to any one of claims 1 to 9.

20. An automatic driving system, characterized in that: It includes the image processing device as described in any one of claims 10 to 18 or the automatic driving device as described in claim 19.

21. A vehicle, characterized in that: Including an automatic driving system as described in claim 20.

22. A server, characterized in that: The invention comprises the image processing device as claimed in any one of claims 10 to 18.

23. An image processing device, characterized in that: include: Memory and processor; The memory is used to store program instructions; The processor is configured to call the program instructions in the memory to execute the method according to any one of claims 1 to 9.

24. A computer readable medium, characterized in that The computer-readable medium stores a program code for computer execution, the program code including instructions for executing the method according to any one of claims 1 to 9.

25. A computer program product, comprising computer program code, characterized in that: When the computer program code is executed on a computer, the computer is caused to implement the method according to claims 1 to 9.

Images