Image processing method and related apparatus

Abstract

Provided in the present application are an image processing method and a related apparatus. The image processing method comprises: acquiring a first image; determining blur information indicating a blur region in the first image; and using a first neural network to perform deblurring processing on the first image on the basis of the first image and the blur information of the first image, so as to obtain a second image. In the method, the first neural network is trained to deblur an image. Specifically, during training, the first neural network is based on an image and blur information for indicating a blur region of the image. Therefore, compared with a neural network obtained by means of training on the basis of only an image, when a first neural network is used for deblurring an image, it is helpful to improve the definition of the deblurred image.

Description

Image processing method and related device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 9, 2023, with application number 202311002207.8 and application name “Image Processing Method and Related Devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of autonomous driving, and in particular to an image processing method and related devices. Background Art

[0003] Image quality is crucial in the fields of autonomous driving and video surveillance. However, in real-world conditions, captured images are often blurred due to factors such as camera shake or rapid motion of the target object. Therefore, image deblurring is necessary.

[0004] Currently, the method for deblurring an image is as follows: constructing a training sample, which includes a blurred image and a label, and the label indicates the clear image corresponding to the blurred image; training a neural network based on the training sample to obtain a trained neural network; inputting a new blurred image into the trained neural network to obtain an image after deblurring the new blurred image.

[0005] However, the clarity of the deblurred image cannot meet the clarity requirements in many scenarios.

[0006] Summary of the Invention

[0007] The present application provides an image processing method and related devices, in order to improve the clarity of the deblurred image.

[0008] In a first aspect, the present application provides an image processing method, comprising: acquiring a first image; determining blur information of the first image, where the blur information indicates a blurry area in the first image; using a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, wherein the first neural network is used to deblur the image based on the image and the blur information of the image.

[0009] The blur information of the first image may be considered as some information that can indicate a blurry area in the first image.

[0010] It should be noted that this embodiment does not limit the method of determining the blur information of the first image.

[0011] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the image-level blur ranking between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network being an image, and then after the first image is input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0012] For example, a plurality of training samples are constructed, each of the plurality of training samples including a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the pixel-level blur level sorting between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network being an image, and then after the first image is input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0013] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the image-level blur level sorting and the pixel-level blur level sorting between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network being an image, and then after the first image is input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as blur information of the first image.

[0014] In this embodiment, after the blur information of the first image is determined, a first neural network is used to deblur the first image based on the first image and the blur information of the first image, wherein the first neural network is trained for image deblurring. In other words, in this embodiment, the first image and the blur information of the first image are input into the first neural network, and the image output by the first neural network serves as the image after the deblurring of the first image.

[0015] Specifically, the first neural network in this application is trained based on the image and the acquired blur information of the image, rather than being trained based solely on the image. The blur information of the image can be considered to be information that can indicate blurred areas in the image. Therefore, compared to a neural network trained based solely on the image, deblurring the image using the first neural network helps improve the clarity of the deblurred image.

[0016] In combination with the first aspect, in a possible implementation, the method further includes: obtaining first event data generated by an event camera within a first time length, where the first time length is the exposure time length used by a first camera to capture a first image; determining blur information of the first image, including: determining the blur information of the first image based on the first image and the first event data.

[0017] In this embodiment, when acquiring the blur information of the first image, since the event data generated by the event camera, i.e., the first event data, is also referred to, it helps to improve the accuracy of the blur information of the acquired first image, thereby helping to further improve the deblurring effect.

[0018] In one implementation, determining the fuzzy information of the first image based on the first image and the first event data includes: inputting the first image and the first event data into a fuzzy feature extractor, and determining features output by the fuzzy feature extractor as the fuzzy information of the first image;

[0019] Among them, the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data. The first training data includes a pair of images with different blur levels, event data corresponding to the pair of images with different blur levels, and labels. The event data corresponding to the pair of images with different blur levels are the same, and the label is used to indicate the image-level blur level sorting and / or pixel-level blur level sorting between a pair of images with different blur levels.

[0020] For example, in one implementation, multiple training samples are constructed, each of which includes a pair of images with different blur levels (also referred to as a sorted sample pair), event data corresponding to the pair of images with different blur levels, and a label. The event data corresponding to the pair of images with different blur levels are the same, and the label indicates the image-level blur ranking between the pair of images with different blur levels. A neural network is trained using the multiple training samples, with the inputs of the neural network being the images and the event data. After the first image and the first event data are input to the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as the blur information of the first image.

[0021] For example, a plurality of training samples are constructed, each of the plurality of training samples including a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the pair of images with different blur levels, and labels, wherein the event data corresponding to the pair of images with different blur levels are the same, and the label indicates the pixel-level blur level sorting between the pair of images with different blur levels; a neural network is trained using the plurality of training samples, the input of the neural network being the image and the event data, and then after the first image and the first event data are input into the trained neural network, a feature output by a convolutional layer in the trained neural network is determined as the blur information of the first image.

[0022] For example, a plurality of training samples are constructed, each of the plurality of training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data and labels corresponding to the pair of images with different blur levels, wherein the event data corresponding to the pair of images with different blur levels are the same, and the label indicates the image-level blur level sorting and the pixel-level blur level sorting between the pair of images with different blur levels, and a neural network is trained using the plurality of training samples, the input of the neural network being the image and the event data, and then after the first image and the first event data are input into the trained neural network, the feature output by a convolutional layer in the trained neural network is determined as the blur information of the first image.

[0023] Optionally, a pair of images with different blur levels have different exposure times, or a pair of images with different blur levels have different resolutions.

[0024] In combination with the first aspect, in a possible implementation, the method further includes: obtaining j images of different resolutions based on the first image, the j images of different resolutions including a first image of the first resolution and j-1 images of different resolutions after reducing the first image; using a fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images of different resolutions and first event data; wherein, using a first neural network to deblur the first image based on the first image and the fuzzy information of the first image to obtain the second image includes: using the first neural network to deblur the first image based on the first image, the fuzzy information of the first image, the j-1 images of different resolutions, and the fuzzy information of each of the j-1 images of different resolutions to obtain the second image.

[0025] Optionally, j is 3, and the j-1 images with different resolutions include an image obtained by downsampling the first image by a factor of 2 and an image obtained by further downsampling the image obtained by downsampling the first image by a factor of 2 by a factor of 2.

[0026] Specifically, in the method of obtaining the first neural network, second training data is obtained, where the second training data includes a third image of a first resolution and blur information of the third image, j-1 images of different resolutions after reducing the third image and their respective blur information, the third image and respective labels of the j-1 images of different resolutions after reducing the third image, and the label of each image is used to indicate the image of each image after deblurring; the neural network is trained based on the second training data to obtain the first neural network.

[0027] With reference to the first aspect, in a possible implementation, the first camera is an RGB camera.

[0028] In combination with the first aspect, in one possible implementation, the first neural network is a convolutional neural network.

[0029] In a second aspect, the present application provides an image processing device, comprising: an acquisition module for acquiring a first image; a processing module for determining blur information of the first image, wherein the blur information indicates a blurred area in the first image; the processing module is also used to use a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, wherein the first neural network is used to deblur the image based on the image and the blur information of the image.

[0030] In combination with the second aspect, in a possible implementation, the acquisition module is also used to: acquire first event data generated by the event camera within a first time length, where the first time length is the exposure time length used by the first camera to capture the first image; the processing module is also used to: determine the blur information of the first image based on the first image and the first event data.

[0031] In combination with the second aspect, in a possible implementation, when the processing module determines the blur information of the first image based on the first image and the first event data, the processing module is specifically used to: input the first image and the first event data into a fuzzy feature extractor, and then determine the features output by the fuzzy feature extractor as the blur information of the first image; wherein the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data, and the first training data includes a pair of images with different blur degrees, event data corresponding to a pair of images with different blur degrees, and labels, the event data corresponding to a pair of images with different blur degrees are the same, and the label is used to indicate the image-level blur degree ranking and / or pixel-level blur degree ranking between a pair of images with different blur degrees.

[0032] In conjunction with the second aspect, in a possible implementation, a pair of images with different blur levels have different exposure times, or a pair of images with different blur levels have different resolutions.

[0033] In conjunction with the second aspect, in one possible implementation, the processing module is further configured to: obtain j images of different resolutions based on the first image, where the j images of different resolutions include the first image of the first resolution and j-1 images of different resolutions that are obtained by reducing the first image;

[0034] A fuzzy feature extractor is used to obtain fuzzy information of each image based on each image in j-1 images with different resolutions and the first event data; and a first neural network is used to deblur the first image based on the first image, the fuzzy information of the first image, the j-1 images with different resolutions, and the fuzzy information of each of the j-1 images with different resolutions to obtain a second image.

[0035] Optionally, j is 3, and the j-1 images with different resolutions include an image obtained by downsampling the first image by 2 times, an image obtained by further downsampling the image obtained by downsampling the first image by 2 times, and an image obtained by further downsampling the image obtained by downsampling the first image by 2 times.

[0036] In combination with the second aspect, in a possible implementation, the acquisition module is further used to: acquire second training data, the second training data including a third image of a first resolution and blur information of the third image, j-1 images of different resolutions after reducing the third image and their respective blur information, the third image and their respective labels of j-1 images of different resolutions after reducing the third image, the label of each image is used to indicate the image of each image after deblurring; the processing module is further used to train a neural network based on the second training data to obtain a first neural network.

[0037] With reference to the second aspect, in a possible implementation, the first camera is an RGB camera.

[0038] In combination with the second aspect, in one possible implementation, the first neural network is a convolutional neural network.

[0039] In a fourth aspect, the present application provides an automatic driving device for executing the image processing method described in the first aspect or any one of its implementation methods.

[0040] In a fifth aspect, the present application provides an autonomous driving system, comprising an image processing device as described in the second aspect.

[0041] In a sixth aspect, an image processing device includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions in the memory to execute the method described in the first aspect.

[0042] In a seventh aspect, the present application provides a computer-readable medium, wherein the computer-readable medium stores a program code for computer execution, wherein the program code includes instructions for executing the method described in the first aspect or any possible implementation thereof.

[0043] In an eighth aspect, the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the method described in the first aspect or any possible implementation method thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a structural diagram of an image deblurring system provided by the present application;

[0045] FIG2 is a schematic diagram of the process of an image deblurring method provided by one embodiment of the present application;

[0046] FIG3 is a schematic diagram of a neural network training method based on sorted sample pairs provided by the present application;

[0047] FIG4 is a schematic diagram of obtaining a first neural network provided by the present application;

[0048] FIG5 is a structural diagram of another image deblurring system provided by the present application;

[0049] FIG6 is a schematic diagram illustrating the process of an image deblurring method provided by another embodiment of the present application;

[0050] FIG7 is a schematic diagram of a neural network training method based on event data provided by the present application;

[0051] FIG8 is a structural diagram of an image deblurring method provided by another embodiment of the present application;

[0052] FIG9 is a structural schematic diagram of an image deblurring device provided in one embodiment of the present application;

[0053] FIG10 is a structural schematic diagram of an image deblurring apparatus provided in another embodiment of the present application. DETAILED DESCRIPTION

[0054] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] To facilitate understanding, the relevant terms involved in this application are first explained.

[0056] 1. Event Camera

[0057] An event camera, sometimes also called a dynamic vision sensor (DVS), is a new type of camera that mimics the workings of the human brain. It boasts high dynamic range, low latency, high frame rate, and low power consumption, enabling it to capture high-precision temporal and spatial information of fast-moving targets. Each pixel in the camera independently senses changes in light brightness. When the change exceeds a threshold, a specially designed circuit outputs an event. Each pixel operates independently and can output data asynchronously, resulting in an extremely high response speed (in microseconds), effectively improving tasks such as motion perception and image enhancement.

[0058] 2. Neural Networks

[0059] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0060] The key technology of today's artificial intelligence is neural networks (NNs). NNs simulate the connections between nerve cells in the human brain and widely interconnect a large number of simple processing units (called neurons) to form a complex network system.

[0061] A simple neural network consists of three layers: the input layer, the output layer, and the hidden layer (also called the middle layer). Each connection between layers is associated with a weight (called a parameter). Neural networks excel in fields like computer vision and natural language processing because they optimize their predictions by adjusting these weights through training algorithms.

[0062] Neural network training generally involves two computational steps: the first is the forward calculation, and the second is the backward calculation. The forward calculation involves calculating the input and parameters, then applying a nonlinear function to produce an output. This output either serves as the network's final output or serves as the input for subsequent similar calculations. The deviation between the network's output and the actual label of the corresponding sample is measured using the model's loss function, which is expressed as a function f(x, w) of the input sample x and the network parameters w. To minimize the loss function, the network parameters w must be continuously adjusted. The backward calculation is performed to obtain the updated value of the parameters w. In gradient descent algorithms, the backward calculation starts from the last layer of the neural network and calculates the partial derivatives of the loss function with respect to the parameters of each layer. Finally, the partial derivatives of all parameters, called the gradient, are obtained. At each iteration, the parameters w are updated in the opposite direction of the gradient by a fixed step size η to obtain the new parameters w, completing one step of training.

[0063] The process of training a neural network is the process of learning the weights corresponding to neurons. Its ultimate goal is to obtain the weights corresponding to each layer of neurons in the trained neural network model.

[0064] 3. Exposure duration

[0065] Exposure time refers to the time interval from the opening to the closing of the shutter, during which the target object can leave an image on the film.

[0066] Image quality is crucial in areas such as autonomous driving, video surveillance, and sports photography. For example, in autonomous driving scenarios, autonomous vehicles need to make decisions based on captured images, such as whether to turn left, right, or go straight. In these scenarios, image quality is a key factor influencing autonomous vehicle decision-making, and thus impacts the vehicle's safety.

[0067] However, in real-world conditions, captured images are often blurred due to various factors. For example, when the target object moves quickly or the camera shakes, the captured image will be blurred. Therefore, how to deblur images has become an urgent problem to be solved.

[0068] Currently, one method for deblurring an image is as follows: constructing training samples, each of which includes a blurred image and a label indicating the corresponding sharp image; training a neural network based on the training samples to obtain a trained neural network; and inputting a new blurred image into the trained neural network to obtain a deblurred image of the new blurred image. In other words, in this deblurring method, the neural network directly learns the mapping relationship between blurred and sharp images, and then inputs the blurred image to be deblurred into the trained neural network to obtain the deblurred image.

[0069] However, analysis has found that, in the above deblurring methods, the image quality after deblurring is still not good enough.

[0070] In view of this, the present application provides an image deblurring method to improve the quality of the deblurred image.

[0071] Figure 1 is a schematic diagram of the structure of the image deblurring system provided in this application. As shown in Figure 1, the deblurring system includes a first camera 101 and a processing module 102. The first camera 101 is used to capture an image that can describe information about the external environment, and then inputs the image into the processing module 102. The processing module 102 is used to receive the image sent by the first camera 101 and use a preset processing algorithm to perform deblurring on the image to obtain a deblurred image, thereby improving the quality of the video.

[0072] It should be noted that the embodiment of the present application does not limit the specific form of the first camera 101. For example, it can be a fisheye camera, an eagle-eye camera, or a monocular or binocular camera, which does not constitute a limitation of the present application.

[0073] It should also be noted that this embodiment does not limit the specific application scenario of the image deblurring system.

[0074] For example, the image deblurring system can be applied to the field of sports photography: when athletes are moving at high speed, the blurring phenomenon is obvious. Through the method of the present application, a deblurred image is provided.

[0075] For example, the image deblurring system can be applied to autonomous driving scenarios: when autonomous vehicles are driving and in low-light environments, the image quality of the autonomous vehicle degrades, and the image is easily blurred, which affects the autonomous driving perception and decision-making. The deblurring method provided in this application provides the autonomous vehicle with a deblurred image.

[0076] For example, the image deblurring system can be applied to video surveillance scenarios: in low-light environments, camera exposure time increases, and moving objects in surveillance images and videos are prone to local blurring, affecting the recognition of pedestrians and vehicles. The deblurring method provided in this application can produce a deblurred image.

[0077] Of course, the image deblurring system can also be applied to other fields that perform other image-based processing.

[0078] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0079] FIG2 is a flow chart of an image deblurring method provided by the present application, wherein the method of this embodiment may include S201, S202 and S203. The image processing method may be executed by the processing module shown in FIG1.

[0080] S201, acquiring a first image.

[0081] In this embodiment, acquiring the first image specifically refers to acquiring an image captured by the first camera within the exposure time for capturing information that can describe the external environment. As an example, the first camera is an RGB camera.

[0082] It should be noted that this embodiment does not limit the specific value of the exposure time of the first camera.

[0083] S202: Determine blur information of the first image, where the blur information indicates a blur area in the first image.

[0084] In this embodiment, after acquiring a first image, blur information of the first image is first obtained. The blur information specifically indicates a blurred area in the first image. In other words, in this embodiment, the blur information of the image refers to information that can describe the blurred area of ​​the image.

[0085] Several implementations for obtaining blur information indicating a blur region in the first image are described below.

[0086] The first implementation includes:

[0087] 1) Construct multiple training samples.

[0088] Specifically, each of the multiple training samples includes a pair of images with different blur levels and a label, wherein the label indicates an image-level blur ranking between the pair of images with different blur levels.

[0089] In this application, a pair of images with different blur levels included in a training sample is also referred to as a sorted sample pair.

[0090] For example, a pair of sorted samples is called image 1 and image 2; the label value is a scalar, for example, when it is 1, it indicates that the blurriness of image 1 is greater than that of image 2, and when it is 0, it indicates that the blurriness of image 1 is less than or equal to that of image 2. Alternatively, the label is a two-dimensional data, for example, when it is 01, it indicates that the blurriness of image 1 is greater than that of image 2, and when it is 10, it indicates that the blurriness of image 1 is less than or equal to that of image 2.

[0091] 2) Train the neural network through multiple training samples.

[0092] For example, the neural network includes a feature extractor and a global ranker. The feature extractor is used to receive an image, extract image features and output the image features to the global ranker. The global ranker is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the image-level fuzzy degree ranking between the sorted sample pairs.

[0093] It should be understood that in this manner, when training the neural network, what the neural network learns is the blur degree at the image level, that is, the neural network can be considered as a neural network for learning the blur degree at the image level.

[0094] 3) The feature extractor is determined to be a fuzzy feature extractor for extracting fuzzy information of the image.

[0095] At this time, the first image collected in S201 is input into the feature extractor to obtain the blur information of the first image.

[0096] The second implementation includes:

[0097] 1) Construct multiple training samples.

[0098] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, where the label indicates the pixel-level blur ranking between the two images with different blur levels. For example, for a sorted sample pair with two 8*8 resolution images, the label consists of 64 values, each indicating the blur ranking between the same pixels.

[0099] 2) Train the neural network through multiple training samples.

[0100] For example, the neural network includes a feature extractor and a local ranker, where the feature extractor is used to receive an image, extract image features and output the image features to the local ranker, and the local ranker is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the pixel-level blur degree ranking between the sorted sample pairs.

[0101] It should be understood that in this manner, when training the neural network, the neural network learns the blur degree at the pixel level, that is, the neural network can be considered as a neural network for learning the blur degree at the pixel level of an image.

[0102] 3) The feature extractor is determined to be a fuzzy feature extractor for extracting fuzzy information of the image.

[0103] At this time, the first image collected in S201 is input into the feature extractor to obtain blur information of the first image.

[0104] The third implementation is a combination of the first and second implementations. The implementation includes:

[0105] 1) Construct multiple training samples.

[0106] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair) and a label, wherein the label indicates the image-level blur level sorting and the pixel-level blur level sorting between the pair of images with different blur levels.

[0107] 2) Train the neural network through multiple training samples.

[0108] For example, as shown in Figure 3, the neural network includes a feature extractor, a global sorter and a local sorter. The feature extractor is used to receive images, extract image features and output the image features to the global sorter and the local sorter. The global sorter is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the image-level blur ranking between the sorted sample pairs. The local sorter is used to receive the features of each image in the sorted sample pair extracted by the feature extractor and output the pixel-level blur ranking between the sorted sample pairs.

[0109] It should be understood that in this way, when training the neural network, the neural network learns both the blur degree at the image level and the blur degree at the pixel level, that is, the neural network can be considered as a neural network for learning the blur degree at the image level and the blur degree at the pixel level.

[0110] 3) The feature extractor is determined to be a fuzzy feature extractor for extracting fuzzy information of the image.

[0111] At this time, the first image collected in S201 is input into the feature extractor to obtain blur information of the first image.

[0112] In this application, the blur information of an image is also referred to as blur degree information of the image.

[0113] S203: Deblurring the first image using a first neural network based on the first image and the blur information of the first image to obtain a second image, wherein the first neural network is used to deblur the image based on the image and the blur information of the image.

[0114] Specifically, in this embodiment, the first neural network is a trained neural network for obtaining a deblurred image based on an image and blur information of the image, wherein the blur information of the image is obtained based on the collected image.

[0115] As an example, a method of obtaining the first neural network is described in conjunction with FIG4 .

[0116] 1) Construct multiple training samples.

[0117] Specifically, each of the multiple training samples includes a blurred image and a label, wherein the label indicates a clear image corresponding to the blurred image.

[0118] 2) Build a neural network model and train it through multiple training samples.

[0119] In this embodiment, as shown in Figure 4, when building a neural network model, the neural network includes an input layer, multiple feature extractors and an output layer, wherein the input layer is used to receive an input image, and the output layer is used to output a deblurred image based on the received features.

[0120] Specifically, at least one of the multiple feature extractors receives input, in addition to the image feature data from the previous feature extractor, that includes the fuzzy information obtained after the image input to the neural network model passes through the fuzzy feature extractor. In other words, the fuzzy information indicating the fuzzy areas of the input image is embedded into the image features.

[0121] For example, for the neural network shown in Figure 4, the input to the second feature extractor includes fuzzy information. In this case, when training the neural network, after the image is input to the neural network through the input layer, the neural network first extracts features through the first feature extractor to obtain feature data 1. Feature data 1 and fuzzy information are then input to the second feature extractor to obtain further extracted features. These further extracted features are then input into subsequent models until an output image is obtained. The output image is then compared with the label of the input image to optimize the parameters of the neural network.

[0122] It should be noted that this application does not limit the specific model of the neural network shown in Figure 4. For example, the first feature extractor may include at least one convolutional layer, or the fuzzy information may be introduced into the first feature extractor or the last feature extractor, without limitation.

[0123] After completing the training through multiple training samples, a trained neural network is obtained, that is, the first neural network.

[0124] It should be understood that compared to a neural network trained only by images, since there is fuzzy information during the training process, that is, the neural network can also learn the image based on the fuzzy information of the image, when the trained first neural network is used to deblur the image to be tested, the deblurring effect is better than when the neural network obtained by training only by images is used to deblur the image to be tested, that is, the quality of the deblurred image can be improved.

[0125] Furthermore, in this embodiment, when the first image is to be deblurred, after obtaining the blur information of the first image, the first image and the blur information of the first image are input into the first neural network to obtain a second image after deblurring the first image.

[0126] It should be noted that the input to the first neural network described in this embodiment is different from the input to the input layer of the first neural network. For example, in the example shown in FIG4 , the input layer of the neural network is only used to receive the input image and does not receive blur information. However, the input to the first neural network includes both the image and the blur information of the image. In other words, the input to the first neural network considers the first neural network as a whole.

[0127] Since event cameras can capture changes in brightness, have the characteristics of high temporal resolution, high dynamic range, low data redundancy and low power, and can perceive changes in image grayscale values, the present application also provides another image deblurring method.

[0128] In this method, when deblurring an image, event data obtained by the event camera is also integrated to further improve the quality of the deblurred image. In other words, when deblurring an image, data from two cameras is used: one is captured by the first camera, and the other is captured by the event camera. Alternatively, when deblurring an image, the deblurring is performed jointly based on the data from both cameras. The event camera and the first camera are two independent cameras.

[0129] Before introducing the method for deblurring an image by fusing event data obtained by an event camera, we first introduce the structural schematic diagram of another image deblurring system provided by this application in conjunction with Figure 5. As shown in Figure 5, the image deblurring system includes a first camera 501, an event camera 502, and a processing module 503.

[0130] Among them, the first camera 501 is used to capture images that can describe external environmental information, and the event camera 502 is used to obtain event data corresponding to the images captured by the first camera 501. Among them, the event data corresponding to the images captured by the first camera 501 specifically refers to the event data captured by the event camera within the exposure time of the image taken by the first camera.

[0131] The processing module 503 is configured to perform deblurring processing on the image based on the image captured by the first camera and the event data captured by the event camera using a preset processing algorithm to obtain a deblurred image, thereby improving the deblurring effect.

[0132] It should also be noted that this embodiment does not limit the specific application scenario of the image deblurring system.

[0133] For example, the image deblurring system can be applied to sports photography scenarios, autonomous driving scenarios, or video surveillance scenarios.

[0134] Of course, the image deblurring system can also be applied to other fields that perform other image-based processing.

[0135] Specifically, when the event data obtained by the event camera is fused to further improve the deblurring effect, the blur information of the image is obtained based on the image and the event data corresponding to the image.

[0136] In conjunction with FIG6 , a schematic diagram of the process of another image deblurring method provided by the present application is illustrated. The method can be executed by the processing module shown in FIG5 . As shown in FIG6 , the method includes:

[0137] S601: Acquire a first image captured by a first camera.

[0138] The description of this step can refer to the description of S201 in the embodiment shown in FIG2 , and will not be repeated here.

[0139] S602 : Acquire first event data generated by an event camera within an exposure time of a first camera capturing a first image.

[0140] In this embodiment, the first event data generated by the event camera within the exposure time of the first camera shooting the first image is also referred to as the first event data corresponding to the first image.

[0141] Since the data generated by the event camera is usually in the form of a sequence, in this embodiment, the first event data is also referred to as first sequence data.

[0142] It should be noted that, in this embodiment, there is no sequential execution relationship between S601 and S602.

[0143] S603: Obtain fuzzy information of the first image based on the first image and the first event data using a fuzzy feature extractor.

[0144] In this embodiment, the fuzzy feature extractor is used to obtain fuzzy information of the image based on the image and event data corresponding to the image.

[0145] In this embodiment, after the first image and the first event data are acquired, the fuzzy information of the first image is obtained through a fuzzy feature extractor.

[0146] Below, several methods for obtaining fuzzy feature extractors are described:

[0147] The first implementation includes:

[0148] 1) Construct multiple training samples.

[0149] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the sorted sample pair, and a label.

[0150] Among them, the event data corresponding to the two images in the sorted sample pair are the same.

[0151] The label indicates the image-level blur ranking between the pair of images with different blur levels. For details, please refer to the description in the embodiment shown in FIG2 , which will not be repeated here.

[0152] 2) Train the neural network through multiple training samples.

[0153] For example, the neural network includes a feature extractor and a global sorter. The feature extractor is used to receive an image, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the global sorter. The global sorter is used to receive the cascaded features of each image in the sorted sample pair sent by the feature extractor and output the image-level fuzzy degree ranking between the sorted sample pairs.

[0154] 3) Use the feature extractor in the trained neural network as a fuzzy feature extractor.

[0155] At this time, the fuzzy information of the first image can be obtained by inputting the first image and the first event data into the fuzzy feature extractor.

[0156] The second implementation includes:

[0157] 1) Construct multiple training samples.

[0158] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the sorted sample pair, and a label.

[0159] The label indicates the pixel-level blur ranking between the pair of images with different blur levels. Detailed descriptions can be made in the embodiment shown in FIG2 , which will not be repeated here.

[0160] 2) Train the neural network through multiple training samples.

[0161] For example, the neural network includes a feature extractor and a local sorter, where the feature extractor is used to receive an image, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the local sorter, which is used to receive the cascaded features of each image in the sorted sample pair sent by the feature extractor and output the pixel-level fuzzy degree ranking between the sorted sample pairs.

[0162] 3) Use the feature extractor in the trained neural network as a fuzzy feature extractor.

[0163] At this time, the fuzzy information of the first image can be obtained by inputting the first image and the first event data into the fuzzy feature extractor.

[0164] The third implementation is a combination of the first and second implementations, including:

[0165] In implementation, a method for obtaining a fuzzy feature extractor for extracting image-level fuzzy information and pixel-level fuzzy information includes:

[0166] 1) Construct multiple training samples.

[0167] Specifically, each of the multiple training samples includes a pair of images with different blur levels (also called a sorted sample pair), event data corresponding to the sorted sample pair, and a label.

[0168] The labels indicate the image-level blur ranking and the pixel-level blur ranking between the pair of images with different blur levels.

[0169] 2) Train the neural network through multiple training samples.

[0170] For example, as shown in Figure 7, the neural network includes a feature extractor, a global sorter and a local sorter. The feature extractor is used to receive images, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the global sorter and the local sorter. The global sorter is used to receive the cascaded features of each image in the sorted sample pairs sent by the feature extractor and output the image-level fuzzy degree ranking between the sorted sample pairs. The local sorter is used to receive the cascaded features of each image in the sorted sample pairs sent by the feature extractor and output the image-level fuzzy degree ranking between the sorted sample pairs and the pixel-level fuzzy program ranking.

[0171] 3) Use the feature extractor in the trained neural network as a fuzzy feature extractor.

[0172] At this time, the fuzzy information of the first image can be obtained by inputting the first image and the first event data into the feature extractor.

[0173] In this embodiment, no matter it is the first implementation manner, the second implementation manner or the third implementation manner, the neural network trained in these three manners is also referred to as a second neural network.

[0174] S604: Obtain a second image after deblurring the first image based on the first image and blur information of the first image using a first neural network, wherein the first neural network is used to obtain the image after deblurring the image based on the image and the blur information of the image.

[0175] Specifically, in this embodiment, the first neural network is a trained neural network for obtaining a deblurred image based on an image and blur information of the image, wherein the blur information of the image is obtained based on the image and event data corresponding to the image.

[0176] As an example, a method of obtaining the first neural network is described.

[0177] 1) Construct multiple training samples.

[0178] Specifically, each of the multiple training samples includes an image, event data corresponding to the image, blur information of the image, and a label, wherein the label indicates a clear image corresponding to the image.

[0179] 2) Building a neural network model and training it through multiple training samples, and determining the trained neural network as the first neural network.

[0180] In this embodiment, when building a neural network model, the input to at least one feature extractor included in the neural network model includes not only the image feature data input by the previous feature extractor, but also the fuzzy information obtained after the input image of the neural network model is passed through the fuzzy feature extractor. In other words, the fuzzy information of the image is embedded in the image features.

[0181] It can be seen that in this embodiment, when the first image is to be deblurred, the blur information of the first image is first obtained based on the first image and the first event data corresponding to the first image, that is, the first event data collected by the event camera is integrated when obtaining the blur information of the first image. Therefore, the deblurring effect when deblurring the first image can be further improved, thereby improving the quality of the image after deblurring.

[0182] Optionally, it is possible that the event camera collects multiple sets of sequence data within the exposure time of the image. In this case, the label indicating the clear image corresponding to the image in the embodiment shown in FIG6 can also be replaced with an indication of the corresponding clear image sequence (i.e., multiple clear images). Accordingly, when the first event data is multiple sets of sequence data, after the first image, the first event data, and the blur information of the first image are input into the trained first neural network, the first neural network can output a clear image sequence (i.e., output multiple clear images).

[0183] Furthermore, in this embodiment, before using the first neural network to deblur the image, it can also include: obtaining j images of different resolutions based on the first image, the j images of different resolutions include the first image of the first resolution and j-1 images of different resolutions after reducing the first image; using a fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images of different resolutions and the first event data; accordingly, using the first neural network to deblur the first image based on the first image and the fuzzy information of the first image to obtain the second image, including: the first neural network deblurring the first image based on the first image, the fuzzy information of the first image, the j-1 images of different resolutions, and the fuzzy information of each of the j-1 images of different resolutions to obtain the second image.

[0184] Below, taking j as 3, taking the case where the event camera collects multiple sets of sequence data within the exposure time of the first camera, and when the first neural network is used for deblurring, the first neural network can output a clear image sequence as an example, combined with Figure 8, a complete image deblurring method is introduced.

[0185] Specifically, the method includes:

[0186] Step 1: Acquire a first image (for example, represented as B1) of a first resolution and first event data corresponding to the first image.

[0187] Step 2: down-sample the first image B1 by a factor of 2 to obtain an image B2 of a second resolution, and down-sample the image B2 by a factor of 2 to obtain an image B3 of a third resolution.

[0188] Through step 2, three images with different scales will be obtained.

[0189] Step 3: Use the feature extractor in the trained blur estimation network to obtain the blur information corresponding to image B1, image B2, and image B3 respectively.

[0190] The feature extractor in the fuzzy estimation network is used to obtain the fuzzy information of the image, that is, the fuzzy feature extractor. Specifically, during implementation, it may include:

[0191] 1) Construct at least one training sample pair, where each training sample includes a sorted sample pair, event data corresponding to the sorted sample pair, and a label.

[0192] More specifically, based on the assumption that the longer the exposure time, the greater the image blur, that is, the more clear frames are used to synthesize blurred frames, the blurrier the image, and image-based downsampling can reduce the blur of the image, when constructing the sorting sample pairs, the sorting sample pairs can include two types, namely, sorting sample pairs with different exposure times and sorting sample pairs with different resolutions.

[0193] For example, for sorted sample pairs with different exposure times, a single frame can be selected as the center frame, and then T frames around it can be selected to synthesize a blurred frame (the value of T is not restricted, for example, one of {3, 5, 7, 9, 11}). The two blurred samples generated by selecting different T values ​​constitute the sorted sample pair with different exposure times. Optionally, the selected frames can be interpolated using the "SuperSlomo" method and then all frames can be averaged to avoid ghosting in the image.

[0194] For example, for sorted sample pairs with different resolutions, in specific implementation, T frames can be randomly selected to generate blurred frames, and then they are downsampled S times using the bicubic interpolation method (the value of S is not restricted, for example, any value in {1, 2, 4}) to generate two blurred samples with different resolutions, which are sorted sample pairs with different resolutions.

[0195] The generation of event data corresponding to the sorted sample pairs is simulated using, for example, a V2E method.

[0196] 2) Training a fuzzy estimation network using the at least one training sample pair.

[0197] For example, as shown in Figure 8, the neural network includes a feature extractor, a global sorter and a local sorter. The feature extractor is used to receive images, extract image features, receive event data corresponding to the image, extract event data features, and cascade the extracted image features and the extracted event data features and then send them to the global sorter and local sorter. The global sorter is used to receive the cascaded features of each image in the sorted sample pairs sent by the feature extractor and output the image-level blur ranking between the sorted sample pairs. The local sorter is used to receive the cascaded features of each image in the blurred image pairs sent by the feature extractor and output the pixel-level blur ranking between the sorted sample pairs. Then, through the image-level blur ranking output by the global sorter and the pixel-level ranking output by the local sorter and the optimized neural network of the label, that is, using the global sorting loss and the local sorting loss as supervision, when the conditions are met, such as when all training sample pairs are trained or when the preset number of iterations is reached, a trained blur estimation network is obtained.

[0198] It should be noted that this embodiment does not limit the specific forms of the feature extractor, global sorter, and local sorter in the fuzzy estimation network. For example, the feature extractor includes four residual modules, the global sorter includes one convolutional layer, one global average pooling layer, and one fully connected layer. The local sorter includes one convolutional layer and one sigmoid activation function.

[0199] It should be understood that due to the size mismatch between the sorted sample pairs with different resolutions, it is impossible to calculate the local loss sorting. In this case, during training, optimization can be performed only through the global sorting network without passing through the local sorting network.

[0200] During testing, we obtain the information output by the feature extractor in the blur estimation network for subsequent deblurring tasks.

[0201] Step 4: Use the trained clear sequence recovery network (i.e., the first neural network) to obtain a clear sequence after deblurring the first image. The clear sequence recovery network is used for deblurring processing and outputs a clear image sequence after deblurring.

[0202] Exemplarily, as shown in Figure 8, the clear sequence recovery network in this embodiment includes 3 layers, the first layer includes a convolutional layer, a feature learning module 1 (i.e., the learning module 1 in the figure), a feature aggregation module 1 (i.e., the aggregation module 1 in the figure), a residual module and a convolutional layer, the second layer includes a convolutional layer, a feature learning module 2 (i.e., the learning module 2 in the figure), a feature aggregation module 2 (i.e., the aggregation module 2 in the figure), a residual module and a convolutional layer, and the third layer includes a convolutional layer, a feature learning module 1 (i.e., the learning module 3 in the figure), a residual module and a convolutional layer.

[0203] Specifically, it includes:

[0204] 1) Image B1 is passed through a convolutional layer to extract image features, and then passed through a feature learning module 1 to obtain the modulation features of image B1 (i.e., the features output by the feature learning module 1). The input of the feature learning module 1 also includes the fuzzy information of image B1.

[0205] 2) Image B2 is processed through a convolutional layer to extract image features. This is then concatenated with the features output by feature learning module 1 and passed through feature learning module 2 to obtain the modulation features of image B2 (i.e., the features output by feature learning module 2). The input of feature learning module 2 also includes the blur information of image B2.

[0206] 3) Image B3 is processed through a convolutional layer to extract image features, which are then concatenated with the features output by feature learning module 2 and then processed through feature learning module 3 to obtain the modulation features of image B3 (i.e., the features output by feature learning module 3). The input of feature learning module 3 also includes the blur information of image B3.

[0207] The input to each of the above feature learning modules not only includes the image of the corresponding resolution, but also includes the fuzzy information of the corresponding resolved image. In other words, through this method, the fuzzy information can be embedded into the image features, and the fuzzy information can be used to guide the feature extraction process.

[0208] It should be noted that this embodiment does not limit the specific form of the feature learning module. For example, the feature learning module consists of two convolutional layers and one residual module.

[0209] 4) The modulation features of images B1, B2, and B3 are input into the feature aggregation module 1 to obtain the aggregated features of image B1 (i.e., the features output by the feature aggregation module 1). The input of the feature aggregation module 1 also includes the fuzzy information of image B1.

[0210] 5) The modulation features of images B1, B2, and B3 are input to the feature aggregation module 2 to obtain the aggregated features of image B2 (i.e., the features output by the feature aggregation module 2). The input of the feature aggregation module 2 also includes the fuzzy information of image B2.

[0211] 6) The modulation features of image B3 are used as the aggregate features of image B3.

[0212] In addition to the modulation features output by the previous layer of feature aggregation modules, the input of each of the above feature aggregation modules also includes the fuzzy information of the corresponding resolved image. In this way, the fuzzy information can be embedded into the feature aggregation module, and the fuzzy information can be used to guide the feature aggregation process.

[0213] It should be noted that this embodiment does not limit the specific form of the feature aggregation module. For example, the feature aggregation module includes four convolutional layers and one softmax function. The blurred feature information of images B1, B2, and B3 is passed through the convolutional layers, and then a position-by-position softmax operation is performed to generate aggregation weights. The modulated features of images B1, B2, and B3 are concatenated and multiplied element-by-element by the aggregation weights through the convolutional layer to obtain the aggregated features.

[0214] 7) The aggregated features of image B3 are added to image B3 after passing through the residual module and convolutional layer in layer 3, resulting in a clear sequence with the same scale as image B3. The output of the residual module in layer 3 is upsampled for use in the next step.

[0215] 8) The aggregated features of image B2 are concatenated with the upsampled results from step 7), passed through the residual modulus and convolution layers of layer 2, and then added to image B2 to produce a clear sequence with the same scale as image B2. The residual modulus output of layer 2 is upsampled for use in the next step.

[0216] 9) The aggregated features of image B1 are concatenated with the upsampling result in step 8), and after passing through the residual modulus and convolution layer of the first layer, they are added to the image B1 to obtain a clear sequence with the same scale as image B1.

[0217] The following describes a method for training a clear sequence restoration network. For example, a second training sample is obtained, which includes three images, blur information for each of the three images, and a label for each of the three images, where the label for each image is a clear sequence image after deblurring. Based on the second training sample, the neural network structure shown in Figure 7 is trained. For example, L1 loss is used as supervision for clear sequences of various resolutions, and stochastic gradient descent is used to iteratively update model parameters to obtain a trained clear sequence restoration network.

[0218] The image processing method of an embodiment of the present application is described in detail above in conjunction with Figures 2 to 8 . The image processing device provided by the present application will be described in detail below in conjunction with Figures 9 and 10 .

[0219] FIG9 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application. Specifically, as shown in FIG9 , the apparatus 900 includes: an acquisition module 901 and a processing module 902 .

[0220] Specifically, 901, an acquisition module 901 is used to acquire a first image; a processing module 902 is used to determine blur information of the first image, where the blur information indicates a blurred area in the first image; the processing module 902 is also used to use a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, where the first neural network is used to deblur the image based on the image and the blur information of the image.

[0221] In an example, the acquisition module 901 is used to execute the step of acquiring the first image in S201 in the embodiment of FIG. 2 .

[0222] In another example, the processing module 902 is used to execute the step of using a first neural network to deblur the first image based on the first image and blur information of the first image to obtain a second image in S203 in the embodiment of Figure 2.

[0223] In one possible implementation, the acquisition module 901 is further used to: acquire first event data generated by the event camera within a first time length, where the first time length is the exposure time length used by the first camera to capture the first image; the processing module 902 is further used to: determine the blur information of the first image based on the first image and the first event data.

[0224] In one possible implementation, when the processing module 902 determines the blur information of the first image based on the first image and the first event data, the processing module 902 is specifically used to: input the first image and the first event data into a fuzzy feature extractor, and then determine the features output by the fuzzy feature extractor as the blur information of the first image; wherein the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data, and the first training data includes a pair of images with different blur levels, event data corresponding to a pair of images with different blur levels, and labels, the event data corresponding to a pair of images with different blur levels are the same, and the label is used to indicate the image-level blur level ranking and / or pixel-level blur level ranking between a pair of images with different blur levels.

[0225] In one possible implementation, the processing module 902 is further configured to: obtain j images of different resolutions based on the first image, where the j images of different resolutions include the first image of the first resolution and j-1 images of different resolutions that are obtained by reducing the first image; and obtain fuzzy information of each image in the j-1 images of different resolutions based on the first event data using a fuzzy feature extractor;

[0226] A first neural network is used to deblur the first image based on the first image, the blur information of the first image, j-1 images with different resolutions, and the blur information of each of the j-1 images with different resolutions to obtain a second image.

[0227] In one possible implementation, the acquisition module 901 is further used to: acquire second training data, the second training data including a third image of a first resolution and blur information of the third image, j-1 images of different resolutions after reducing the third image and their respective blur information, the third image and respective labels of j-1 images of different resolutions after reducing the third image, wherein the label of each image is used to indicate the image of each image after deblurring; the processing module 902 is further used to train a neural network based on the second training data to obtain a first neural network.

[0228] Optionally, the image processing device described in Figure 9 can be deployed in a general-purpose computer, a server, or as an embedded module. Among them, the image processing device described in this application is installed in a general-purpose computer, which can process the collected RGB and event data to obtain a clear image or image sequence and save it locally. The image processing device described in this application is installed in the server and runs as a service in the server. Other devices can restore the image by sending requests to call the computing resources of the server and the image processing device described in this application. When it is an embedded module, it may exist in devices such as smartphones, tablet computers, vehicle-mounted mobile devices, monitoring systems, and VR / AR glasses.

[0229] Figure 10 is a schematic structural diagram of an image processing device provided in another embodiment of the present application. The device shown in Figure 10 can be used to execute the method described in any of the above embodiments.

[0230] As shown in Figure 10, the apparatus 1000 of this embodiment includes a memory 1001 and a processor 1002. Optionally, the apparatus 1000 further includes a communication interface 1003 and a bus 1004. The memory 1001, the processor 1002, and the communication interface 1003 are connected to each other via the bus 1004.

[0231] The memory 1001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1001 may store a program. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 is configured to perform the steps of the method shown in Figures 2 to 8.

[0232] The processor 1002 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs to implement the methods shown in Figures 2 to 8 of the present application.

[0233] The processor 1002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method of FIG. 2 to FIG. 8 of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 1002 or by instructions in the form of software.

[0234] The processor 1002 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor 1002 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor.

[0235] The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001 and, in combination with its hardware, completes the functions required to be performed by the units included in the device of the present application. For example, the various steps / functions of the embodiments shown in Figures 2 to 8 can be executed.

[0236] The communication interface 1003 may use, but is not limited to, a transceiver or other transceiver device to implement communication between the apparatus 1000 and other devices or a communication network.

[0237] The bus 1004 may include a path for transmitting information between various components of the device 1000 (eg, the memory 1001 , the processor 1002 , and the communication interface 1003 ).

[0238] It should be understood that the apparatus 1000 shown in the embodiment of the present application can be an electronic device, or a chip configured in an electronic device. The apparatus 1000 can be deployed in a terminal device, or can also be deployed in a network device.

[0239] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0240] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0241] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0242] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0243] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0244] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0245] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0246] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0247] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0248] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0249] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: acquiring a first image; determining blur information of the first image, the blur information indicating a blurry area in the first image; A first neural network is used to deblur the first image based on the first image and blur information of the first image to obtain a second image, wherein the first neural network is used to deblur the image based on the image and blur information of the image.

2. The method according to claim 1, characterized in that: The method further comprises: Acquire first event data generated by an event camera within a first time period, where the first time period is an exposure time period used by a first camera to capture the first image; The determining the blur information of the first image includes: Blur information of the first image is determined based on the first image and the first event data.

3. The method according to claim 2, characterized in that The determining the blur information of the first image based on the first image and the first event data comprises: After the first image and the first event data are input into a fuzzy feature extractor, the feature output by the fuzzy feature extractor is determined as fuzzy information of the first image; Among them, the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data. The first training data includes a pair of images with different blur levels, event data corresponding to the pair of images with different blur levels, and labels. The event data corresponding to the pair of images with different blur levels are the same, and the label is used to indicate the image-level blur level sorting and / or pixel-level blur level sorting between the pair of images with different blur levels.

4. The method according to claim 3, characterized in that The pair of images with different blur levels have different exposure durations, or the pair of images with different blur levels have different resolutions.

5. The method according to claim 4, characterized in that The method further comprises: Obtain j images with different resolutions based on the first image, wherein the j images with different resolutions include the first image with a first resolution and j-1 images with different resolutions that are obtained by reducing the first image; Using the fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images with different resolutions and the first event data; The step of using a first neural network to deblur the first image based on the first image and blur information of the first image to obtain a second image includes: A first neural network is used to deblur the first image based on the first image, the blur information of the first image, the j-1 images with different resolutions, and the blur information of each of the j-1 images with different resolutions to obtain a second image.

6. The method according to claim 5, characterized in that j is 3, and the j-1 images with different resolutions include an image obtained by downsampling the first image by a factor of 2 and an image obtained by further downsampling the image obtained by downsampling the first image by a factor of 2 by a factor of 2.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: Acquire second training data, wherein the second training data includes a third image of a first resolution and blur information of the third image, j-1 images with different resolutions reduced from the third image and their respective blur information, the third image and labels of the j-1 images with different resolutions reduced from the third image, wherein the label of each image is used to indicate the image after each image is deblurred; A neural network is trained based on the second training data to obtain the first neural network.

8. The method according to any one of claims 1 to 7, characterized in that The first camera is an RGB camera.

9. The method according to any one of claims 1 to 8, characterized in that The first neural network is a convolutional neural network.

10. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image; A processing module, configured to determine blur information of the first image, wherein the blur information indicates a blur area in the first image. domain; The processing module is further used to use a first neural network to deblur the first image based on the first image and the blur information of the first image to obtain a second image, and the first neural network is used to deblur the image based on the image and the blur information of the image.

11. The device according to claim 10, characterized in that: The acquisition module is also used for: Acquire first event data generated by an event camera within a first time period, where the first time period is an exposure time period used by a first camera to capture the first image; The processing module is also used for: Blur information of the first image is determined based on the first image and the first event data.

12. The device according to claim 11, characterized in that When the processing module determines the blur information of the first image based on the first image and the first event data, the processing module is specifically configured to: After the first image and the first event data are input into a fuzzy feature extractor, the feature output by the fuzzy feature extractor is determined as fuzzy information of the first image; Among them, the fuzzy feature extractor is the first i layers in the second neural network, and the second neural network is trained based on the first training data. The first training data includes a pair of images with different blur levels, event data corresponding to the pair of images with different blur levels, and labels. The event data corresponding to the pair of images with different blur levels are the same, and the label is used to indicate the image-level blur level sorting and / or pixel-level blur level sorting between the pair of images with different blur levels.

13. The device according to claim 12, characterized in that The pair of images with different blur levels have different exposure durations, or the pair of images with different blur levels have different resolutions.

14. The device according to claim 13, characterized in that The processing module is also used for: Obtain j images with different resolutions based on the first image, wherein the j images with different resolutions include the first image with a first resolution and j-1 images with different resolutions that are obtained by reducing the first image; Using the fuzzy feature extractor to obtain fuzzy information of each image based on each image in the j-1 images with different resolutions and the first event data; A first neural network is used to deblur the first image based on the first image, the blur information of the first image, the j-1 images with different resolutions, and the blur information of each of the j-1 images with different resolutions to obtain a second image.

15. The device according to claim 14, characterized in that j is 3, and the j-1 images with different resolutions include an image obtained by downsampling the first image by a factor of 2 and an image obtained by further downsampling the image obtained by downsampling the first image by a factor of 2 by a factor of 2.

16. The device according to claim 14 or 15, characterized in that The acquisition module is also used for: Acquire second training data, wherein the second training data includes a third image of a first resolution and blur information of the third image, j-1 images with different resolutions reduced from the third image and their respective blur information, the third image and labels of the j-1 images with different resolutions reduced from the third image, wherein the label of each image is used to indicate the image after each image is deblurred; The processing module is further used to train a neural network based on the second training data to obtain the first neural network.

17. The device according to any one of claims 10 to 16, characterized in that The first camera is an RGB camera.

18. The device according to any one of claims 10 to 16, characterized in that The first neural network is a convolutional neural network.

19. An automatic driving device, characterized in that: Used to execute the image processing method according to any one of claims 1 to 9.

20. An automatic driving system, characterized in that: It includes the image processing device as described in any one of claims 10 to 18 or the automatic driving device as described in claim 19.

21. A vehicle, characterized in that: Including an automatic driving system as described in claim 20.

22. A server, characterized in that: The invention comprises the image processing device as claimed in any one of claims 10 to 18.

23. An image processing device, characterized in that: include: Memory and processor; The memory is used to store program instructions; The processor is configured to call the program instructions in the memory to execute the method according to any one of claims 1 to 9.

24. A computer readable medium, characterized in that The computer-readable medium stores a program code for computer execution, the program code including instructions for executing the method according to any one of claims 1 to 9.

25. A computer program product, comprising computer program code, characterized in that: When the computer program code is executed on a computer, the computer is caused to implement the method according to claims 1 to 9.

Images