Image processing method and device, equipment and medium

Abstract

The invention discloses an image processing method and device, equipment and a medium, and belongs to the technical field of electronic equipment. The image processing method comprises the following steps: acquiring a first image and event information corresponding to the first image obtained by using an event camera; determining a blurred region from the first image according to the event information; performing deblurring processing on the second image included in the blurred region to obtain a third image; and generating a fourth image according to the third image and the first image.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an image processing method, device, equipment and medium. Background Art

[0002] Motion blur is a common image quality issue in the image acquisition process. It is mainly caused by the relative motion between the imaging plane and the subject during camera exposure, such as hand-held camera shaking or shooting moving objects.

[0003] In order to improve the image quality, the image needs to be deblurred to obtain an image with higher image quality.

[0004] In the related art, when deblurring an image, the deblurring is usually performed on the entire image. However, deblurring the entire image will occupy a high computing resource. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, apparatus, device and medium, which can solve the problem that image deblurring processing occupies high computing resources.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Acquire a first image and event information corresponding to the first image obtained by using an event camera;

[0008] Determine a blurred area from the first image according to the event information;

[0009] Deblurring the second image included in the blurred area to obtain a third image;

[0010] A fourth image is generated based on the third image and the first image.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0012] An acquisition module, used to acquire a first image and event information corresponding to the first image obtained by using an event camera;

[0013] A determination module, used for determining a blurred area from the first image according to the event information;

[0014] A deblurring module, used for performing a deblurring process on the second image included in the blurred area to obtain a third image;

[0015] The generating module is used to generate a fourth image according to the third image and the first image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the image processing method provided in the embodiment of the present application are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the image processing method provided in the embodiment of the present application are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the image processing method provided in the embodiment of the present application.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the image processing method provided in the embodiment of the present application.

[0020] In the embodiment of the present application, the first image and the event information corresponding to the first image obtained by using the event camera are obtained; the blurred area is determined from the first image according to the event information; the second image included in the blurred area is deblurred to obtain a third image; and the fourth image is generated according to the third image and the first image. In this way, only the blurred area in the first image is deblurred, and the entire first image is not deblurred, which can save computing resources and improve image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of an image processing method provided in an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of determining a fuzzy area provided in an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of image overlap provided in an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The image processing method, apparatus, device and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0030] Figure 1 : is a flowchart of an image processing method provided in an embodiment of the present application. The image processing method may include:

[0031] Step 101: Acquire a first image and event information corresponding to the first image obtained by using an event camera;

[0032] In some possible implementations of the embodiments of the present application, an event camera (Event-based camera) is a new type of visual sensor that works differently from a traditional camera. Instead of capturing complete image frames at a fixed frame rate, it asynchronously generates events for brightness changes of each pixel. Event information can be expressed as [x, y, t, p], where x is the horizontal coordinate component of the pixel point whose brightness changes, y is the vertical coordinate component of the pixel point whose brightness changes, t is the time of the brightness change, and p is the polarity. When p is positive, it indicates that a positive event has occurred, that is, the brightness of the pixel point has increased, and when p is negative, it indicates that a negative event has occurred, that is, the brightness of the pixel point has decreased. The event information [x, y, t, p] indicates that the brightness of the pixel point with coordinates (x, y) has changed at time t.

[0033] Step 102: determining a blurred area from the first image according to the event information;

[0034] In some possible implementations of the embodiments of the present application, step 102 may include: using a sliding window to determine the number of events in the area corresponding to the sliding window in the first image; when the number of events is greater than or equal to a first threshold, determining the area corresponding to the sliding window as a blurred area.

[0035] In some possible implementations of the embodiments of the present application, a two-dimensional matrix with the same size as the first image may be created. Based on the coordinates and time in the event information, the events occurring within the exposure time corresponding to the shooting of the first image are screened out, and mapped into the two-dimensional matrix according to the coordinates. When an event occurs at a certain coordinate position, the value at the coordinate in the two-dimensional matrix is ​​increased by 1.

[0036] In some possible implementations of the embodiments of the present application, when using a sliding window to determine the number of events in the area corresponding to the sliding window in the first image, the size of the window, the sliding trajectory and rules of the window can be preset so that the window follows the preset sliding trajectory and rules. For example, starting from the upper left corner of the two-dimensional matrix, move horizontally from left to right at a certain interval, wherein the interval is less than or equal to the width of the window, when the window moves horizontally to the rightmost boundary of the two-dimensional matrix, move the window vertically downward by a certain distance, wherein the distance is less than or equal to the height of the window, and then move the window to the far left, and move horizontally from left to right again at a certain interval, and so on, to complete the traversal of the two-dimensional matrix from left to right and from top to bottom. For another example, taking the upper left corner of the two-dimensional matrix as the starting position, move from top to bottom in the vertical direction at a certain interval, where the interval is less than or equal to the height of the window. When the window moves to the bottom boundary of the two-dimensional matrix in the vertical direction, move the window to the right in the horizontal direction by a certain distance, where the distance is less than or equal to the width of the window. Then move the window to the top, and move from top to bottom in the vertical direction again at a certain interval, and so on, to complete the traversal of the two-dimensional matrix from top to bottom and from left to right.

[0037] For each movement of the window, the number of events in the area in the two-dimensional matrix corresponding to the window is counted, and the number of events is the number of events in the area corresponding to the sliding window in the first image; when the number of events is greater than a first threshold, the area corresponding to the sliding window in the first image is used as a blurred area.

[0038] In some possible implementations of the embodiments of the present application, motion blur is the displacement of an object at the moment of exposure, which causes a streak in the image in the direction of the object's movement and produces a blurred area. This blurred area usually has more events per unit time. Therefore, the blurred area can be identified through a sliding window, that is, the area where the blur needs to be removed.

[0039] For example, Figure 2 As shown, Figure 2It is a schematic diagram of determining a fuzzy area provided in an embodiment of the present application.

[0040] exist Figure 2 In the example, as the window slides, the fuzzy area is gradually determined. When the number of events in the area corresponding to the window does not reach the first threshold, the area corresponding to the window will not be determined as a fuzzy area. When the number of events in the area corresponding to the window reaches the first threshold, the area corresponding to the window is determined as a fuzzy area. Figure 2 In the figure, the gray area indicates that the area is a fuzzy area.

[0041] In some possible implementations of the embodiments of the present application, the first threshold can be set according to actual needs.

[0042] In some possible implementations of the embodiments of the present application, the first threshold value may be dynamically adjusted according to the current energy consumption and / or remaining power of the electronic device to adjust the sensitivity of the blurred area detection. For example, when the remaining power of the electronic device is low, the first threshold value may be increased while ensuring the basic quality requirements of the image to reduce the area that needs to be deblurred, thereby saving power of the electronic device and increasing the battery life of the electronic device.

[0043] Step 103: Deblurring the second image included in the blurred area to obtain a third image;

[0044] In some possible implementations of the embodiments of the present application, the second image included in the blurred area can be input into a pre-trained image deblurring model, and the image deblurring model outputs a third image, which is a deblurred image corresponding to the second image.

[0045] In some possible implementations of the embodiments of the present application, the second image included in the blurred area and the event information corresponding to the second image can also be input into the image deblurring model, and the image deblurring model outputs a third image, which is a deblurred image corresponding to the second image. Among them, the event information is mapped to multiple channels according to the specific time interval of each timestamp, and then converted into a three-dimensional tensor. Inside the image deblurring model, the three-dimensional tensor and the second image establish a close connection through the mutual attention mechanism, so that the two can guide and supplement each other. For example, the dynamic change characteristics in the event information can guide the image deblurring model to pay attention to the corresponding blurred part in the second image, and the texture, color and other information of the second image can assist the event information to play a better role. After the two fully interact through the mutual attention mechanism, they enter the convolutional neural network of the image deblurring model together. The convolutional neural network uses its powerful feature extraction and transformation capabilities to process the blurred second image, remove the blurred part in the second image, and obtain the third image. In this process, the convolutional neural network accurately identifies the location and degree of blurring based on the rich information obtained from the mutual attention mechanism, and performs targeted restoration and enhancement operations to achieve effective removal of blur.

[0046] Step 104: Generate a fourth image according to the third image and the first image.

[0047] In some possible implementations of the embodiments of the present application, the second image included in the blurred area in the first image may be replaced by the third image to obtain a fourth image.

[0048] In the embodiment of the present application, the first image and the event information corresponding to the first image obtained by using the event camera are obtained; the blurred area is determined from the first image according to the event information; the second image included in the blurred area is deblurred to obtain the third image; and the fourth image is generated according to the third image and the first image. In this way, only the blurred area in the first image is deblurred, and the entire first image is not deblurred, which can save computing resources and improve image processing efficiency.

[0049] In some possible implementations of the embodiments of the present application, the size of the second image may exceed the input size of the image deblurring model. Based on this, step 103 may include: dividing the second image into multiple sub-images; inputting each of the multiple sub-images into the image deblurring processing model respectively to obtain a deblurred image corresponding to each sub-image, wherein the image deblurring processing model is used to deblur the image, and the third image includes the deblurred images corresponding to the multiple sub-images.

[0050] In some possible implementations of the embodiments of the present application, the second image can be divided into multiple sub-images according to the input size of the image deblurring model, so that the image deblurring model can smoothly receive and process the sub-images, thereby ensuring the accuracy of the deblurring processing.

[0051] In some possible implementations of the embodiments of the present application, two adjacent sub-images partially overlap.

[0052] In some possible implementations of the embodiments of the present application, the overlap ratio of two adjacent sub-images may be preset, for example, 50%; or the overlap width of two adjacent sub-images in the horizontal direction and the overlap height of two adjacent sub-images in the vertical direction may be preset. When dividing sub-images, the sub-images are divided according to the preset overlap ratio of two adjacent sub-images, or, when dividing sub-images, in the horizontal direction, adjacent sub-images are divided according to the preset overlap width, and in the vertical direction, adjacent sub-images are divided according to the preset overlap height.

[0053] In some possible implementations of the embodiments of the present application, step 104 may include: for a first pixel point, determining the distance from the first pixel point to the first boundary and the second boundary respectively, wherein the first pixel point is any pixel point in the overlapping part of the first deblurred image and the second deblurred image, the first deblurred image and the second deblurred image are images obtained after deblurring two sub-images with overlapping parts, the first boundary is a boundary shared by the overlapping part and the first deblurred image but not shared with the second deblurred image, and the second boundary is a boundary shared by the overlapping part and the second deblurred image but not shared with the first deblurred image; according to the distance, determining the weight of the first pixel point relative to the first deblurred image and the second deblurred image respectively; and determining the third pixel value of the first pixel point according to the weight, the first pixel value of the first pixel point in the first deblurred image, and the second pixel value of the first pixel point in the second deblurred image.

[0054] For example, Figure 3 As shown, Figure 3 is a schematic diagram of image overlap provided in an embodiment of the present application. Figure 3In , the first deblurred image 301 and the second deblurred image 302 are two adjacent deblurred images; the overlapping portion of the first deblurred image 301 and the second deblurred image 302 is surrounded by boundary 31, boundary 32, boundary 33 and boundary 34; wherein boundary 31 is the portion where the upper boundary of the first deblurred image 301 and the upper boundary of the second deblurred image 302 overlap, boundary 32 is the portion where the lower boundary of the first deblurred image 301 and the lower boundary of the second deblurred image 302 overlap, boundary 33 is the left boundary of the second deblurred image 302, and boundary 34 is the right boundary of the first deblurred image 301. The boundaries shared by the overlapping portion and the first deblurred image 301 include boundary 31, boundary 32 and boundary 34; the boundaries shared by the overlapping portion and the second deblurred image 302 include boundary 31, boundary 32 and boundary 33. In Figure 3 In the example shown, boundary 34 is the boundary shared by the overlapping portion and the first deblurred image 301 but not shared by the second deblurred image 302, and boundary 33 is the boundary shared by the overlapping portion and the second deblurred image 302 but not shared by the first deblurred image 301. The first boundary is boundary 34, and the second boundary is boundary 33.

[0055] In some possible implementations of the embodiments of the present application, for the non-overlapping portion of two adjacent sub-images, the image included in the non-overlapping portion of the two adjacent sub-images in the first image can be replaced by an image obtained after deblurring the non-overlapping portion of the two adjacent sub-images.

[0056] As for the overlapping part of two adjacent sub-images, the overlapping part will correspond to two images obtained after deblurring processing, that is, the overlapping part will correspond to two deblurred images. At this time, the two deblurred images need to be fused to obtain a fused image, and the image included in the overlapping part of the two adjacent sub-images in the first image is replaced by the fused image.

[0057] In some possible implementations of the embodiments of the present application, determining the weights of the first pixel relative to the first deblurred image and the second deblurred image respectively based on the distance can include: calculating the sum of a first distance from the first pixel to the first boundary and a second distance from the first pixel to the second boundary to obtain a distance sum; using the ratio of the second distance to the distance sum as the first weight of the first pixel relative to the first deblurred image; using the ratio of the first distance to the distance sum as the second weight of the first pixel relative to the second deblurred image.

[0058] In some possible implementations of the embodiments of the present application, the weight of the first pixel relative to the first deblurred image can be determined according to the following formula (1), and the weight of the first pixel relative to the second deblurred image can be determined according to the following formula (2).

[0059]

[0060] In formulas (1) and (2), ω 1 is the weight of pixel A relative to the first deblurred image, ω 2 is the weight of pixel A relative to the second deblurred image, d (A,1) is the first distance, d (A,2) is the second distance. Pixel point A is any pixel point in the overlapping part of the first deblurred image and the second deblurred image.

[0061] In some possible implementations of the embodiments of the present application, when determining the third pixel value of the first pixel based on the weight, the first pixel value of the first pixel in the first deblurred image, and the second pixel value of the first pixel in the second deblurred image, the first pixel value and the second pixel value can be weightedly fused using two weights to obtain the third pixel value of the first pixel, wherein the two weights include the weight of the first pixel relative to the first deblurred image and the weight of the first pixel relative to the second deblurred image.

[0062] In some possible implementations of the embodiments of the present application, determining the third pixel value of the first pixel point based on the weight, the first pixel value of the first pixel point in the first deblurred image, and the second pixel value of the first pixel point in the second deblurred image can include: calculating the product of the first weight and the first pixel value to obtain the fourth pixel value; calculating the product of the second weight and the second pixel value to obtain the fifth pixel value; and taking the sum of the fourth pixel value and the fifth pixel value as the third pixel value.

[0063] In some possible implementations of the embodiments of the present application, the third pixel value of the first pixel point may be determined using the following formula (3):

[0064] v A =ω 1 v A1 +ω 2 v A2 (3)

[0065] In formula (3), v A is the third pixel value of pixel A, v A1 is the first pixel value of pixel A in the first deblurred image, v A2 is the second pixel value of pixel A in the second deblurred image, ω 1 is the weight of pixel A relative to the first deblurred image, ω 2 is the weight of pixel A relative to the second deblurred image, and pixel A is any pixel in the overlapping part of the first sub-image and the second sub-image.

[0066] In the embodiment of the present application, by fusing the pixel values ​​of each pixel in the overlapping part of two adjacent sub-images, the two adjacent sub-images can be smoothly transitioned, avoiding obvious boundaries or discontinuities in the image, and improving the image quality.

[0067] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0068] Figure 4 400 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device 400 may include:

[0069] An acquisition module 401 is used to acquire a first image and event information corresponding to the first image obtained by using an event camera;

[0070] A determination module 402, configured to determine a blurred area from the first image according to the event information;

[0071] A deblurring module 403, configured to perform a deblurring process on the second image included in the blurred area to obtain a third image;

[0072] The generating module 404 is configured to generate a fourth image according to the third image and the first image.

[0073] In the embodiment of the present application, the first image and the event information corresponding to the first image obtained by using the event camera are obtained; the blurred area is determined from the first image according to the event information; the second image included in the blurred area is deblurred to obtain a third image; and the fourth image is generated according to the third image and the first image. In this way, only the blurred area in the first image is deblurred, and the entire first image is not deblurred, which can save computing resources and improve image processing efficiency.

[0074] In some possible implementations of the embodiment of the present application, the determination module 402 is specifically used to:

[0075] Determining the number of events in a region of the first image corresponding to the sliding window using the sliding window;

[0076] When the number of events is greater than or equal to the first threshold, the area corresponding to the sliding window is determined as a fuzzy area.

[0077] In some possible implementations of the embodiments of the present application, the deblurring module 403 is specifically used for:

[0078] dividing the second image into a plurality of sub-images;

[0079] Each of the multiple sub-images is input into an image deblurring processing model to obtain a deblurred image corresponding to each sub-image, wherein the image deblurring processing model is used to perform deblurring processing on the image, and the third image includes the deblurred images corresponding to the multiple sub-images.

[0080] In the embodiment of the present application, by dividing the second image into multiple sub-images and performing deblurring processing on each sub-image, the image deblurring processing model can smoothly receive and process the sub-images, thereby ensuring the accuracy of the deblurring processing.

[0081] In some possible implementations of the embodiments of the present application, two adjacent sub-images partially overlap.

[0082] In some possible implementations of the embodiment of the present application, the generating module 404 includes:

[0083] A first determination submodule is used to determine, for a first pixel point, the distances from the first pixel point to the first boundary and the second boundary respectively, wherein the first pixel point is any pixel point in the overlapping portion of the first deblurred image and the second deblurred image, the first deblurred image and the second deblurred image are images obtained after deblurring two sub-images with overlapping portions, the first boundary is a boundary shared by the overlapping portion and the first deblurred image but not shared by the second deblurred image, and the second boundary is a boundary shared by the overlapping portion and the second deblurred image but not shared by the first deblurred image;

[0084] A second determination submodule, used to determine, according to the distance, the weights of the first pixel point relative to the first deblurred image and the second deblurred image respectively;

[0085] The third determination submodule is used to determine a third pixel value of the first pixel according to the weight, the first pixel value of the first pixel in the first deblurred image, and the second pixel value of the first pixel in the second deblurred image.

[0086] In some possible implementations of the embodiments of the present application, the second determining submodule is specifically used to:

[0087] Calculate the sum of a first distance from the first pixel to the first boundary and a second distance from the first pixel to the second boundary to obtain a distance sum;

[0088] Using the ratio of the second distance to the sum of the distances as a first weight of the first pixel relative to the first deblurred image;

[0089] The ratio of the first distance to the sum of the distances is used as a second weight of the first pixel relative to the second deblurred image.

[0090] In some possible implementations of the embodiments of the present application, the third determining submodule is specifically used to:

[0091] Calculate the product of the first weight and the first pixel value to obtain a fourth pixel value;

[0092] Calculate the product of the second weight and the second pixel value to obtain a fifth pixel value;

[0093] The sum of the fourth pixel value and the fifth pixel value is taken as the third pixel value.

[0094] In the embodiment of the present application, by fusing the pixel values ​​of each pixel in the overlapping part of two adjacent sub-images, the two adjacent sub-images can be smoothly transitioned, avoiding obvious boundaries or discontinuities in the image, and improving the image quality.

[0095] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0096] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0097] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the image processing method embodiment are not described here.

[0098] Alternatively, if Figure 5As shown, an embodiment of the present application further provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the image processing method embodiment provided in the embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0099] Figure 6 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0100] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0101] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0102] Among them, the processor 610 is used to: obtain the first image and event information corresponding to the first image obtained by using the event camera; determine the blurred area from the first image according to the event information; deblur the second image included in the blurred area to obtain a third image; and generate a fourth image according to the third image and the first image.

[0103] In the embodiment of the present application, the first image and the event information corresponding to the first image obtained by using the event camera are obtained; the blurred area is determined from the first image according to the event information; the second image included in the blurred area is deblurred to obtain a third image; and the fourth image is generated according to the third image and the first image. In this way, only the blurred area in the first image is deblurred, and the entire first image is not deblurred, which can save computing resources and improve image processing efficiency.

[0104] In some possible implementations of the embodiments of the present application, the processor 610 is specifically configured to:

[0105] Determining the number of events in a region of the first image corresponding to the sliding window using the sliding window;

[0106] When the number of events is greater than or equal to the first threshold, the area corresponding to the sliding window is determined as a fuzzy area.

[0107] In some possible implementations of the embodiments of the present application, the processor 610 is specifically configured to:

[0108] dividing the second image into a plurality of sub-images;

[0109] Each of the multiple sub-images is input into an image deblurring processing model to obtain a deblurred image corresponding to each sub-image, wherein the image deblurring processing model is used to deblur the image, and the third image includes the deblurred images corresponding to the multiple sub-images.

[0110] In the embodiment of the present application, by dividing the second image into multiple sub-images and performing deblurring processing on each sub-image, the image deblurring processing model can smoothly receive and process the sub-images, thereby ensuring the accuracy of the deblurring processing.

[0111] In some possible implementations of the embodiments of the present application, two adjacent sub-images partially overlap.

[0112] In some possible implementations of the embodiments of the present application, the processor 610 is specifically configured to:

[0113] For a first pixel point, determine the distances from the first pixel point to the first boundary and the second boundary respectively, wherein the first pixel point is any pixel point in the overlapping portion of the first deblurred image and the second deblurred image, the first deblurred image and the second deblurred image are images obtained after deblurring two sub-images with overlapping portions, the first boundary is a boundary shared by the overlapping portion and the first deblurred image but not shared by the second deblurred image, and the second boundary is a boundary shared by the overlapping portion and the second deblurred image but not shared by the first deblurred image;

[0114] Determine, according to the distance, the weight of the first pixel point relative to the first deblurred image and the second deblurred image respectively;

[0115] A third pixel value of the first pixel is determined according to the weight, a first pixel value of the first pixel in the first deblurred image, and a second pixel value of the first pixel in the second deblurred image.

[0116] In the embodiment of the present application, by fusing the pixel values ​​of each pixel in the overlapping part of two adjacent sub-images, the two adjacent sub-images can be smoothly transitioned, avoiding obvious boundaries or discontinuities in the image, and improving the image quality.

[0117] In some possible implementations of the embodiments of the present application, the processor 610 is specifically configured to:

[0118] Calculate the sum of a first distance from the first pixel to the first boundary and a second distance from the first pixel to the second boundary to obtain a distance sum;

[0119] Using the ratio of the second distance to the sum of the distances as a first weight of the first pixel relative to the first deblurred image;

[0120] The ratio of the first distance to the sum of the distances is used as a second weight of the first pixel relative to the second deblurred image.

[0121] In some possible implementations of the embodiments of the present application, the processor 610 is specifically configured to:

[0122] Calculate the product of the first weight and the first pixel value to obtain a fourth pixel value;

[0123] Calculate the product of the second weight and the second pixel value to obtain a fifth pixel value;

[0124] The sum of the fourth pixel value and the fifth pixel value is taken as the third pixel value.

[0125] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0126] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0127] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0128] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the image processing method embodiment provided in the embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0129] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, and examples of computer-readable storage media include non-transitory computer-readable media, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0130] An embodiment of the present application also provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the image processing method embodiment provided in the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0131] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0132] The embodiment of the present application also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the image processing method embodiment provided in the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0133] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0135] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image and event information corresponding to the first image obtained by using an event camera; Determining a blurred area from the first image according to the event information; Deblurring the second image included in the blurred area to obtain a third image; A fourth image is generated according to the third image and the first image.

2. The method according to claim 1, characterized in that The step of determining a blurred area from the first image according to the event information includes: Determining the number of events in an area of ​​the first image corresponding to the sliding window using a sliding window; When the number of events is greater than or equal to a first threshold, the area corresponding to the sliding window is determined as the fuzzy area.

3. The method according to claim 1, characterized in that The deblurring process is performed on the second image included in the blurred area to obtain a third image, comprising: dividing the second image into a plurality of sub-images; Each of the multiple sub-images is input into an image deblurring processing model to obtain a deblurred image corresponding to each sub-image, wherein the image deblurring processing model is used to deblur the image, and the third image includes the deblurred images corresponding to the multiple sub-images.

4. The method according to claim 3, characterized in that Two adjacent sub-images partially overlap.

5. The method according to claim 4, characterized in that Generating a fourth image according to the third image and the first image includes: For a first pixel point, determining distances from the first pixel point to a first boundary and a second boundary respectively, wherein the first pixel point is any pixel point in an overlapping portion of a first deblurred image and a second deblurred image, the first deblurred image and the second deblurred image are images obtained after deblurring two sub-images having an overlapping portion, the first boundary is a boundary shared by the overlapping portion and the first deblurred image but not shared by the second deblurred image, and the second boundary is a boundary shared by the overlapping portion and the second deblurred image but not shared by the first deblurred image; Determining, according to the distance, weights of the first pixel points relative to the first deblurred image and the second deblurred image respectively; A third pixel value of the first pixel is determined according to the weight, a first pixel value of the first pixel in the first deblurred image, and a second pixel value of the first pixel in the second deblurred image.

6. The method according to claim 5, characterized in that The determining, according to the distance, the weights of the first pixel points respectively relative to the first deblurred image and the second deblurred image comprises: Calculating the sum of a first distance from the first pixel to the first boundary and a second distance from the first pixel to the second boundary to obtain a distance sum; Using a ratio of the second distance to the sum of the distances as a first weight of the first pixel relative to the first deblurred image; The ratio of the first distance to the sum of the distances is used as a second weight of the first pixel relative to the second deblurred image.

7. The method according to claim 6, characterized in that The determining, according to the weight, a first pixel value of the first pixel in the first deblurred image, and a second pixel value of the first pixel in the second deblurred image, comprises: Calculate the product of the first weight and the first pixel value to obtain a fourth pixel value; Calculate the product of the second weight and the second pixel value to obtain a fifth pixel value; The sum of the fourth pixel value and the fifth pixel value is used as the third pixel value.

8. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a first image and event information corresponding to the first image obtained by using an event camera; A determination module, configured to determine a blurred area from the first image according to the event information; a deblurring module, configured to perform a deblurring process on the second image included in the fuzzy area to obtain a third image; A generating module is used to generate a fourth image according to the third image and the first image.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images