Image processing method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring a to-be-processed image and a reference image corresponding to the to-be-processed image; the resolution of the to-be-processed image is a first resolution, and the resolution of the reference image is a second resolution; the first resolution is different from the second resolution; based on the first resolution and the second resolution, sampling the to-be-processed image to obtain a sampled image; the resolution of the sampling image is a second resolution; fusing the sampling image and the reference image to obtain a fused image; and performing deblurring processing on the fused image to obtain a target image. By adopting the method, the accuracy of image processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, apparatus, computer device and computer-readable storage medium. Background Art

[0002] With the development of image processing technology, blur removal processing technology has emerged. Blur removal processing technology refers to the technology of processing the blurred area in the image to be processed to improve the image clarity.

[0003] In the traditional method, the image to be processed is input into a trained defuzzification neural network model, and the defuzzification neural network model removes blur from the image to be processed to obtain a target image, resulting in lower clarity of the target image, that is, lower accuracy of image processing. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device, and computer-readable storage medium, which can improve the accuracy of image processing.

[0005] In a first aspect, the present application provides an image processing method. The method comprises:

[0006] Acquire an image to be processed and a reference image corresponding to the image to be processed; the resolution of the image to be processed is a first resolution, and the resolution of the reference image is a second resolution; the first resolution is different from the second resolution;

[0007] Based on the first resolution and the second resolution, sampling the image to be processed to obtain a sampled image; the resolution of the sampled image is the second resolution;

[0008] fusing the sampled image and the reference image to obtain the fused image;

[0009] The fused image is deblurred to obtain a target image.

[0010] In a second aspect, the present application also provides an image processing device. The device comprises:

[0011] An acquisition module, used for acquiring an image to be processed and a reference image corresponding to the image to be processed; the resolution of the image to be processed is a first resolution, and the resolution of the reference image is a second resolution; the first resolution is different from the second resolution;

[0012] A sampling module, configured to perform sampling processing on the image to be processed based on the first resolution and the second resolution to obtain a sampled image; the resolution of the sampled image is the second resolution;

[0013] A fusion module, used for fusing the sampled image and the reference image to obtain the fused image;

[0014] The processing module is used to perform a deblurring process on the fused image to obtain a target image.

[0015] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method in the first aspect are implemented.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In the above-mentioned image processing method, the image to be processed is sampled according to the first resolution and the second resolution to obtain a sampled image with the same resolution as the reference image, which provides a fusion basis for the image to be processed and the reference image. The reference image is an image formed according to the event stream. The pixel value of each pixel in the reference image represents the brightness change of the pixel. The sampled image and the reference image are fused, and the pixel value of the pixel in the obtained fused image includes the brightness value of the pixel and the brightness change of the pixel. The fused image is deblurred, that is, in the process of deblurring, the brightness value and brightness change of each pixel in the fused image are used, thereby improving the accuracy of the deblurring and improving the clarity of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flowchart of an image processing method in one embodiment;

[0021] Figure 2 A flowchart of a reference image determination step in one embodiment;

[0022] Figure 3 is a schematic diagram of a residual block convolution module in one embodiment;

[0023] Figure 4 is a schematic diagram of brightness change of an event pixel in one embodiment;

[0024] Figure 5 A flowchart of a target image determination step in one embodiment;

[0025] Figure 6 A flowchart of the steps of determining an up-sampled image in one embodiment;

[0026] Figure 7 is a schematic diagram of a deblurring model in one embodiment;

[0027] Figure 8 is a structural block diagram of an image processing device in one embodiment;

[0028] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] In one embodiment, Figure 1 As shown, an image processing method is provided, and the method is applied to a computer device as an example for explanation. The computer device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, smart cars, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. It can be understood that the method can also be applied to a server, and can also be applied to a system including a computer device and a server, and is implemented through the interaction between the computer device and the server. In this embodiment, the method includes the following steps 102 to 108, wherein:

[0031] Step 102, obtaining an image to be processed and a reference image corresponding to the image to be processed; the resolution of the image to be processed is a first resolution, and the resolution of the reference image is a second resolution; the first resolution is different from the second resolution.

[0032] The image to be processed refers to an image containing a blurred area. The image to be processed may be an image containing a blurred area taken by a camera during movement, or an image containing a blurred area obtained by a camera taking a moving object. The image to be processed includes a plurality of pixels, and the pixel value of the pixel represents the brightness of the pixel. The reference image refers to an image obtained according to the event stream corresponding to the image to be processed. The resolution of the reference image is the same as the resolution corresponding to the event stream. The reference image includes a plurality of event pixels, and the pixel value of the event pixel represents the brightness change of the pixel. The pixel value of the event pixel may be a positive number, a negative number or zero. The resolution refers to the number of pixels in the image. For example, the resolution of the image is 1920*1080, that is, each row in the image contains 1920 pixels and each column contains 1080 pixels. The first resolution refers to the resolution of the image to be processed. The second resolution refers to the resolution of the reference image. The first resolution is different from the second resolution. The first resolution may be greater than the second resolution, or may be less than the second resolution.

[0033] Exemplarily, a computer device obtains an image to be processed and a reference image for deblurring the image to be processed, wherein the resolution of the image to be processed is a first resolution, the resolution of the reference image is a second resolution, and the first resolution is different from the second resolution.

[0034] In one embodiment, the computer device obtains the image to be processed and the candidate event stream corresponding to the image to be processed, determines the exposure time period of the image to be processed, obtains the event stream corresponding to the exposure time period from the candidate event stream, and determines the reference image based on the event stream. The exposure time period refers to the time period between the start exposure time and the end exposure time corresponding to the image to be processed.

[0035] In one embodiment, the computer device obtains the image to be processed and the candidate event stream corresponding to the image to be processed, determines the exposure time period of the image to be processed, determines the target time period from the exposure time period, obtains the event stream corresponding to the target time period from the candidate event stream, and determines the reference image based on the event stream. The target time period refers to a time period in the exposure time period, and the time period can be set according to the demand. For example, if the exposure time period of the image to be processed is 12 seconds to 18 seconds, the target time period can be 14 seconds to 16 seconds.

[0036] Step 104: based on the first resolution and the second resolution, the image to be processed is sampled to obtain a sampled image; the resolution of the sampled image is the second resolution.

[0037] Among them, sampling processing refers to the process of obtaining a limited number of pixels from the image to be processed, and the sampling processing can be upsampling processing or downsampling processing. Upsampling processing refers to the process of improving the image resolution, which can be understood as the process of increasing the number of pixels in the image. The upsampling processing method can be but not limited to nearest neighbor interpolation, linear interpolation and bilinear interpolation; downsampling processing refers to the process of reducing the image resolution, which can be understood as the process of reducing the number of pixels in the image. The downsampling processing method can be but not limited to mean sampling, Gaussian filter sampling and smoothing filter sampling. The sampled image refers to the image obtained by sampling the image to be processed, and the resolution of the sampled image is the same as the resolution of the reference image.

[0038] Exemplarily, the computer device obtains a first resolution of the image to be processed and a second resolution of the reference image, determines a target sampling method and a target sampling multiple of the sampling process based on the first resolution and the second resolution, and performs sampling process on the image to be processed based on the target sampling method and the target sampling multiple to obtain a sampled image with a resolution of the second resolution, wherein the target method is one of upsampling process and downsampling process.

[0039] Step 106: Fuse the sampled image and the reference image to obtain a fused image.

[0040] Among them, fusion refers to the process of merging the sampled image and the reference image. The fusion method can be set according to actual needs and is not limited here. For example, the sampled image is represented by a two-dimensional array, and the reference image is represented by a two-dimensional array. The two two-dimensional data are merged into a three-dimensional array, and the obtained three-dimensional array is the fused image.

[0041] Exemplarily, the computer device fuses the sampled image and the reference image to obtain a fused image.

[0042] Step 108: Deblur the fused image to obtain a target image.

[0043] Deblurring refers to the process of processing the fuzzy area in the fused image to improve the clarity of the fused image. The target image refers to the image obtained by deblurring the fused image.

[0044] Exemplarily, the computer device performs deblurring processing on the fused image to obtain a target image.

[0045] In one embodiment, the computer device determines a target dynamic image based on the fused image, determines a target static image based on the image to be processed and the reference image, and obtains a target image based on the target dynamic image and the target static image.

[0046] In one embodiment, the computer device determines the target dynamic image based on the fused image, determines the target static image based on the image to be processed and the event stream corresponding to the image to be processed, and obtains the target image based on the target dynamic image and the target static image.

[0047] In the above-mentioned image processing method, the image to be processed is sampled according to the first resolution and the second resolution to obtain a sampled image with the same resolution as the reference image, which provides a fusion basis for the image to be processed and the reference image. The reference image is an image formed according to the event stream. The pixel value of each pixel in the reference image represents the brightness change of the pixel. The sampled image and the reference image are fused, and the pixel value of the pixel in the obtained fused image includes the brightness value of the pixel and the brightness change of the pixel. The fused image is deblurred, that is, in the process of deblurring, the brightness value and brightness change of each pixel in the fused image are used, thereby improving the accuracy of the deblurring and improving the clarity of the target image.

[0048] In one embodiment, Figure 2 As shown, obtaining the image to be processed and the reference image corresponding to the image to be processed includes:

[0049] Step 202, obtaining an image to be processed and an event stream corresponding to the image to be processed; the resolution corresponding to the event stream is a second resolution.

[0050] Among them, the event stream refers to a continuous sequence composed of events, and the event includes a timestamp and polarity information. The timestamp included in the event represents the moment when the brightness of the corresponding pixel changes, and the polarity information included in the event represents the situation in which the brightness of the corresponding pixel changes. The event stream corresponding to the image to be processed refers to the event stream within the exposure time period of the image to be processed. It can be understood that the occurrence time period of the event stream corresponding to the image to be processed is the same as the exposure time period of the image to be processed. The image to be processed and the event stream corresponding to the image to be processed can be captured by a time-synchronized RGB camera (Red Green Blue, color camera) and a DVS camera (Event-Based Vision Sensor, event camera), respectively. The image to be processed and the event stream corresponding to the image to be processed can also be captured by a Hybrid DVS (Hybrid Dynamic Vision Sensor, hybrid dynamic vision sensor) camera.

[0051] Exemplarily, a computer device obtains an image to be processed, determines an exposure time period of the image to be processed, and obtains an event stream within the exposure time period, wherein a resolution corresponding to the event stream is the second resolution.

[0052] Step 204, divide the event stream into multiple sub-event streams.

[0053] Among them, the sub-event flow refers to a part of the event flow.

[0054] Exemplarily, the computer device divides the event stream into multiple sub-event streams based on the configuration partitioning parameter. The configuration partitioning parameter refers to the number of sub-event streams that are set, and the configuration partitioning parameter is generally set to an even number. For example, if the configuration partitioning parameter is 4, the event stream is divided into 4 sub-event streams.

[0055] In one embodiment, the computer device divides the event stream based on a configured time interval to obtain a plurality of sub-event streams, wherein the configured time interval refers to a configured time interval of the sub-event streams.

[0056] Step 206: for each sub-event stream, based on each event in the sub-event stream, determine the event image corresponding to the sub-event stream.

[0057] Among them, an event refers to information recording changes in the brightness of a pixel, and an event image refers to an image determined based on a sub-event stream.

[0058] Exemplarily, for each sub-event stream, the event pixel point corresponding to each event in the sub-event stream is determined, and based on multiple events corresponding to the same event pixel point, the pixel value of the event pixel point is determined; based on the pixel values ​​of each event pixel point, an event image corresponding to the sub-event stream is obtained.

[0059] Step 208: Perform feature extraction processing on multiple event images to obtain a reference image.

[0060] The feature extraction process refers to the process of extracting informative and representative features from multiple event images to form a new image. The feature extraction process may include but is not limited to at least one of low-level feature extraction and high-level feature extraction. Low-level feature extraction refers to the process of extracting basic visual information such as edges, corners, colors, and textures in an image. Low-level feature extraction can use a convolution module, such as a 3×3 convolution module; high-level feature extraction refers to the process of extracting semantics, complex structures, and patterns in an image. High-level feature extraction can use a residual block convolution module, such as Figure 3 As shown, a residual block convolution module is composed of two convolution modules. The feature extraction process can be performed using a neural network including a convolution module and a residual block convolution module. The reference image refers to an image obtained by performing feature extraction on multiple event images.

[0061] Exemplarily, the computer device performs feature extraction processing on a plurality of event images to obtain a reference image.

[0062] In this embodiment, the event stream is divided into sub-event streams, and event images are determined based on multiple events in the sub-event streams. Compared with determining the event image based on the event stream, the offset of brightness change information of event pixels is reduced, more brightness change information of event pixels is retained, and feature extraction is performed on multiple event images. The obtained reference image contains richer brightness change information, which improves the accuracy of the reference image. In addition, the noise in the event image is suppressed during the feature extraction process, which further improves the accuracy of the reference image.

[0063] In one embodiment, the event includes a pixel change parameter value; for each sub-event stream, based on each event in the sub-event stream, determining an event image corresponding to the sub-event stream includes:

[0064] Determine the event pixel point corresponding to the event; for each sub-event stream, count the pixel change parameter values ​​of multiple events corresponding to the same event pixel point to obtain the pixel change parameter statistics corresponding to the event pixel point; based on the pixel change parameter statistics corresponding to each event pixel point, obtain the event image corresponding to the sub-event stream.

[0065] Among them, event pixel point refers to the pixel point corresponding to the event, which can be understood as the pixel point corresponding to the brightness change information recorded by the event. For example, the brightness change information of pixel point A recorded in event 1, pixel point A is the event pixel point corresponding to event 1. The pixel change parameter value refers to the numerical value of the polarity information in the event, and the pixel change parameter value can be positive and negative, for example, +1 and -1. When the pixel change parameter value is a positive number, the brightness of the event pixel point increases, and the increase is equal to the increase threshold; when the pixel change parameter value is a negative number, the brightness of the event pixel point decreases, and the decrease is equal to the decrease threshold. For example, Figure 4 The figure shows the brightness change of a certain event pixel. The horizontal axis is time and the vertical axis is brightness. logI(x,y,t) is the brightness of the event pixel at time t, logI(x,y,t-Δt) is the brightness of the event pixel at time t-Δt, +C and -C are the brightness change amplitudes between the current moment and the previous moment. +C indicates that the brightness of the event pixel increases, and the increase amplitude is equal to the increase threshold. The pixel change parameter value of the event pixel at the current moment is +1; -C indicates that the brightness of the event pixel decreases, and the decrease amplitude is equal to the decrease threshold. The pixel change parameter value of the event pixel at the current moment is -1. The pixel change parameter statistics refer to the values ​​obtained by statistically analyzing the pixel change parameter values ​​of multiple events corresponding to the same event pixel in an event stream.

[0066] Exemplarily, for each sub-event stream, the computer device first determines the event pixel points corresponding to each event in the sub-event stream, then adds the pixel change parameter values ​​of multiple events corresponding to the same event pixel points to obtain the pixel change parameter statistics corresponding to the event pixel points, and then obtains the event image corresponding to the sub-event stream based on the pixel change parameter statistics corresponding to each event pixel point.

[0067] In this embodiment, the pixel change parameter values ​​of multiple events corresponding to the same event pixel point are added to obtain the pixel change parameter statistics corresponding to the event pixel point. The pixel change parameter statistics reflect the change in the brightness of the event pixel point over a period of time. Based on the pixel change parameter statistics corresponding to each event pixel point, the event image corresponding to the sub-event stream is obtained, which provides basic data for the subsequent determination of the reference image.

[0068] In one embodiment, based on the first resolution and the second resolution, sampling the image to be processed to obtain the sampled image includes:

[0069] Based on the first resolution and the second resolution, a target sampling multiple is determined; based on the target sampling multiple, a sampling process is performed on the image to be processed to obtain a sampled image.

[0070] The target sampling multiple refers to the multiple by which the resolution of the image to be processed is enlarged or reduced.

[0071] Exemplarily, the computer device determines a target sampling multiple based on a proportional relationship between the first resolution and the second resolution, determines a target sampling method for sampling processing based on a size relationship between the first resolution and the second resolution, and performs sampling processing on the image to be processed based on the target sampling multiple and the target sampling method to obtain a sampled image with a resolution of the second resolution. If the first resolution is greater than the second resolution, the target sampling method is downsampling; if the first resolution is less than the second resolution, the target sampling method is upsampling.

[0072] In this embodiment, the target sampling multiple is determined by the first resolution and the second resolution, which satisfies both the situation that the first resolution is greater than the second resolution and the situation that the first resolution is less than the second resolution. The image to be processed is sampled according to the target sampling multiple, and the obtained sampled image has the same resolution as the reference image, laying a foundation for the subsequent fusion of the sampled image and the reference image.

[0073] In one embodiment, the first resolution is greater than the second resolution; based on the first resolution and the second resolution, sampling the image to be processed to obtain the sampled image includes:

[0074] Based on the proportional relationship between the first resolution and the second resolution, a target sampling multiple is determined; based on the target sampling multiple, the image to be processed is downsampled to obtain a sampled image.

[0075] Exemplarily, when the first resolution is greater than the second resolution, the first resolution is divided by the second resolution to obtain a resolution ratio, the square root of the resolution ratio is taken to obtain a target sampling multiple, and the processed image is downsampled by the target sampling multiple to obtain a sampled image.

[0076] In one embodiment, when the first resolution is smaller than the second resolution, the second resolution is divided by the first resolution to obtain a resolution ratio, the square root of the resolution ratio is taken to obtain a target sampling multiple, and the image to be processed is upsampled by the target sampling multiple to obtain a sampled image.

[0077] In this embodiment, when the first resolution is greater than the second resolution, the processed image is downsampled by a target sampling multiple, and the resolution of the obtained sampled image is the same as the resolution of the reference image, laying a foundation for subsequent fusion of the sampled image and the reference image.

[0078] In one embodiment, based on the target sampling multiple, downsampling the image to be processed to obtain a sampled image includes:

[0079] Perform feature extraction processing on the image to be processed to obtain a feature image to be processed; perform downsampling processing on the feature image to be processed based on the target sampling multiple to obtain a sampled image.

[0080] The feature image to be processed refers to an image obtained by performing feature extraction processing on the image to be processed.

[0081] Exemplarily, the computer device first performs feature extraction on the image to be processed to obtain a feature image to be processed, and then performs downsampling processing on the feature image to be processed at a target sampling multiple to obtain a sampled image with a second resolution.

[0082] In this embodiment, during the feature extraction process of the image to be processed, the noise in the image to be processed is suppressed, thereby improving the accuracy of the feature image to be processed, and then the feature image to be processed is downsampled to improve the accuracy of the sampled image.

[0083] In one embodiment, Figure 5 As shown, the fused image is deblurred to obtain the target image, including:

[0084] Step 502: Perform upsampling processing based on the target sampling multiple and the fused image to obtain an upsampled image.

[0085] Exemplarily, a computer device performs feature extraction processing on an image to be processed to obtain a feature image to be processed; performs downsampling processing on the feature image to be processed at a target sampling multiple to obtain a sampled image; fuses the sampled image and a reference image to obtain a fused image; and performs upsampling processing on the fused image at a target sampling multiple to obtain an upsampled image of a first resolution.

[0086] In one embodiment, a computer device inputs an image to be processed and multiple event images into a deblurring model. The first layer of the neural network of the deblurring model performs low-level feature extraction and high-level feature extraction on the multiple event images to obtain a reference image, and the reference image is input into the second layer of the neural network. The first layer of the neural network performs low-level feature extraction and high-level feature extraction on the image to be processed to obtain a feature image to be processed, and the feature image to be processed is input into the second layer of the neural network. The second layer of the neural network performs downsampling processing of the feature image to be processed at a target sampling multiple and high-level feature extraction to obtain a sampled image, and the sampled image and the reference image are fused to obtain a fused image, which is input into the third layer of the neural network. The third layer of the neural network performs upsampling processing of the fused image at a target sampling multiple and high-level feature extraction to obtain an upsampled image.

[0087] Step 504: perform fusion processing on the upsampled image and the feature image to be processed to obtain an upsampled fused image.

[0088] Exemplarily, the computer device performs fusion processing on the upsampled image and the feature image to be processed to obtain an upsampled fused image.

[0089] In one embodiment, for each pixel in the upsampled image, the upsampled pixel value corresponding to the pixel and the to-be-processed feature pixel value of the pixel corresponding to the pixel in the to-be-processed feature image are obtained, and the upsampled pixel value corresponding to the pixel and the to-be-processed feature pixel value are weighted averaged to obtain the upsampled fused pixel value of the pixel, and an upsampled fused image is obtained based on the upsampled fused pixel values ​​of each pixel.

[0090] In one embodiment, the first-layer neural network of the deblurring model inputs the feature image to be processed into the third-layer neural network through a skip connection, and the third-layer neural network fuses the upsampled image and the feature image to be processed to obtain an upsampled fused image.

[0091] Step 506: Perform feature extraction processing on the upsampled fused image to obtain a target dynamic image.

[0092] Among them, the target dynamic image refers to the image obtained by performing feature extraction processing on the upsampled fused image. It can be understood that the fused image is processed through steps 502 to 506, and the pixel values ​​of each pixel in the fused image are adjusted. The blurred area after adjustment becomes clear, and the pixel values ​​of the pixels in the clear area after adjustment also change, which reduces the accuracy of the clear area. Combined with the statistical value of the pixel change parameter contained in the pixel point, the clear area after adjustment with reduced accuracy is removed. Therefore, only the blurred area after adjustment is retained in the target dynamic image, and the target dynamic image reflects the clear image of the blurred area caused by movement.

[0093] In one embodiment, the third layer neural network of the deblurring model performs low-level feature extraction on the upsampled fused image to obtain a target dynamic image.

[0094] Exemplarily, the computer device performs feature extraction processing on the upsampled fused image to obtain a target dynamic image.

[0095] Step 508: upsample the reference image based on the target sampling multiple to obtain a reference upsampled image.

[0096] Exemplarily, the computer device performs upsampling processing on the reference image by a target multiple to obtain a reference upsampled image of the first resolution.

[0097] In one embodiment, the third layer neural network of the deblurring model performs upsampling processing of the reference image at a target sampling multiple and performs high-level feature extraction to obtain a reference upsampled image.

[0098] Step 510: Obtain a target static image based on the reference up-sampled image and the image to be processed.

[0099] The target static image refers to an image containing image content of a clear area in the image to be processed.

[0100] Exemplarily, the computer device determines static pixel points based on the reference up-sampled image, and determines a target static image based on the static pixel points and the image to be processed.

[0101] In one embodiment, the third layer of the neural network of the deblurring model obtains a target static image based on the reference upsampled image and the image to be processed.

[0102] Step 512, obtaining a target image based on the target dynamic image and the target static image.

[0103] Exemplarily, for each pixel in the target static image, the computer device obtains the static pixel value of the pixel in the target static image and the dynamic pixel value of the corresponding pixel in the target dynamic image, performs weighted fusion on the static pixel value and the dynamic pixel value of the pixel to obtain the target pixel value of the pixel, and obtains the target image based on the target pixel values ​​of each pixel.

[0104] In one embodiment, the third layer neural network of the deblurring model fuses the target dynamic image and the target static image to obtain the target image.

[0105] In one embodiment, the third-layer neural network of the deblurring model fuses the target dynamic image and the target static image to obtain an intermediate image, and inputs the intermediate image into the fourth-layer neural network; the fourth-layer neural network performs low-level feature extraction and high-level feature extraction on the intermediate image to obtain the target image.

[0106] In this embodiment, the fused image is upsampled by the target sampling multiple to obtain an upsampled image of the first resolution. The fused image is interpolated during the upsampling process. The pixel value of the interpolated pixel is determined by the pixel value of the adjacent pixel, thereby reducing the fuzzy area in the upsampled fused image and providing a basis for the subsequent fusion of the upsampled image and the feature image to be processed. The upsampled image of the first resolution and the feature image to be processed of the first resolution are fused to obtain an upsampled fused image of the first resolution, which contains the image information of the upsampled image and the image information of the feature image to be processed. The upsampled fused image is feature extracted, thereby improving the accuracy of the target dynamic image. Then, the reference image is upsampled by the target sampling multiple to obtain a reference upsampled image of the first resolution. The pixel value of each pixel in the reference upsampled image reflects whether the brightness of the pixel has changed. According to the upsampled image of the first resolution and the image to be processed of the first resolution, the target static image can be accurately determined. According to the target static image and the target dynamic image, the target image is obtained, thereby improving the accuracy of the target image.

[0107] In one embodiment, Figure 6 As shown, based on the target sampling multiple and the fused image, upsampling processing is performed to obtain an upsampled image, including:

[0108] Step 602: based on the configured sampling multiple, down-sampling processing and feature extraction processing are performed on the fused image to obtain a fused down-sampled image.

[0109] The configured sampling multiple refers to the set sampling multiple. For example, the configured sampling multiple is 2.

[0110] Exemplarily, the computer device performs downsampling processing and feature extraction processing on the fused image with a configured sampling multiple to obtain a fused downsampled image of a third resolution, wherein the third resolution is smaller than the second resolution.

[0111] In one embodiment, a computer device inputs an image to be processed and multiple event images into a deblurring model. The first layer of the neural network of the deblurring model performs low-level feature extraction and high-level feature extraction on the multiple event images to obtain a reference image, and the reference image is input into the second layer of the neural network. The first layer of the neural network performs low-level feature extraction and high-level feature extraction on the image to be processed to obtain a feature image to be processed, and the feature image to be processed is input into the second layer of the neural network. The second layer of the neural network performs downsampling processing of the target sampling multiple and high-level feature extraction on the feature image to be processed to obtain a sampled image, and fuses the sampled image and the reference image to obtain a fused image, which is input into the third layer of the neural network. The third layer of the neural network performs downsampling processing of the configured sampling multiple, low-level feature extraction, and high-level feature extraction on the fused image to obtain a fused downsampled image.

[0112] Step 604 , based on the configured sampling multiple, down-sampling processing and feature extraction processing are performed on the reference image to obtain a reference down-sampled image.

[0113] Exemplarily, the computer device performs downsampling processing with a configured sampling multiple and feature extraction processing on the reference image to obtain a reference downsampled image of the third resolution.

[0114] In one embodiment, the third layer neural network of the deblurring model performs downsampling processing with a configured sampling multiple and high-level feature extraction on the reference image to obtain a reference downsampled image.

[0115] Step 606: perform fusion processing on the fused down-sampled image and the reference down-sampled image to obtain an aggregated down-sampled image.

[0116] Exemplarily, the computer device performs a fusion process on the fused down-sampled image of the third resolution and the reference down-sampled image of the third resolution to obtain an aggregated down-sampled image of the third resolution.

[0117] In one embodiment, the third-layer neural network of the deblurring model fuses the fused down-sampled image and the reference down-sampled image to obtain an aggregated down-sampled image, and the aggregated down-sampled image is input into the fourth-layer neural network model.

[0118] Step 608: upsample the aggregated downsampled image based on the configured sampling multiple to obtain an aggregated upsampled image.

[0119] Exemplarily, the computer device performs upsampling processing on the aggregated downsampled image by a configured multiple to obtain an aggregated upsampled image of the second resolution.

[0120] In one embodiment, the fourth layer neural network of the deblurring model performs upsampling processing on the aggregated downsampled image with a configured sampling multiple to obtain an aggregated upsampled image.

[0121] Step 610: Fusing the aggregated upsampled image and the sampled image to obtain an updated fused image.

[0122] Exemplarily, the computer device performs a fusion process on the aggregated up-sampled image of the second resolution and the sampled image of the second resolution to obtain an updated fused image of the second resolution.

[0123] In one embodiment, the second layer neural network in the deblurring model inputs the sampled image to the fourth layer neural network through a skip connection, and the fourth layer neural network fuses the aggregated upsampled image and the sampled image to obtain an updated fused image.

[0124] Step 612: Perform feature extraction processing on the updated fused image to obtain an updated feature image.

[0125] Exemplarily, the computer device performs feature extraction processing on the updated fused image with the second resolution to obtain an updated feature image with the second resolution.

[0126] In one embodiment, the fourth layer of the neural network in the deblurring model performs high-level feature extraction on the updated fused image to obtain an updated feature image, and transmits the updated feature image to the fifth layer of the neural network model.

[0127] Step 614: upsample the updated feature image based on the target sampling multiple to obtain an upsampled image.

[0128] Exemplarily, the computer device performs upsampling processing on the updated feature image of the second resolution at a target sampling multiple to obtain an upsampled image.

[0129] In one embodiment, the fifth layer of the neural network in the deblurring model performs upsampling processing on the updated feature image by a target sampling multiple to obtain an upsampled image.

[0130] In one embodiment, after step 614, executing steps 504 to 512 includes: the first layer of neural network in the deblurring model inputs the feature image to be processed to the fifth layer of neural network through a skip connection, the fifth layer of neural network fuses the upsampled image and the feature image to be processed to obtain an upsampled fused image, and performs low-level feature extraction on the upsampled fused image to obtain a target dynamic image. The fifth layer of neural network in the deblurring model performs upsampling processing and high-level feature extraction on the reference image at a target sampling multiple to obtain a reference upsampled image, and obtains a target static image based on the reference upsampled image and the image to be processed. The fifth layer of neural network in the deblurring model fuses the target dynamic image and the target static image to obtain a target image;

[0131] In one embodiment, the fifth neural network in the deblurring model fuses the target dynamic image and the target static image to obtain an intermediate image, and inputs the intermediate image into the sixth neural network; the sixth neural network performs low-level feature extraction and high-level feature extraction on the intermediate image to obtain the target image.

[0132] In this embodiment, the fused image and the reference image are downsampled again to obtain a fused downsampled image and a reference downsampled image of a third resolution, the fused downsampled image and the reference downsampled image are fused, the obtained aggregated downsampled image includes the image features of the fused downsampled image and the image features of the reference downsampled image, the aggregated downsampled image is upsampled, that is, the aggregated downsampled image is interpolated, the pixel value of the interpolated pixel is determined according to the pixel values ​​of the adjacent pixels, the blurred area in the aggregated downsampled image is reduced, and a basis for implementing the subsequent fusion of the aggregated upsampled image and the sampled image is provided. The aggregated upsampled image and the sampled image are fused, the obtained updated fused image includes the image features of the aggregated upsampled image and the image features of the sampled image, thereby improving the accuracy of the updated feature image, the updated feature image is upsampled, and the blurred area in the updated feature image is further reduced, thereby improving the accuracy of the upsampled image.

[0133] In one embodiment, obtaining a target image based on a target dynamic image and a target static image includes:

[0134] The target dynamic image and the target static image are fused to obtain an intermediate image; and the intermediate image is feature extracted to obtain a target image.

[0135] The intermediate image refers to an image obtained by fusing the target dynamic image and the target static image.

[0136] Exemplarily, for each pixel in the target static image, the computer device obtains the static pixel value of the pixel in the target static image and the dynamic pixel value of the corresponding pixel in the target dynamic image, performs weighted fusion on the static pixel value and the dynamic pixel value of the pixel to obtain the target pixel value of the pixel, obtains an intermediate image based on the target pixel value of each pixel, and then performs feature extraction processing on the intermediate image to obtain the target image.

[0137] In this embodiment, the feature extraction process is performed again on the intermediate image, which can be understood as suppressing the noise in the intermediate image, thereby improving the accuracy of the target image.

[0138] In one embodiment, obtaining a target static image based on a reference upsampled image and an image to be processed includes:

[0139] The pixel points whose pixel values ​​in the reference upsampled image meet the pixel threshold are determined as static pixel points; the static pixel values ​​of the corresponding pixel points in the image to be processed are obtained, and the target static image is obtained based on the static pixel values ​​corresponding to each static pixel point.

[0140] The pixel threshold refers to the threshold value or threshold interval for determining a pixel point as a static pixel point. The pixel threshold can be set according to actual needs, for example, the pixel threshold is set to 0, or the pixel threshold is set to (-1, 1). A static pixel point refers to a pixel point whose pixel change parameter statistical value meets the pixel threshold. A static pixel value refers to the pixel value of a pixel point in the image to be processed that has the same position coordinates as the static pixel point.

[0141] Exemplarily, for each pixel in the reference upsampled image, the computer device obtains the pixel change parameter statistics of the pixel, compares the pixel change parameter statistics with the pixel threshold, and if the pixel change parameter statistics meets the pixel threshold, the pixel is determined as a static pixel; if the pixel change parameter statistics does not meet the pixel threshold, the pixel is determined as a non-static pixel. The static pixel value of the pixel with the same position coordinates as the static pixel in the image to be processed is obtained, and the target static image is obtained according to the static pixel values ​​corresponding to each static pixel and the set pixel values ​​corresponding to each non-static pixel. The set pixel value can be zero.

[0142] In this embodiment, static pixels are determined based on statistical values ​​of pixel change parameters corresponding to the pixels in a reference upsampled image, and then a target static image is determined based on static pixel values ​​corresponding to the static pixels in the image to be processed, thereby ensuring that the pixel values ​​of the pixels in the non-blurred area of ​​the target static image are the same as the pixel values ​​of the pixels in the non-blurred area of ​​the image to be processed, thereby improving the accuracy of the target static image.

[0143] In one embodiment, fusing the sampled image and the reference image to obtain a fused image includes:

[0144] For each target pixel in the sampled image, obtain the reference pixel value of the pixel corresponding to the target pixel in the reference image and the sampled pixel value corresponding to the target pixel, and determine the pixel value set consisting of the reference pixel value and the sampled pixel value as the fused pixel value of the target pixel; based on the fused pixel values ​​corresponding to each target pixel, obtain a fused image.

[0145] The reference pixel value refers to the statistical value of the pixel change parameter of the pixel point with the same position coordinates as the target pixel point in the reference image. The sampled pixel value refers to the pixel value of the target pixel point in the sampled image. The pixel value set refers to the set consisting of the reference pixel value and the sampled pixel value corresponding to a target pixel point.

[0146] Exemplarily, for each target pixel in the sampled image, the computer device obtains the reference pixel value of the pixel in the reference image with the same position coordinates as the target pixel, and the sampling pixel value corresponding to the target pixel in the sampled image, and determines a pixel value set consisting of the reference pixel value and the sampling pixel value as the fused pixel value of the target pixel, and obtains a fused image based on the fused pixel values ​​corresponding to each target pixel.

[0147] In this embodiment, by determining a pixel value set consisting of reference pixel values ​​and sampled pixel values ​​as the fused pixel value of the target pixel point, two images with pixel values ​​of different dimensions are fused to obtain a fused image, which contains the image features of the sampled image and the image features of the reference image.

[0148] In an exemplary embodiment, the deblurring model used in the image processing process is as follows: Figure 7 As shown, it includes 6 layers of neural networks, of which 3 layers have two channels and 3 layers have 1 channel. Each layer of neural network is composed of at least two modules of 3×3 convolution module A, residual convolution module B, and transposed convolution module C. The sliding stride of 3×3 convolution module is 1, so the function of 3×3 convolution module is low-level feature extraction; the sliding stride of 3×3 convolution module is 2, so the function of 3×3 convolution module is down sampling with a sampling multiple of 2; the function of residual convolution module is high-level feature extraction; the function of transposed convolution module is up sampling.

[0149] The computer device obtains an image to be processed, determines an exposure time period of the image to be processed, obtains an event stream within the exposure time period, and divides the event stream into multiple sub-event streams based on the configured division parameters. For each sub-event stream, the computer device first determines the event pixel corresponding to each event in the sub-event stream, then adds the pixel change parameter values ​​of multiple events corresponding to the same event pixel to obtain the pixel change parameter statistics corresponding to the event pixel, and then obtains the event image corresponding to the sub-event stream based on the pixel change parameter statistics corresponding to each event pixel.

[0150] The computer device inputs the image to be processed and multiple event images into the deblurring model. The first channel of the first layer of the neural network of the deblurring model performs low-level feature extraction and high-level feature extraction on the multiple event images to obtain a reference image, which is input into the second layer of the neural network. The second channel of the first layer of the neural network performs low-level feature extraction and high-level feature extraction on the image to be processed to obtain a feature image to be processed, which is input into the second layer of the neural network. The first layer of the neural network in the deblurring model inputs the feature image to be processed into the fifth layer of the neural network through a skip connection.

[0151] The second layer of neural network performs downsampling processing and high-level feature extraction on the feature image to be processed to obtain a sampled image, fuses the sampled image with the reference image to obtain a fused image, and inputs the fused image to the third layer of neural network. The second layer of neural network in the deblurring model inputs the sampled image to the fourth layer of neural network through a skip connection.

[0152] The first channel of the third layer neural network performs downsampling processing of the configuration sampling multiple and high-level feature extraction on the reference image to obtain a reference downsampling image. The second channel of the third layer neural network performs downsampling processing of the configuration sampling multiple, low-level feature extraction and high-level feature extraction on the fused image to obtain a fused downsampling image, fuses the fused downsampling image and the reference downsampling image to obtain an aggregated downsampling image, and inputs the aggregated downsampling image into the fourth layer neural network model.

[0153] The fourth-layer neural network performs upsampling processing on the aggregated downsampled image with a configured sampling multiple to obtain an aggregated upsampled image; the aggregated upsampled image and the sampling image are fused to obtain an updated fused image; advanced feature extraction is performed on the updated fused image to obtain an updated feature image, and the updated feature image is transmitted to the fifth-layer neural network model.

[0154] The first channel of the fifth layer neural network performs upsampling processing of the reference image at the target sampling multiple and high-level feature extraction to obtain a reference upsampled image. The second channel of the fifth layer neural network performs upsampling processing of the updated feature image at the target sampling multiple to obtain an upsampled image; the upsampled image and the feature image to be processed are fused to obtain an upsampled fused image; low-level feature extraction is performed on the upsampled fused image to obtain a target dynamic image; based on the reference upsampled image and the image to be processed, a target static image is obtained; the target dynamic image and the target static image are fused to obtain an intermediate image, which is input into the sixth layer neural network.

[0155] The sixth layer of the neural network performs low-level and high-level feature extraction on the intermediate image to obtain the target image.

[0156] In the above-mentioned image processing method, the image to be processed is sampled according to the first resolution and the second resolution to obtain a sampled image with the same resolution as the reference image, which provides a fusion basis for the image to be processed and the reference image. The reference image is an image formed according to the event stream. The pixel value of each pixel in the reference image represents the brightness change of the pixel. The sampled image and the reference image are fused, and the pixel value of the pixel in the obtained fused image includes the brightness value of the pixel and the brightness change of the pixel. The fused image is deblurred, that is, in the process of deblurring, the brightness value and brightness change of each pixel in the fused image are used, thereby improving the accuracy of the deblurring and improving the clarity of the target image.

[0157] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0158] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0159] In one embodiment, Figure 8 As shown, an image processing device is provided, including: an acquisition module 802, a sampling module 804, a fusion module 806 and a processing module 808, wherein:

[0160] The acquisition module 802 is used to acquire the image to be processed and the reference image corresponding to the image to be processed; the resolution of the image to be processed is a first resolution, and the resolution of the reference image is a second resolution; the first resolution is different from the second resolution;

[0161] A sampling module 804 is used to perform sampling processing on the image to be processed based on the first resolution and the second resolution to obtain a sampled image; the resolution of the sampled image is the second resolution;

[0162] A fusion module 806 is used to fuse the sampled image and the reference image to obtain a fused image;

[0163] The processing module 808 is used to perform deblurring processing on the fused image to obtain a target image.

[0164] In one embodiment, the acquisition module 802 is also used to: acquire the image to be processed and the event stream corresponding to the image to be processed; the resolution corresponding to the event stream is the second resolution; divide the event stream to obtain multiple sub-event streams; for each sub-event stream, based on each event in the sub-event stream, determine the event image corresponding to the sub-event stream; perform feature extraction processing on multiple event images to obtain a reference image.

[0165] In one embodiment, the acquisition module 802 is also used to: determine the event pixel point corresponding to the event; for each sub-event stream, count the pixel change parameter values ​​of multiple events corresponding to the same event pixel point to obtain the pixel change parameter statistical value corresponding to the event pixel point; based on the pixel change parameter statistical value corresponding to each event pixel point, obtain the event image corresponding to the sub-event stream.

[0166] In one embodiment, the sampling module 804 is further used to: determine a target sampling multiple based on the first resolution and the second resolution; and perform sampling processing on the image to be processed based on the target sampling multiple to obtain a sampled image.

[0167] In one embodiment, the sampling module 804 is further used to: determine a target sampling multiple based on a proportional relationship between the first resolution and the second resolution; and perform downsampling processing on the image to be processed based on the target sampling multiple to obtain a sampled image.

[0168] In one embodiment, the sampling module 804 is further used to: perform feature extraction processing on the image to be processed to obtain a feature image to be processed; and perform down-sampling processing on the feature image to be processed based on a target sampling multiple to obtain a sampled image.

[0169] In one embodiment, the processing module 808 is also used to: perform upsampling processing based on the target sampling multiple and the fused image to obtain an upsampled image; perform fusion processing on the upsampled image and the feature image to be processed to obtain an upsampled fused image; perform feature extraction processing on the upsampled fused image to obtain a target dynamic image; perform upsampling processing on the reference image based on the target sampling multiple to obtain a reference upsampled image; obtain a target static image based on the reference upsampled image and the image to be processed; obtain a target image based on the target dynamic image and the target static image.

[0170] In one embodiment, the processing module 808 is further used to: based on the configured sampling multiple, downsample and extract features on the fused image to obtain a fused downsampled image; based on the configured sampling multiple, downsample and extract features on the reference image to obtain a reference downsampled image; fuse the fused downsampled image and the reference downsampled image to obtain an aggregated downsampled image; based on the configured sampling multiple, upsample the aggregated downsampled image to obtain an aggregated upsampled image; fuse the aggregated upsampled image and the sampling image to obtain an updated fused image; perform feature extraction on the updated fused image to obtain an updated feature image; based on the target sampling multiple, upsample the updated feature image to obtain an upsampled image.

[0171] In one embodiment, the processing module 808 is further used to: perform fusion processing on the target dynamic image and the target static image to obtain an intermediate image; and perform feature extraction processing on the intermediate image to obtain a target image.

[0172] In one embodiment, the processing module 808 is also used to: determine the pixel points in the reference upsampled image whose pixel values ​​meet the pixel threshold as static pixel points; obtain the static pixel values ​​of the corresponding pixel points in the image to be processed, and obtain the target static image based on the static pixel values ​​corresponding to each static pixel point.

[0173] In one embodiment, the fusion module 806 is also used to: for each target pixel point in the sampled image, obtain the reference pixel value of the pixel point corresponding to the target pixel point in the reference image, and the sampling pixel value corresponding to the target pixel point, and determine the pixel value set composed of the reference pixel value and the sampling pixel value as the fused pixel value of the target pixel point; based on the fused pixel values ​​corresponding to each target pixel point, obtain a fused image.

[0174] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0175] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0176] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0177] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0178] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0180] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed and a reference image corresponding to the image to be processed; The resolution of the image to be processed is a first resolution, and the resolution of the reference image is a second resolution; the first resolution is different from the second resolution; Based on the first resolution and the second resolution, sampling the image to be processed to obtain a sampled image; The resolution of the sampled image is the second resolution; fusing the sampled image and the reference image to obtain the fused image; The fused image is deblurred to obtain a target image.

2. The method according to claim 1, characterized in that: The obtaining of the image to be processed and a reference image corresponding to the image to be processed includes: Acquire an image to be processed and an event stream corresponding to the image to be processed; the resolution corresponding to the event stream is the second resolution; Dividing the event stream to obtain multiple sub-event streams; For each of the sub-event streams, based on each event in the sub-event stream, determining an event image corresponding to the sub-event stream; Feature extraction processing is performed on the plurality of event images to obtain a reference image.

3. The method according to claim 2, characterized in that The event includes a pixel change parameter value; The step of determining, for each of the sub-event streams, an event image corresponding to the sub-event stream based on each event in the sub-event stream comprises: Determine an event pixel point corresponding to the event; For each of the sub-event streams, pixel change parameter values ​​of multiple events corresponding to the same event pixel point are counted to obtain a pixel change parameter statistical value corresponding to the event pixel point; Based on the pixel change parameter statistics corresponding to each of the event pixels, an event image corresponding to the sub-event stream is obtained.

4. The method according to claim 1, characterized in that: The step of sampling the image to be processed based on the first resolution and the second resolution to obtain a sampled image includes: Determining a target sampling multiple based on the first resolution and the second resolution; Based on the target sampling multiple, the image to be processed is sampled to obtain a sampled image.

5. The method according to claim 4, characterized in that The first resolution is greater than the second resolution; and the sampling process is performed on the image to be processed based on the first resolution and the second resolution to obtain a sampled image, including: Determining a target sampling multiple based on a proportional relationship between the first resolution and the second resolution; Based on the target sampling multiple, down-sampling processing is performed on the image to be processed to obtain a sampled image.

6. The method according to claim 5, characterized in that The step of downsampling the image to be processed based on the target sampling multiple to obtain a sampled image includes: Performing feature extraction processing on the image to be processed to obtain a feature image to be processed; Based on the target sampling multiple, down-sampling processing is performed on the feature image to be processed to obtain a sampled image.

7. The method according to claim 6, characterized in that The deblurring process is performed on the fused image to obtain a target image, comprising: Based on the target sampling multiple and the fused image, up-sampling processing is performed to obtain an up-sampled image; Performing fusion processing on the upsampled image and the feature image to be processed to obtain an upsampled fused image; Performing feature extraction processing on the upsampled fused image to obtain a target dynamic image; Based on the target sampling multiple, upsampling the reference image to obtain a reference upsampled image; Obtaining a target static image based on the reference up-sampled image and the image to be processed; A target image is obtained based on the target dynamic image and the target static image.

8. The method according to claim 7, characterized in that The upsampling process is performed based on the target sampling multiple and the fused image to obtain an upsampled image, including: Based on the configured sampling multiple, downsampling processing and feature extraction processing are performed on the fused image to obtain a fused downsampled image; Based on the configured sampling multiple, downsampling processing and feature extraction processing are performed on the reference image to obtain a reference downsampled image; fusing the fused down-sampled image and the reference down-sampled image to obtain an aggregated down-sampled image; Based on the configured sampling multiple, upsampling the aggregated downsampled image to obtain an aggregated upsampled image; fusing the aggregated upsampled image and the sampled image to obtain an updated fused image; Performing feature extraction processing on the updated fused image to obtain an updated feature image; Based on the target sampling multiple, up-sampling processing is performed on the updated feature image to obtain an up-sampled image.

9. The method according to claim 7, characterized in that: The step of obtaining a target image based on the target dynamic image and the target static image comprises: Performing fusion processing on the target dynamic image and the target static image to obtain an intermediate image; Perform feature extraction processing on the intermediate image to obtain a target image.

10. The method according to claim 7, characterized in that The step of obtaining a target static image based on the reference up-sampled image and the image to be processed includes: Determine the pixel points whose pixel values ​​in the reference up-sampled image meet the pixel threshold as static pixel points; The static pixel values ​​of the pixels corresponding to the static pixel points in the image to be processed are obtained, and the target static image is obtained based on the static pixel values ​​corresponding to the static pixel points.

11. The method according to claim 1, characterized in that: The fusing the sampled image and the reference image to obtain the fused image includes: For each target pixel in the sampled image, obtain a reference pixel value of a pixel corresponding to the target pixel in the reference image and a sampled pixel value corresponding to the target pixel, and determine a pixel value set consisting of the reference pixel value and the sampled pixel value as a fused pixel value of the target pixel; A fused image is obtained based on the fused pixel values ​​corresponding to each of the target pixels.

12. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed and a reference image corresponding to the image to be processed; The resolution of the image to be processed is a first resolution, and the resolution of the reference image is a second resolution; the first resolution is different from the second resolution; A sampling module, configured to perform sampling processing on the image to be processed based on the first resolution and the second resolution to obtain a sampled image; The resolution of the sampled image is the second resolution; A fusion module, used for fusing the sampled image and the reference image to obtain the fused image; The processing module is used to perform deblurring processing on the fused image to obtain a target image.

13. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images