Image processing method, electronic device, computer readable medium

Abstract

The present disclosure provides an image processing method, comprising: performing restoration processing on a to-be-processed image according to a blur kernel to obtain a first intermediate restoration image; generating a first mask image, wherein a pixel value of a pixel in the first mask image represents a degree of dispersion of a pixel value of a corresponding region in the to-be-processed image; performing fusion processing on the first intermediate restoration image and the to-be-processed image according to the first mask image to obtain a second intermediate restoration image; generating a second mask image, wherein a pixel value of a pixel in the second mask image represents an image gradient of a corresponding region in the to-be-processed image; and performing fusion processing on the second intermediate restoration image and the to-be-processed image according to the second mask image to obtain a target image. The present disclosure also provides an electronic device and a computer readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, an electronic device, and a computer-readable medium. Background Technology

[0002] During the camera's exposure time, the relative motion between the camera and the target causes motion blur in the image. This relative motion includes situations where the camera is stationary while the target moves, the camera moves while the target is stationary, and both the camera and target move simultaneously. Motion blur in scenes where the camera moves linearly while the target is stationary can be divided into two categories: uniform motion blur, which conforms to the spatial invariance assumption, and local motion blur, caused by varying object distances within the camera's field of view. Uniform motion blur is typically restored using traditional deblurring algorithms or deep learning algorithms; however, when restoring local motion blur, traditional deblurring algorithms not only fail to improve the quality of the restored image but sometimes also degrade the overall image quality. Summary of the Invention

[0003] This disclosure provides an image processing method, an electronic device, and a computer-readable medium.

[0004] In a first aspect, embodiments of this disclosure provide an image processing method, comprising: performing restoration processing on an image to be processed according to a blur kernel to obtain a first intermediate restored image; generating a first mask image, wherein the pixel values ​​of pixels in the first mask image represent the dispersion of pixel values ​​in corresponding regions of the image to be processed; fusing the first intermediate restored image with the image to be processed according to the first mask image to obtain a second intermediate restored image; generating a second mask image, wherein the pixel values ​​of pixels in the second mask image represent the image gradient in corresponding regions of the image to be processed; and fusing the second intermediate restored image with the image to be processed according to the second mask image to obtain a target image.

[0005] In a second aspect, embodiments of this disclosure provide an electronic device, including: one or more processors; and a memory storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method described in the first aspect of this disclosure.

[0006] Thirdly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method described in the first aspect of embodiments of this disclosure.

[0007] In the image processing method provided in this embodiment, the original image to be processed is fused with the first intermediate restored image after restoration using a first mask image, which can suppress the ringing effect in the restored image; the original image to be processed is fused with the second intermediate restored image after ringing using a second mask image, which can eliminate the degradation and defects generated in the clear area after the image to be processed is restored, which is beneficial to improving the restoration effect of non-uniform blurred images. Attached Figure Description

[0008] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0009] Figure 2 This is a flowchart of some steps in another image processing method according to an embodiment of this disclosure;

[0010] Figure 3 This is a flowchart of some steps in another image processing method according to an embodiment of this disclosure;

[0011] Figure 4 This is a flowchart of some steps in another image processing method according to an embodiment of the present disclosure;

[0012] Figure 5 This is a schematic diagram of the composition of an electronic device according to an embodiment of this disclosure;

[0013] Figure 6 This is a schematic diagram illustrating the composition of a computer-readable medium according to an embodiment of the present disclosure. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions of this disclosure, the image processing method, electronic device, and computer-readable medium provided in this disclosure will be described in detail below with reference to the accompanying drawings.

[0015] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this disclosure.

[0016] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0017] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0019] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0020] Image degradation can be viewed as a convolution operation between the original sharp image and the blur kernel, and it is subject to noise interference. It can be represented by the following formula:

[0021]

[0022] Where b(x,y) represents the blurred image after image degradation, f(x,y) represents the original sharp image, k(x,y) represents the blur kernel, and n(x,y) represents noise. This represents the convolution operation.

[0023] If the blur kernel k(x,y) is known, restoring the blurred image is called non-blind restoration; if the blur kernel k(x,y) is unknown, restoring the blurred image is called blind restoration. Restoring local motion blur caused by the camera moving in a straight line while the target is stationary is a type of non-blind restoration.

[0024] In some related technologies, traditional restoration algorithms or deep learning algorithms are mainly used for non-blind restoration of local motion blur.

[0025] Traditional image restoration algorithms solve non-blind restoration problems based on maximum a posteriori probability, including inverse filtering, Wiener filtering, constrained least squares filtering, geometric mean filtering, and the Richardson-Lucy Algorithm (RL). However, using traditional restoration algorithms to restore blurred images has the following problems: the restored images generally exhibit significant ringing effects; the quality of sharp areas in non-uniformly blurred images is severely degraded after restoration using the above-mentioned traditional restoration algorithms; most of the above-mentioned traditional restoration algorithms involve fast Fourier transform and inverse transform, and their implementation is mostly based on a coarse-to-fine multi-scale iterative framework, resulting in large computational load and high computational complexity.

[0026] Deep learning algorithms can achieve end-to-end adaptive image restoration, reducing the amount of manual parameter tuning. However, deep learning algorithms are complex, have a large number of training parameters, require high hardware resources for deployment, and struggle to handle the restoration of non-uniformly blurred images. Furthermore, artificially synthesized data is often used as the training dataset when training deep learning models, resulting in poor generalization performance.

[0027] In view of this, firstly, referring to Figure 1 This disclosure provides an image processing method, including:

[0028] S1. Perform restoration processing on the image to be processed according to the blur kernel to obtain the first intermediate restored image;

[0029] S2. Generate a first mask image, wherein the pixel value of the pixel in the first mask image represents the degree of dispersion of the pixel value of the corresponding region in the image to be processed;

[0030] S3. Based on the first mask image, fuse the first intermediate restored image with the image to be processed to obtain the second intermediate restored image;

[0031] S4. Generate a second mask image, wherein the pixel value of the pixel in the second mask image represents the image gradient of the corresponding region in the image to be processed;

[0032] S5. Based on the second mask image, the second intermediate restored image and the image to be processed are fused to obtain the target image.

[0033] In this embodiment of the disclosure, the image to be processed is a non-uniformly blurred image that needs to be restored, and the blur kernel is known when performing non-blind restoration on the image to be processed.

[0034] This disclosure does not impose any particular limitation on how to perform non-blind restoration of the image to be processed. For example, the image to be processed can be non-blindly restored based on traditional restoration algorithms, or it can be non-blindly restored based on deep learning algorithms. In some embodiments, an RL algorithm model with prior terms is used to perform non-blind restoration of the image to be processed.

[0035] In this embodiment, the size of the first mask image is the same as the size of the image to be processed. The pixel value of a pixel in the first mask image represents the dispersion of pixel values ​​in the corresponding region of the image to be processed, meaning that the pixel value of a pixel in the first mask image is related to the dispersion of pixel values ​​in the corresponding region of the image to be processed. In some embodiments, the pixel value of a pixel in the first mask image is positively correlated with the dispersion of pixel values ​​in the corresponding region of the image to be processed. In some embodiments, the dispersion of pixel values ​​in the corresponding region of the image to be processed is measured by the variance of pixel values ​​in that region. Based on the interval in which the variance falls, a corresponding pixel value is set for the corresponding pixel in the first mask image. For example, if the variance is greater than a certain threshold, the pixel value of the corresponding pixel in the first mask image is set to 1; if the variance is less than a certain threshold, the pixel value of the corresponding pixel in the first mask image is set to 0. This embodiment does not impose any special limitations on this.

[0036] In this embodiment, the first mask image is generated based on the variance of pixel values ​​in the image to be processed, and can segment smooth and non-smooth regions in the image to be processed. By fusing the image to be processed and the first intermediate restored image using the first mask image, the ringing effect in the first intermediate restored image can be suppressed.

[0037] In this embodiment of the disclosure, the size of the second mask image is consistent with the size of the image to be processed. The pixel value of a pixel in the second mask image represents the image gradient of the corresponding region in the image to be processed, meaning that the pixel value of a pixel in the second mask image is correlated with the image gradient of the corresponding region in the image to be processed. In some embodiments, the pixel value of a pixel in the second mask image is positively correlated with the image gradient of the corresponding region in the image to be processed. In some embodiments, corresponding pixel values ​​are set for the corresponding regions in the second mask image based on the interval into which the image gradient falls.

[0038] In this embodiment, the second mask image is generated based on the image gradient in the image to be processed, and can segment the blurred and sharp regions in the image to be processed. By fusing the image to be processed and the second intermediate restored image after ringing removal using the second mask image, the degradation and defects in the sharp regions generated after the restoration processing of the image to be processed can be eliminated.

[0039] In the image processing method provided in this embodiment, the original image to be processed is fused with the first intermediate restored image after restoration using a first mask image, which can suppress the ringing effect in the restored image; the original image to be processed is fused with the second intermediate restored image after ringing using a second mask image, which can eliminate the degradation and defects generated in the clear area after the image to be processed is restored, which is beneficial to improving the restoration effect of non-uniform blurred images.

[0040] This disclosure does not impose any special limitations on how the first mask image is generated.

[0041] In some embodiments, a first mask image is determined based on the variance of pixel values ​​in the image to be processed using a first sliding unit. The first sliding unit can be a sliding window or a slider; this disclosure does not specifically limit its use. In this disclosure, a sliding window assigns the calculation result to the pixel at its center, while a slider assigns the calculation result to all pixels within the sliding window.

[0042] Accordingly, in some embodiments, reference is made to Figure 2 Generate the first mask image, including:

[0043] S21. Determine the variance of pixel values ​​of pixels in multiple first regions based on the first sliding unit size and the first sliding step size, wherein the first region is a region in the image to be processed that is the size of the first sliding unit.

[0044] S22. Based on the variance of the pixel values ​​of the pixels in the first region, determine the pixel value of at least one pixel in the first mask region corresponding to the first region, wherein the plurality of first mask regions in the first mask image correspond one-to-one with the plurality of first regions.

[0045] In this embodiment of the disclosure, the first region is the region in the image to be processed corresponding to each position of the first sliding unit during the sliding process according to the first sliding step size. The first sliding unit corresponds to a first region and a first mask region at any position.

[0046] This embodiment of the disclosure does not impose any special limitation on the size of the first sliding unit. For example, the size of the first sliding unit is smaller than the size of the image to be processed, denoted as m×n, where m and n are odd numbers. This embodiment of the disclosure also does not impose any special limitation on the first sliding step size. For example, the first sliding step size is 1. This embodiment of the disclosure also does not impose any special limitation on the sliding direction of the first sliding unit. For example, in a Cartesian coordinate system, it slides first along the x-direction and then along the y-direction.

[0047] This disclosure does not impose any special limitations on how to determine the pixel value of at least one pixel in the first mask region corresponding to the first region based on the variance of the pixel values ​​of the pixels in the first region.

[0048] In some embodiments, the first mask image is a non-binarized mask image.

[0049] Accordingly, in some embodiments, determining the pixel value of at least one pixel in the first mask region corresponding to the first region based on the variance of the pixel values ​​of the pixels in the first region includes:

[0050] If the variance of the pixel values ​​of the pixels in the first region is greater than a first upper threshold, the pixel value of at least one pixel in the first mask region corresponding to the first region is set to 0.

[0051] If the variance of the pixel values ​​of the pixels in the first region is less than a first lower threshold, the pixel value of at least one pixel in the first mask region corresponding to the first region is set to 1.

[0052] If the variance of the pixel values ​​of a pixel in the first region is less than the first upper threshold and greater than the first lower threshold, the pixel value of at least one pixel in the first mask region corresponding to the first region is set as the ratio of the first difference and the second difference. The first difference is equal to the difference between the first upper threshold and the variance, and the second difference is equal to the difference between the first upper threshold and the first lower threshold.

[0053] In this embodiment, the first upper threshold is greater than the first lower threshold. The values ​​of the first upper threshold and the first lower threshold can be set as needed, and this embodiment does not impose any special limitations on them.

[0054] In this embodiment of the disclosure, if the first sliding unit is a sliding window, the pixel value of the pixel at the center of the first mask area corresponding to the first region is determined based on the variance of the pixel values ​​of the pixels in the first region; if the first sliding unit is a slider, the pixel value of each pixel in the first mask area corresponding to the first region is determined based on the variance of the pixel values ​​of the pixels in the first region.

[0055] Accordingly, in some embodiments, determining the pixel value of at least one pixel in the first mask region corresponding to the first region based on the variance of the pixel values ​​of the pixels in the first region includes:

[0056] Based on the variance of the pixel values ​​in the first region, determine the pixel value of the pixel at the center of the first mask region corresponding to the first region; or

[0057] The pixel values ​​of each pixel in the first mask region corresponding to the first region are determined based on the variance of the pixel values ​​in the first region.

[0058] Taking the first sliding unit as a sliding window as an example, the process of generating the first mask image includes: initializing the non-binarized first mask image α rThe first sliding window is an all-zero matrix with the same size as the image b(x,y) to be processed. The size of the first sliding unit is m×n, the first sliding step is 1, the first upper threshold is ring_max, and the first lower threshold is ring_min. During the sliding window process, the variance var in the first region corresponding to the sliding window is calculated. If var≥ring_max, the pixel at the center of the first mask region corresponding to the sliding window is assigned a value of 0. If var≤ring_min, the pixel at the center of the first mask region corresponding to the sliding window is assigned a value of 1. If ring_max>var>ring_min, the pixel at the center of the first mask region corresponding to the sliding window is assigned a value of (ring_max-var) / (ring_max-ring_min). Finally, the first mask image is obtained.

[0059] This disclosure does not impose any special limitations on how to fuse the first intermediate restored image with the image to be processed based on the first mask image.

[0060] In some embodiments, fusing the first intermediate restored image with the image to be processed according to the first mask image to obtain a second intermediate restored image includes:

[0061] Based on the first mask image and the image to be processed, a first image to be fused is determined;

[0062] Based on the first inverted color mask and the first intermediate restored image, a second image to be fused is determined, wherein the first mask is obtained by inverting the colors of the first mask.

[0063] The first image to be fused and the second image to be fused are superimposed to obtain the second intermediate restored image.

[0064] It should be noted that color inversion refers to reversing the colors of an image. Color inversion of the first mask image is achieved by subtracting the first mask image from a matrix of all ones with the same dimensions, resulting in a first inverted mask image. Superimposing the first mask image and the first inverted mask image yields a matrix of all ones.

[0065] In some embodiments, the second intermediate restored image is obtained by fusing the first intermediate restored image with the image to be processed according to the first mask image, which can be expressed by the following formula:

[0066] I(x,y)=α r ×b(x,y)+(1-α r )×f(x,y);

[0067] Where I(x,y) represents the second intermediate restored image, b(x,y) represents the image to be processed, f(x,y) represents the first intermediate restored image, and α rThis represents the first mask image, 1-α r This represents the first inverse color mask. The multiplication in the above formula means that the values ​​at the same position in the matrix are multiplied accordingly.

[0068] This disclosure does not impose any special limitations on how the second mask image is generated.

[0069] In some embodiments, a second mask image is determined based on the image gradient in the image to be processed using a second sliding unit. The second sliding unit can be a sliding window or a slider, and this disclosure does not impose any particular limitation on it.

[0070] Accordingly, in some embodiments, reference is made to Figure 3 Generate a second mask image, including:

[0071] S41. Determine the energy functionals of multiple second regions based on the second sliding unit size and the second sliding step size, wherein the second region is a region in the image to be processed that is the size of the second sliding unit, and the energy functional of the second region characterizes the image gradient of the second region.

[0072] S42. Based on the energy functional of the second region, determine the pixel value of at least one pixel in the second mask region corresponding to the second region, wherein the plurality of second mask regions in the second mask diagram correspond one-to-one with the plurality of second regions.

[0073] In this embodiment of the disclosure, the energy functional characterizes the image gradient, representing both the intensity and direction of the image gradient.

[0074] In this embodiment of the disclosure, the second region is the region in the image to be processed corresponding to each position of the second sliding unit during the sliding process according to the second sliding step size. The second sliding unit corresponds to a second region and a second mask region at any position.

[0075] This disclosure does not impose any special limitation on the size of the second sliding unit. For example, the size of the second sliding unit is smaller than the size of the image to be processed, represented as m×n, where m and n are odd numbers. This disclosure also does not impose any special limitation on the second sliding step size. For example, the second sliding step size is 1. This disclosure also does not impose any special limitation on the sliding direction of the second sliding unit. For example, in a Cartesian coordinate system, it slides first along the x-direction and then along the y-direction.

[0076] This disclosure does not impose any special limitations on how to determine the pixel value of at least one pixel in the second mask region corresponding to the second region based on the image gradient of the second region.

[0077] In some embodiments, the first mask image is a non-binarized mask image.

[0078] Accordingly, in some embodiments, determining the pixel value of at least one pixel in the second mask region corresponding to the second region based on the energy functional of the second region includes:

[0079] If the energy functional of the second region is greater than the second upper threshold, the pixel value of at least one pixel in the second mask region corresponding to the second region is set to 1.

[0080] If the energy functional of the second region is less than the second lower threshold, the pixel value of at least one pixel in the second mask region corresponding to the second region is set to 0.

[0081] When the energy functional of the second region is less than the second upper threshold and greater than the second lower threshold, the pixel value of at least one pixel in the second mask region corresponding to the second region is set as the ratio of the third difference and the fourth difference. The third difference is equal to the difference between the energy functional and the second lower threshold, and the fourth difference is equal to the difference between the second upper threshold and the second lower threshold.

[0082] In this embodiment, the second upper threshold is greater than the second lower threshold. The values ​​of the second upper threshold and the second lower threshold can be set as needed, and this embodiment does not impose any special limitations on them.

[0083] In this embodiment of the disclosure, if the second sliding unit is a sliding window, the pixel value of the pixel at the center of the second mask region corresponding to the second region is determined according to the image gradient of the second region; if the second sliding unit is a slider, the pixel value of each pixel in the second mask region corresponding to the second region is determined according to the variance of the pixel values ​​of the pixels in the second region.

[0084] Accordingly, in some embodiments, determining the pixel value of at least one pixel in the second mask region corresponding to the second region based on the energy functional of the second region includes:

[0085] Based on the energy functional of the second region, determine the pixel value of the pixel at the center of the second mask region corresponding to the second region; or

[0086] Based on the energy functional of the second region, determine the pixel value of each pixel in the second mask region corresponding to the second region.

[0087] The embodiments disclosed herein do not impose any special limitations on the energy functional.

[0088] In some embodiments, the energy functional of the second region is determined according to the following formula:

[0089]

[0090] Alternatively, the energy functional of the second region can be determined according to the following formula:

[0091]

[0092] Where E represents the energy functional, gx represents the gradient in the x-direction of the second region, gy represents the gradient in the y-direction of the second region, θ represents the angle between the fuzzy direction and the x-direction, and Ω represents the second region.

[0093] Taking the second sliding unit as an example, the process of generating the second mask image includes: initializing the non-binarized second mask image α l The second sliding unit is an all-zero matrix with the same size as the image b(x,y) to be processed; the second sliding unit size is m×n, the second sliding step size is 1, the second upper threshold is segment_max, and the second lower threshold is segment_min; during the sliding process, the energy functional E of the second region corresponding to the slider is calculated; if E≥segment_max, all pixels in the second mask region corresponding to the slider are assigned a value of 1; if E≤segment_min, all pixels in the second mask region corresponding to the slider are assigned a value of 0; if segment_max>E>segment_min, all pixels in the second mask region corresponding to the slider are assigned a value of (E-segment_min) / (segment_max-segment_min); finally, the second mask image is obtained.

[0094] This disclosure does not impose any special limitations on how to fuse the second intermediate restored image with the image to be processed based on the second mask image.

[0095] In some embodiments, the second intermediate restored image and the image to be processed are fused according to the second mask image to obtain a target image, including:

[0096] Based on the second mask image and the image to be processed, a third image to be fused is determined;

[0097] Based on the second inverted color mask and the second intermediate restored image, a fourth image to be fused is determined, wherein the second mask is obtained by inverting the colors of the second mask;

[0098] The third image to be fused is superimposed with the fourth image to be fused to obtain the target image.

[0099] It should be noted that color inversion refers to reversing the colors of an image. Color inversion of the second mask is achieved by subtracting the second mask from a matrix of all ones with the same dimensions, resulting in a second inverted mask. Superimposing the second mask and the second inverted mask yields a matrix of all ones.

[0100] In some embodiments, the target image is obtained by fusing the second intermediate restored image with the image to be processed according to the second mask image, which can be expressed by the following formula:

[0101] F(x,y)=α l ×b(x,y)+(1-α l )×I(x,y);

[0102] Where F(x,y) represents the target image, b(x,y) represents the image to be processed, I(x,y) represents the second intermediate restored image, and α l This represents the second mask image, 1-α l This represents the second inverted color mask. The multiplication in the above formula means that the values ​​at the same position in the matrix are multiplied accordingly.

[0103] This disclosure does not impose any special limitations on how the image to be restored is processed.

[0104] In some embodiments, the RL algorithm with prior terms is optimized for accelerated convergence, and the image to be processed is restored based on the improved RL algorithm.

[0105] Accordingly, in some embodiments, reference is made to Figure 4 The image to be processed is restored according to the blur kernel to obtain the first intermediate restored image, including:

[0106] S11. Set the model parameters of the RL algorithm model, wherein the model parameters include at least one of the following: number of iterations, acceleration factor, prior term weights, and prior term weight reduction rate;

[0107] S12. Input the blur kernel and the image to be processed into the RL algorithm model for restoration processing to obtain the first intermediate restored image.

[0108] In this embodiment of the disclosure, the accelerated convergence optimization of the RL algorithm with prior terms includes:

[0109] The BA algorithm (Biggs D, Andrews M Acceleration of iterative image restoration algorithms), which is an improvement on the linear search algorithm, is adopted as an acceleration scheme. The iterative computation process of the RL algorithm can be expressed by the following formula:

[0110]

[0111] h k =x k -x k-1 ;

[0112] yk =x k +a k h k ;

[0113] x k+1 =y k +g k ;

[0114] Where, x k Represents the iteration point, y k g represents an intermediate variable. k h represents the estimate of the gradient direction. k a represents the direction vector. k Indicates the acceleration factor. This represents one iteration of the RL algorithm, and k represents the kth iteration.

[0115] In the RL algorithm, a strategy is set to decrease the weight of the prior terms with each iteration;

[0116] The final RL algorithm model is obtained.

[0117] In this embodiment of the disclosure, the improved RL algorithm model is used to perform preliminary restoration of the image to be processed. This means that the image processing in steps S1 to S5 only involves pixel-related calculations in the spatial domain of the image, and does not involve pixel-related calculations in the time domain or frequency domain. It does not require transformations and calculations such as Fourier transform. Moreover, the improved RL algorithm model can converge quickly after a small number of iterations, has a good reuse effect on non-uniform locally blurred images, and has a small amount of computation and low computational complexity, which is beneficial to improving image processing efficiency.

[0118] Secondly, referring to Figure 5 This disclosure provides an electronic device comprising:

[0119] One or more processors 101;

[0120] The memory 102 stores one or more programs that, when executed by one or more processors, enable the one or more processors to implement the image processing method described in the first aspect of the present disclosure.

[0121] One or more I / O interfaces 103 are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0122] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, enabling information exchange between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0123] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0124] Fourthly, refer to Figure 6 This disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method described in the first aspect of this disclosure.

[0125] To enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of this disclosure, the technical solutions provided by the embodiments of this disclosure will be described in detail below through specific embodiments:

[0126] Example

[0127] In this embodiment of the disclosure, the process of restoring a non-uniformly blurred image includes:

[0128] The first step is to input a partially blurred image to be processed, and use the spatial information of the image to calculate and generate a first mask image to suppress ringing and a second mask image to segment the blurred / sharp regions.

[0129] The second step is to use an RL algorithm model with prior terms to restore the image to be processed and output the first intermediate restored image.

[0130] The third step is to fuse the image to be processed and the first intermediate restored image based on the first mask image, and output the second intermediate restored image after ringing removal.

[0131] The fourth step involves fusing the second intermediate restored image after ringing removal and the image to be processed based on the second mask image, and outputting the restored target image.

[0132] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0133] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An image processing method, comprising: The image to be processed is restored based on the blur kernel to obtain the first intermediate restored image; Generate a first mask image, wherein the pixel value of the pixel in the first mask image represents the degree of dispersion of the pixel value of the corresponding region in the image to be processed; The first intermediate restored image is fused with the image to be processed based on the first mask image to obtain the second intermediate restored image; A second mask image is generated, wherein the pixel values ​​of the pixels in the second mask image represent the image gradient of the corresponding region in the image to be processed; The second intermediate restored image is fused with the image to be processed based on the second mask image to obtain the target image.

2. The image processing method according to claim 1, wherein, Generate the first mask image, including: Based on the first sliding unit size and the first sliding step size, the variance of pixel values ​​in multiple first regions is determined, wherein the first region is a region in the image to be processed that is the size of the first sliding unit. Based on the variance of the pixel values ​​of the pixels in the first region, the pixel value of at least one pixel in the first mask region corresponding to the first region is determined, wherein the plurality of first mask regions in the first mask image correspond one-to-one with the plurality of first regions.

3. The image processing method according to claim 2, wherein, Determining the pixel value of at least one pixel in the first mask region corresponding to the first region based on the variance of the pixel values ​​in the first region includes: If the variance of the pixel values ​​of the pixels in the first region is greater than a first upper threshold, the pixel value of at least one pixel in the first mask region corresponding to the first region is set to 0. If the variance of the pixel values ​​of the pixels in the first region is less than a first lower threshold, the pixel value of at least one pixel in the first mask region corresponding to the first region is set to 1. If the variance of the pixel values ​​of a pixel in the first region is less than the first upper threshold and greater than the first lower threshold, the pixel value of at least one pixel in the first mask region corresponding to the first region is set as the ratio of the first difference and the second difference. The first difference is equal to the difference between the first upper threshold and the variance, and the second difference is equal to the difference between the first upper threshold and the first lower threshold.

4. The image processing method according to claim 2 or 3, wherein, Determining the pixel value of at least one pixel in the first mask region corresponding to the first region based on the variance of the pixel values ​​in the first region includes: Based on the variance of the pixel values ​​in the first region, determine the pixel value of the pixel at the center of the first mask region corresponding to the first region; or The pixel values ​​of each pixel in the first mask region corresponding to the first region are determined based on the variance of the pixel values ​​in the first region.

5. The image processing method according to any one of claims 1 to 3, wherein, The first intermediate restored image is fused with the image to be processed based on the first mask image to obtain a second intermediate restored image, including: Based on the first mask image and the image to be processed, a first image to be fused is determined; Based on the first inverted color mask and the first intermediate restored image, a second image to be fused is determined, wherein the first mask is obtained by inverting the colors of the first mask. The first image to be fused and the second image to be fused are superimposed to obtain the second intermediate restored image.

6. The image processing method according to any one of claims 1 to 3, wherein, Generate the second mask image, including: Based on the second sliding unit size and the second sliding step size, energy functionals of multiple second regions are determined, wherein the second region is a region in the image to be processed that is the size of the second sliding unit, and the energy functional of the second region characterizes the image gradient of the second region. Based on the energy functional of the second region, the pixel value of at least one pixel in the second mask region corresponding to the second region is determined, wherein the plurality of second mask regions in the second mask image correspond one-to-one with the plurality of second regions.

7. The image processing method according to claim 6, wherein, Based on the energy functional of the second region, determine the pixel value of at least one pixel in the second mask region corresponding to the second region, including: If the energy functional of the second region is greater than the second upper threshold, the pixel value of at least one pixel in the second mask region corresponding to the second region is set to 1. If the energy functional of the second region is less than the second lower threshold, the pixel value of at least one pixel in the second mask region corresponding to the second region is set to 0. When the energy functional of the second region is less than the second upper threshold and greater than the second lower threshold, the pixel value of at least one pixel in the second mask region corresponding to the second region is set as the ratio of the third difference and the fourth difference. The third difference is equal to the difference between the energy functional and the second lower threshold, and the fourth difference is equal to the difference between the second upper threshold and the second lower threshold.

8. The image processing method according to claim 6, wherein, Based on the energy functional of the second region, determine the pixel value of at least one pixel in the second mask region corresponding to the second region, including: Based on the energy functional of the second region, determine the pixel value of the pixel at the center of the second mask region corresponding to the second region; or Based on the energy functional of the second region, determine the pixel value of each pixel in the second mask region corresponding to the second region.

9. The image processing method according to claim 6, wherein, The energy functional of the second region is determined according to the following formula: Alternatively, the energy functional of the second region can be determined according to the following formula: Where E represents the energy functional, gx represents the gradient in the x-direction of the second region, gy represents the gradient in the y-direction of the second region, θ represents the angle between the fuzzy direction and the x-direction, and Ω represents the second region.

10. The image processing method according to any one of claims 1 to 3, wherein, The second intermediate restored image is fused with the image to be processed according to the second mask image to obtain the target image, including: Based on the second mask image and the image to be processed, a third image to be fused is determined; Based on the second inverted color mask and the second intermediate restored image, a fourth image to be fused is determined, wherein the second mask is obtained by inverting the colors of the second mask; The third image to be fused is superimposed with the fourth image to be fused to obtain the target image.

11. The image processing method according to any one of claims 1 to 3, wherein, The image to be processed is restored based on the blur kernel to obtain the first intermediate restored image, including: Set the model parameters of the RL algorithm model, wherein the model parameters include at least one of the following: number of iterations, acceleration factor, prior term weights, and prior term weight reduction rate; The blur kernel and the image to be processed are input into the RL algorithm model for restoration processing to obtain the first intermediate restored image.

12. An electronic device, comprising: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the image processing method according to any one of claims 1 to 11.

13. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method according to any one of claims 1 to 11.

Images