Image dead pixel removing method and device

Abstract

The invention provides an image dead pixel removing method and device, and the method comprises the steps: calculating a noise estimation value through employing the pixel value of a reference pixel line in an image sensor; screening dead pixels according to the noise estimation value to obtain a dead pixel probability; calculating the gradient of the current pixel point in the cache window to judge the edge type of the current pixel point; and according to the defective pixel probability and the edge type of the current pixel point, carrying out defective pixel removal on the current pixel point. A noise estimation value is calculated by using a reference pixel row, and then a dead pixel probability is obtained; judging the edge type of the current pixel point by calculating the gradient; the intensity of the dead pixel removing algorithm is adaptively adjusted through the dead pixel probability and the edge type, so that the dead pixels are effectively removed while the image clearness is ensured, and the problem of how to effectively remove the image dead pixels while the image quality is ensured is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for removing bad pixels from an image. Background Art

[0002] Image sensor is a device that converts optical images into electronic signals. It is widely used in digital cameras and other electronic optical devices. Early image sensors used analog signals, such as camera tubes. With the rapid development of digital technology, semiconductor manufacturing technology and the Internet, the commonly used image sensor products are mainly divided into two types: CCD image sensors and CMOS image sensors.

[0003] The main parameters of CCD and CMOS include pixel size, sensitivity, number of bad pixels and spectral response, among which the number of bad pixels refers to the number of bad pixels in the image sensor chip (pixels that cannot be effectively imaged or pixels whose response inconsistency is greater than the parameter allowable range), which is an important parameter for measuring the quality of image sensor chips. However, due to the influence of factors such as manufacturing process, temperature aging and dark current, it is almost impossible for all pixels to be able to normally sense light and form images for image sensors with millions of pixels. Therefore, some bad pixels (including black and white dots) will inevitably be generated on the collected original image. If these bad pixels are not removed later, they will spread to the surrounding area due to subsequent image processing operations such as de-mosaicing to form colored dot clusters, significantly reducing image quality and affecting the perception.

[0004] The existing method of removing bad pixels from images usually adopts median filtering. Although median filtering can remove bad pixels, it will also cause image blur and affect image quality. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for removing bad pixels from an image, so as to solve the problem of how to effectively remove bad pixels from an image while ensuring image quality.

[0006] In order to solve the above technical problems, the present invention provides a method for removing bad pixels from an image, comprising:

[0007] calculating a noise estimate using pixel values ​​of a reference pixel row in the image sensor;

[0008] Filter bad pixels according to the noise estimation value to obtain the bad pixel probability;

[0009] Calculate the gradient of the current pixel in the cache window to determine the edge type of the current pixel;

[0010] According to the bad pixel probability and the edge type of the current pixel, the bad pixel of the current pixel is removed.

[0011] Optionally, in the image bad pixel removal method, the method of calculating the noise estimation value using the pixel mean of the reference pixel row in the image sensor includes:

[0012] Obtaining pixel expected values ​​for a reference pixel row in an image sensor;

[0013] Counting the ratio of the number of reference pixel points whose pixel values ​​are the expected pixel values ​​to the total number of reference pixel points in the reference pixel row to obtain a reference ratio;

[0014] Based on the probability distribution of noise, the functional relationship between noise intensity and reference ratio is obtained;

[0015] The noise estimation value is calculated based on the functional relationship between the noise intensity and the reference ratio.

[0016] Optionally, in the image bad pixel removal method, when the probability distribution of noise in the reference pixel is normal distribution, the functional relationship between the obtained noise intensity and the reference ratio is expressed as:

[0017]

[0018] Among them, r represents the reference ratio, T represents the number of reference pixels whose pixel values ​​are the expected pixel values, w represents the number of reference pixels in the length direction of the reference pixel row, h represents the number of reference pixels in the height direction of the reference pixel row; s represents the noise intensity.

[0019] Optionally, in the image bad pixel removal method, the method for calculating the noise estimation value according to the functional relationship between the noise intensity and the reference ratio includes:

[0020] Obtaining a reference ratio of at least one frame of image;

[0021] Substituting the obtained reference ratio into the functional relationship between the noise intensity and the reference ratio, the corresponding noise intensity is calculated;

[0022] A weighted average calculation is performed on all the calculated noise intensities to obtain a noise estimation value.

[0023] Optionally, in the image bad pixel removal method, the method of screening bad pixels according to the noise estimation value to obtain the bad pixel probability includes:

[0024] Calculate the number of reference pixel points in the reference pixel row whose pixel values ​​are greater than a first preset multiple of the noise estimation value, and the number of reference pixel points whose pixel values ​​are less than a second preset multiple of the noise estimation value;

[0025] Counting the ratio of the number of reference pixels whose pixel values ​​are greater than the first preset multiple of the noise estimation value to the total number of reference pixels, so as to obtain the probability of bright bad pixels;

[0026] The ratio of the number of reference pixels whose pixel values ​​are less than the second preset multiple of the noise estimation value to the total number of reference pixels is counted to obtain the probability of dark pixels.

[0027] Optionally, in the image bad pixel removal method, the method of calculating the gradient of the current pixel in the cache window to determine the edge type of the current pixel includes:

[0028] Extract a cache window of size M×N, where M and N are odd numbers and the current pixel is located in the middle row and middle column of the cache window;

[0029] Convolving the cache window with a first convolution kernel to obtain a first gradient window;

[0030] Obtain the pixel absolute value of each pixel in the first gradient window, and extract the minimum pixel absolute value in each row to obtain M minimum values;

[0031] Use M minimum values ​​to find the horizontal gradient of the current pixel;

[0032] Convolving the cache window with a second convolution kernel to obtain a second gradient window;

[0033] Obtain the pixel absolute value of each pixel point in the second gradient window, and extract the minimum pixel absolute value in each column to obtain N minimum values;

[0034] Use N minimum values ​​to find the vertical gradient of the current pixel;

[0035] The horizontal and vertical gradients of the current pixel are used to determine the edge type of the current pixel.

[0036] Optionally, in the image bad pixel removal method, the first convolution kernel is [-1, 0, 1]; the second convolution kernel is

[0037] Optionally, in the image bad pixel removal method, the method of using M minimum values ​​to obtain the horizontal gradient includes: obtaining the maximum value among the M minimum values; multiplying the maximum value by the pixel value sign of the current pixel point in the first gradient window to obtain the horizontal gradient;

[0038] The method for obtaining the vertical gradient using N minimum values ​​includes: obtaining the maximum value among the N minimum values; and multiplying the maximum value by the pixel value sign of the current pixel point in the second gradient window to obtain the vertical gradient.

[0039] Optionally, in the image bad pixel removal method, the method of using the horizontal gradient and vertical gradient of the current pixel to determine the edge type of the current pixel includes:

[0040] Calculate the pixel difference within the cache window;

[0041] The edge strength ES of the current pixel is calculated using the horizontal gradient and vertical gradient of the current pixel, which is expressed as:

[0042]

[0043] Where Dx represents the horizontal gradient of the current pixel, and Dy represents the vertical gradient of the current pixel;

[0044] If the edge strength is greater than the first edge threshold, it is determined that the current pixel is located in the edge area;

[0045] If the edge intensity is less than or equal to the first edge threshold, and the pixel difference is less than the second edge threshold, it is determined that the current pixel is located in a flat area;

[0046] If the edge intensity is less than or equal to the first edge threshold, and the pixel difference is greater than or equal to the second edge threshold, it is determined that the current pixel is located in the texture area.

[0047] Optionally, in the image bad pixel removal method, the method for calculating the pixel difference in the cache window includes:

[0048] The difference between the ath largest pixel value and the bth smallest pixel value in the green pixels in the cache window is calculated to obtain a pixel difference, wherein a and b are adjustable positive integers.

[0049] Optionally, in the image bad pixel removal method, the method of removing bad pixels from the current pixel according to the bad pixel probability and the edge type of the current pixel includes:

[0050] According to the bad pixel probability and the edge type of the current pixel, the upper and lower limits of the pixel value after removing the bad pixels in the cache window are calculated;

[0051] The upper and lower limits of the pixel values ​​after removing bad pixels in the cache window are used to correct the bad pixels of the current pixel to obtain the corrected pixel value o of the current pixel, which is expressed as:

[0052] o=max(min(uL,ct),dL)

[0053] Among them, uL represents the upper limit of the pixel value after bad pixels are removed from the cache window, dL represents the lower limit of the pixel value after bad pixels are removed from the cache window, and ct represents the pixel value of the current pixel before correction.

[0054] Optionally, in the image bad pixel removal method, the method for calculating the upper and lower limits of pixel values ​​after bad pixels are removed from the cache window according to the bad pixel probability and the edge type of the current pixel includes:

[0055] If the current pixel is located in a flat area, U pixels in the same channel as the current pixel are extracted from the cache window, wherein the extracted U pixels do not include the current pixel;

[0056] Sort the pixel values ​​of the extracted U pixels from large to small to obtain a sequence pS;

[0057] Calculate the probability that there are k1 bright bad pixels among the extracted U pixels And the probability that there are k2 dark pixels It is expressed as:

[0058]

[0059] Wherein, pL represents the probability of bright bad pixels in the reference pixel row, and pD represents the probability of dark bad pixels in the reference pixel row; the values ​​of k1 and k2 are 1, 2, ..., U-1;

[0060] If the probability of k1 bright bad pixels is less than the first judgment threshold, it is considered that there are k1 bright bad pixels;

[0061] If the probability of k2 dark bad pixels is less than the second judgment threshold, it is considered that there are k2 dark bad pixels;

[0062] Calculate the upper limit uL and lower limit dL of the pixel value after removing bad pixels in the cache window, expressed as:

[0063] uL=pS[k1]

[0064] dL=pS[U-1-k2].

[0065] Optionally, in the image bad pixel removal method, the method for calculating the upper and lower limits of pixel values ​​after bad pixels are removed from the cache window according to the bad pixel probability and the edge type of the current pixel includes:

[0066] If the current pixel is located in the texture area, U pixels of the same channel as the current pixel are extracted from the cache window, wherein the extracted U pixels do not include the current pixel;

[0067] Sort the pixel values ​​of the extracted U pixels from large to small to obtain a sequence pS;

[0068] The upper limit uL and lower limit dL of the pixel value after removing bad pixels in the calculation cache window are expressed as:

[0069] uL=pS[0]

[0070] dL=pS[U-1].

[0071] Optionally, in the image bad pixel removal method, the method for calculating the upper and lower limits of pixel values ​​after bad pixels are removed from the cache window according to the bad pixel probability and the edge type of the current pixel includes:

[0072] If the current pixel is located in the edge area, the edge direction θ is calculated using the gradient of the current pixel in the cache window, and the distance vector v1 between the current pixel and the point in the same channel is calculated, which is expressed as:

[0073]

[0074] v1=(xx,yy)

[0075] Where Dx represents the horizontal gradient of the current pixel, and Dy represents the vertical gradient of the current pixel;

[0076] According to the edge direction and the distance vector, the edge direction rotation distance vector v2 is calculated, which is expressed as:

[0077] v2=(xx×cos(θ)-yy×sin(θ),xx×sin(θ)+yy×cos(θ))

[0078] Calculate the modulus of the edge direction rotation distance vector v2;

[0079] In the cache window, extract J points whose modulus is less than the modulus threshold;

[0080] Sort the pixel values ​​of the extracted J pixels from large to small to obtain a sequence pX;

[0081] Calculate the probability that there are k3 bright bad pixels among the J pixels extracted And the probability of k4 dark pixels remaining It is expressed as:

[0082]

[0083] Wherein, pL represents the probability of bright bad pixels in the reference pixel row, and pD represents the probability of dark bad pixels in the reference pixel row; the values ​​of k3 and k4 are 1, 2, ..., J-1;

[0084] If the probability of k3 bright bad pixels is less than the third judgment threshold, it is considered that there are k3 bright bad pixels;

[0085] If the probability of k4 dark bad pixels is less than the fourth judgment threshold, it is considered that there are k4 dark bad pixels;

[0086] Calculate the upper limit uL and lower limit dL of the pixel value after removing bad pixels in the cache window, expressed as:

[0087] uL=pX[k3]

[0088] dL=pX[J-1-k4].

[0089] In order to solve the above technical problems, the present invention also provides an image bad pixel removal device, comprising a memory, a processor and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image bad pixel removal method as described in any one of the above items is executed.

[0090] The present invention provides a method and device for removing bad pixels from an image, comprising: calculating a noise estimation value using the pixel values ​​of a reference pixel row in an image sensor; screening bad pixels according to the noise estimation value to obtain a bad pixel probability; calculating the gradient of the current pixel in a cache window to determine the edge type of the current pixel; and removing bad pixels from the current pixel according to the bad pixel probability and the edge type of the current pixel. By calculating the noise estimation value using a reference pixel row, the bad pixel probability is obtained; the edge type of the current pixel is determined by calculating the gradient; and the strength of the bad pixel removal algorithm is adaptively adjusted according to the bad pixel probability and the edge type, thereby achieving effective bad pixel removal while ensuring image clarity, and solving the problem of how to effectively remove bad pixels from an image while ensuring image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 A schematic diagram of the arrangement of reference pixels and effective pixels in the image sensor provided in this embodiment;

[0092] Figure 2 This is a flow chart of the image bad pixel removal method provided in this embodiment. DETAILED DESCRIPTION

[0093] The following is a further detailed description of the image bad pixel removal method and device proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the accompanying drawings is often a part of the actual structure. In particular, the emphasis of each accompanying drawing is different, and sometimes different proportions are used.

[0094] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects in order to describe the embodiments of the present invention, rather than to describe a specific order or sequence. It should be understood that the structures used in this way can be interchanged under appropriate circumstances. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0095] In image sensors, such as Figure 1 As shown in FIG. 1 , the pixel array includes effective pixels (white boxes) and reference pixels (gray boxes) located at the edge of the effective pixels. The reference pixels cannot be normally sensed and imaged, and are generally used to correct dark levels.

[0096] Based on this, this embodiment uses reference pixels to propose a method for removing bad pixels from an image. Figure 2 As shown, including:

[0097] S1, calculating a noise estimate using pixel values ​​of a reference pixel row in an image sensor;

[0098] S2, screening bad pixels according to the noise estimation value to obtain the bad pixel probability;

[0099] S3, calculating the gradient of the current pixel in the cache window to determine the edge type of the current pixel;

[0100] S4, removing bad pixels from the current pixel according to the bad pixel probability and the edge type of the current pixel.

[0101] The image bad pixel removal method provided in this embodiment calculates the noise estimation value by using the reference pixel row, and then obtains the bad pixel probability; determines the edge type of the current pixel by calculating the gradient; and adaptively adjusts the strength of the bad pixel removal algorithm according to the bad pixel probability and the edge type, thereby achieving effective bad pixel removal while ensuring image clarity, and solving the problem of how to effectively remove image bad pixels while ensuring image quality.

[0102] Specifically, in this embodiment, in step S1, the method of calculating the noise estimation value using the pixel values ​​of the reference pixel row in the image sensor includes:

[0103] S11, obtaining expected pixel values ​​of a reference pixel row in an image sensor.

[0104] Specifically, in the process of correcting the dark level, the expected pixel value E of the reference pixel is counted. The expected pixel value E obtained by counting in the process of dark level correction can be directly retrieved. In the process of correcting the dark level, the method of counting the expected pixel value E of the reference pixel is well known to those skilled in the art, and this application will not be repeated here.

[0105] S12, counting the ratio of the number of reference pixel points whose pixel values ​​are the expected pixel values ​​in the reference pixel row to the total number of reference pixel points to obtain a reference ratio.

[0106] Specifically, in this embodiment, the reference pixels have a total of h rows and w columns, that is, the total number of reference pixels is h×w. By obtaining the pixel value of each reference pixel in the current frame image and comparing it with the expected pixel value E, if the pixel value of the reference pixel is the expected pixel value E, it is included in the statistics. Assuming that the number of reference pixels whose pixel value is the expected pixel value E is T, the reference ratio r can be expressed as:

[0107]

[0108] S13, based on the probability distribution of noise, obtaining a functional relationship between noise intensity and reference ratio.

[0109] Specifically, in this embodiment, the probability distribution of noise in the reference pixel point is a normal distribution, and the expected value of the reference ratio r of the current frame image can be expressed as:

[0110]

[0111] Where s represents the noise intensity.

[0112] Of course, in other embodiments, the probability distribution of noise may also satisfy other functional relationships, and the present application does not limit this. The present application only takes the normal distribution as an example to illustrate the calculation method of the noise estimation value of the present application, but the protection scope of the present application is not limited to this. Those skilled in the art can obtain the relationship between the noise intensity and the reference ratio under other noise probability distributions under the technical enlightenment of the present application.

[0113] S14, calculating and obtaining a noise estimation value according to a functional relationship between the noise intensity and the reference ratio.

[0114] Specifically, yes

[0115]

[0116] By performing inverse function calculation, we can get the calculation function of noise intensity, namely:

[0117]

[0118] In practical applications, the noise intensity can be calculated by table lookup or polynomial fitting.

[0119] The noise intensity of the current frame can be directly used as the noise estimation value of the current frame, that is, the noise estimation value NzEst can be expressed as: NzEst=s.

[0120] Preferably, in order to smooth the estimation result of the noise in the video image, in this embodiment, multiple consecutive frames of images can be obtained, and the reference ratio of each frame of the image can be obtained respectively, and then the corresponding noise intensity can be calculated, and then all the calculated noise intensities can be weighted averaged to obtain the noise estimation value.

[0121] In a specific embodiment, three consecutive frames of images may be used to calculate the noise estimation value, wherein the noise intensity of the current frame is recorded as s, the noise intensity of the previous frame is recorded as sDly2, and the noise intensity of the previous frame is recorded as sDly1. Then, the obtained noise estimation value NzEst can be expressed as:

[0122] NzEst=(sDly1+sDly2+2×s) / 4

[0123] In this embodiment, the weight of the current frame is higher, and the weight of the previous frames is lower, so as to ensure that the strength of bad pixel removal meets the image requirements of the current frame. In other embodiments, the weights corresponding to each frame can be reasonably set according to actual needs, and this application does not limit this.

[0124] Of course, those skilled in the art can foresee that when more image frames are used, the strength of bad pixel removal is more stable, but the amount of calculation will increase, and when the scene changes suddenly, the response to the strength of bad pixels will be slower. Therefore, in specific applications, it is necessary to reasonably select the number of image frames used to obtain the noise estimation value according to actual needs and image scenes.

[0125] Furthermore, in this embodiment, in step S2, the method of screening bad pixels according to the noise estimation value to obtain the bad pixel probability includes:

[0126] S21, calculating the number of reference pixel points in the reference pixel row whose pixel values ​​are greater than a first preset multiple of the noise estimation value, and the number of reference pixel points whose pixel values ​​are less than a second preset multiple of the noise estimation value.

[0127] Specifically, the number of reference pixel points whose pixel values ​​are greater than a first preset multiple of the noise estimation value obtained by calculation and statistics is recorded as C1, and the number of reference pixel points whose pixel values ​​are less than a second preset multiple of the noise estimation value is recorded as C2.

[0128] In practical applications, the first preset multiple can be set to k, and the second preset multiple can be set to -k, which correspond to bright bad pixels and dark bad pixels respectively, wherein the value of k is a positive number. Of course, in other embodiments, the absolute values ​​of the first preset multiple and the second preset multiple can be unequal, so that the strictness of the criteria for determining bright bad pixels and dark bad pixels can be adjusted according to actual needs.

[0129] S22, counting the ratio of the number of reference pixels whose pixel values ​​are greater than the first preset multiple of the noise estimation value to the total number of reference pixels, so as to obtain the probability of bright bad pixels, that is, the probability pL of bright bad pixels can be expressed as:

[0130]

[0131] S23, counting the ratio of the number of reference pixels whose pixel values ​​are less than the second preset multiple of the noise estimation value to the total number of reference pixels, so as to obtain the probability of dark pixels, that is, the probability pD of dark pixels can be expressed as:

[0132]

[0133] Furthermore, in this embodiment, step S3, the method of calculating the gradient of the current pixel point in the cache window to determine the edge type of the current pixel point includes:

[0134] S31, extracting a cache window of size M×N, where M and N are odd numbers, and the current pixel is located in the middle row and middle column of the cache window, that is, the current pixel is located at the center of the cache window.

[0135] Specifically, M and N should be odd numbers greater than or equal to 3. For example, when the size of the cache window is 5×5, the current pixel is located in the 3rd row and 3rd column. In this way, it is ensured that there are the same number of pixels around the current pixel that can be used for gradient calculation, ensuring the accuracy of edge type judgment.

[0136] When the values ​​of M and N are larger, more data is involved in the calculation, the amount of calculation is larger, the resource consumption is larger, and at the same time, the judgment of the edge type is more accurate. In practical applications, the size of the cache window can be set according to actual needs.

[0137] S32: Convolve the cache window using a first convolution kernel to obtain a first gradient window.

[0138] Specifically, in this embodiment, the first convolution kernel is [-1, 0, 1], and a first gradient window of M×(N-2) is obtained.

[0139] S33, obtaining the pixel absolute value of each pixel point in the first gradient window, and extracting the minimum pixel absolute value in each row to obtain M minimum values.

[0140] S34, using the M minimum values ​​to obtain the horizontal gradient of the current pixel.

[0141] Specifically, in this embodiment, first, the maximum value among the M minimum values ​​is obtained; then, the maximum value is multiplied by the pixel value sign of the current pixel point in the first gradient window to obtain the horizontal gradient Dx, where the pixel value sign refers to positive (+) or negative (-).

[0142] S35, convolving the cache window using a second convolution kernel to obtain a second gradient window.

[0143] Specifically, in this embodiment, the second convolution kernel is At this time, a second gradient window of (M-2)×N is obtained.

[0144] S36, obtaining the absolute value of each pixel in the second gradient window, and extracting the minimum absolute value of the pixel in each column to obtain N minimum values.

[0145] S37, using the N minimum values ​​to obtain the vertical gradient of the current pixel.

[0146] Specifically, in this embodiment, first, the maximum value among the N minimum values ​​is obtained; then, the maximum value is multiplied by the pixel value sign of the current pixel point in the second gradient window to obtain the vertical gradient Dy, where the pixel value sign refers to positive (+) or negative (-).

[0147] S38, using the horizontal gradient and vertical gradient of the current pixel to determine the edge type of the current pixel.

[0148] Specifically, in this embodiment, first, the pixel difference in the cache window is calculated. In a specific embodiment, considering that in the pixel array, among R pixels, B pixels and G pixels, G pixels account for a larger proportion, and G pixels can better reflect the edge information of the image, therefore, the difference between the ath largest pixel value and the bth smallest pixel value in the green (G) pixel in the cache window is calculated to obtain the pixel difference dM, where a and b are adjustable positive integers. In other words, the pixel values ​​of all green pixels in the cache window are extracted and arranged from large to small, and the difference between the ath pixel value in the positive order and the bth pixel value in the reverse order is calculated to obtain the pixel difference dM.

[0149] Then, the edge strength ES of the current pixel is calculated using the horizontal gradient Dx and the vertical gradient Dy of the current pixel, which is expressed as:

[0150]

[0151] Finally, the edge area is determined based on the pixel difference dM and the edge strength ES:

[0152] If the edge strength ES is greater than the first edge threshold th1, that is, ES>th1, it is determined that the current pixel is located in the edge area;

[0153] If the edge strength ES is less than or equal to the first edge threshold th1, and the pixel difference dM is less than the second edge threshold th2, that is, ES≤th1 and dM<th2, then it is determined that the current pixel point is located in a flat area;

[0154] If the edge strength ES is less than or equal to the first edge threshold th1, and the pixel difference dM is greater than or equal to the second edge threshold th2, that is, ES≤th1 and dM≥th2, it is determined that the current pixel is located in the texture area.

[0155] The first edge threshold th1 and the second edge threshold th2 can be reasonably set according to actual needs.

[0156] In practical applications, the order of implementation of steps S32 to S34 can be swapped with that of steps S35 to S37, or both can be performed simultaneously. The scheme after the order of steps is swapped without violating the subject matter of this application should also fall within the scope of protection of this application.

[0157] Further, in this embodiment, in step S4, the method of removing bad pixels from the current pixel according to the bad pixel probability and the edge type of the current pixel includes:

[0158] S41, calculating the upper limit uL and the lower limit dL of the pixel value after removing the bad pixels in the cache window according to the bad pixel probability and the edge type of the current pixel.

[0159] Specifically, in this embodiment, if the current pixel is located in a flat area, then: first, U pixels in the same channel as the current pixel are extracted from the cache window, wherein the extracted U pixels do not include the current pixel; then, the pixel values ​​of the extracted U pixels are sorted from large to small to obtain a sequence pS; then, the probability that there are k1 bright bad pixels among the extracted U pixels is calculated. And the probability that there are k2 dark pixels It is expressed as:

[0160]

[0161] Wherein, pL represents the probability of bright bad pixels in the reference pixel row, and pD represents the probability of dark bad pixels in the reference pixel row; the values ​​of k1 and k2 are 1, 2, ..., U-1;

[0162] If the probability of k1 bright bad pixels is less than the first judgment threshold th3, that is, It is considered that there are k1 bright bad pixels;

[0163] If the probability of k2 dark pixels is less than the second judgment threshold th4, that is, It is considered that there are k2 dark spots;

[0164] Finally, the upper limit uL and lower limit dL of the pixel value after removing bad pixels in the cache window are calculated, which can be expressed as:

[0165] uL=pS[k1]

[0166] dL=pS[U-1-k2]

[0167] For the flat area, since the pixel values ​​in the sequence pS are arranged from large to small, the k1th pixel value in the sequence pS can represent the dividing point between bright bad pixels and normal pixels, that is, the upper limit of the pixel value for distinguishing bad pixels; the U-1-k2th pixel value can represent the dividing point between dark bad pixels and normal pixels, that is, the lower limit of the pixel value for distinguishing bad pixels.

[0168] If the current pixel is located in the texture area, then: first, extract U pixels of the same channel as the current pixel from the cache window, where the extracted U pixels do not include the current pixel; then, sort the pixel values ​​of the extracted U pixels from large to small to obtain the sequence pS; finally, calculate the upper limit uL and lower limit dL of the pixel value after removing the bad pixels in the cache window, which are expressed as:

[0169] uL=pS[0]

[0170] dL=pS[U-1]

[0171] For the texture area, the upper limit uL and lower limit dL of the pixel value are fixed, that is, the maximum and minimum pixel values ​​of the extracted U pixels, respectively, so as to ensure that the texture features of the image can be effectively preserved after the bad pixels are removed, thereby ensuring the output image quality.

[0172] If the current pixel is located in the edge area, then: First, use the gradient of the current pixel in the cache window to calculate the edge direction θ, and calculate the distance vector v1 between the current pixel and the point in the same channel, expressed as:

[0173]

[0174] v1=(xx,yy)

[0175] Where Dx represents the horizontal gradient of the current pixel, and Dy represents the vertical gradient of the current pixel;

[0176] Then, according to the edge direction and the distance vector, the edge direction rotation distance vector v2 is calculated, which is expressed as:

[0177] v2=(xx×cos(θ)-yy×sin(θ),xx×sin(θ)+yy×cos(θ))

[0178] Next, calculate the modulus V of the edge direction rotation distance vector v2; then, extract J points whose modulus V is less than the modulus threshold Vth in the cache window; then, sort the pixel values ​​of the extracted J pixels from large to small to obtain the sequence pX; then, similar to the calculation method of the flat area, calculate the probability that there are k3 bright bad pixels among the extracted J pixels And the probability of k4 dark pixels remaining It is expressed as:

[0179]

[0180] Wherein, pL represents the probability of bright bad pixels in the reference pixel row, and pD represents the probability of dark bad pixels in the reference pixel row; the values ​​of k3 and k4 are 1, 2, ..., J-1;

[0181] If the probability of k3 bright bad pixels is less than the third judgment threshold th5, that is, It is considered that there are k3 bright bad pixels;

[0182] If the probability of k4 dark pixels is less than the fourth judgment threshold th6, that is, It is considered that there are k4 dark bad pixels;

[0183] Finally, the upper limit uL and lower limit dL of the pixel value after removing bad pixels in the cache window are calculated, which can be expressed as:

[0184] uL=pX[k3]

[0185] dL=pX[J-1-k4]

[0186] For edge areas, by extracting pixels by rotating the modulus length V of the distance vector v2 according to the edge direction, the edge information around the current pixel can be retained, so that the extracted pixels of the same channel and the current pixel belong to the same side edge, ensuring the consistency of pixel values. At the same time, since the pixel values ​​in the sequence pX are arranged from large to small, the k3th pixel value in the sequence pX can represent the dividing point between bright bad pixels and normal pixels, that is, the upper limit of the pixel value for distinguishing bad pixels; the J-1-k4th pixel values ​​can represent the dividing point between dark bad pixels and normal pixels, that is, the lower limit of the pixel value for distinguishing bad pixels.

[0187] In practical applications, the first judgment threshold th3, the second judgment threshold th4, the third judgment threshold th5 and the fourth judgment threshold th6 may be reasonably set according to the bad pixel performance of the image sensor and the strength requirement for removing the bad pixels.

[0188] S42, using the upper and lower limits of the pixel value after bad pixels are removed from the cache window, bad pixel correction is performed on the current pixel to obtain a corrected pixel value o of the current pixel, which is expressed as:

[0189] o=max(min(uL,ct),dL)

[0190] Among them, uL represents the upper limit of the pixel value after bad pixels are removed from the cache window, dL represents the lower limit of the pixel value after bad pixels are removed from the cache window, and ct represents the pixel value of the current pixel before correction.

[0191] The image bad pixel removal method provided by the present embodiment calculates the noise estimation value by using the reference pixel row, thereby obtaining the bad pixel probability, and determines the edge type of the current pixel by calculating the gradient, thereby distinguishing whether the area where the current pixel is located is a flat area, a texture area, or an edge area; thereby, according to the different areas where the current pixel is located, the strength (upper limit and lower limit) of the bad pixel removal algorithm is adaptively adjusted, and the bad pixel removal process is performed, thereby achieving effective removal of bad pixels while ensuring clear image texture details, and solving the problem of how to effectively remove image bad pixels while ensuring image quality.

[0192] This embodiment also provides an image bad pixel removal device, comprising a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image bad pixel removal method described above is executed.

[0193] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. In addition, the different parts between the various embodiments can also be used in combination with each other, and the present invention is not limited to this.

[0194] The method and device for removing bad pixels from an image provided by the present embodiment include: calculating a noise estimation value using the pixel values ​​of a reference pixel row in an image sensor; filtering bad pixels according to the noise estimation value to obtain a bad pixel probability; calculating the gradient of the current pixel in the cache window to determine the edge type of the current pixel; and removing bad pixels from the current pixel according to the bad pixel probability and the edge type of the current pixel. By calculating the noise estimation value using the reference pixel row, and then obtaining the bad pixel probability; determining the edge type of the current pixel by calculating the gradient; and adaptively adjusting the strength of the bad pixel removal algorithm according to the bad pixel probability and the edge type, the bad pixels are effectively removed while ensuring image clarity, solving the problem of how to effectively remove bad pixels from an image while ensuring image quality.

[0195] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A method for removing bad pixels from an image, characterized in that: include: calculating a noise estimate using pixel values ​​of a reference pixel row in the image sensor; Filter bad pixels according to the noise estimation value to obtain the bad pixel probability; Calculate the gradient of the current pixel in the cache window to determine the edge type of the current pixel; According to the bad pixel probability and the edge type of the current pixel, the bad pixel of the current pixel is removed.

2. The image bad pixel removal method according to claim 1, characterized in that: The method for calculating the noise estimation value using the pixel mean of the reference pixel row in the image sensor comprises: Obtaining pixel expected values ​​for a reference pixel row in an image sensor; Counting the ratio of the number of reference pixel points whose pixel values ​​are the expected pixel values ​​to the total number of reference pixel points in the reference pixel row to obtain a reference ratio; Based on the probability distribution of noise, the functional relationship between noise intensity and reference ratio is obtained; The noise estimation value is calculated based on the functional relationship between the noise intensity and the reference ratio.

3. The image bad pixel removal method according to claim 2, characterized in that: When the probability distribution of noise in the reference pixel is normal distribution, the functional relationship between the obtained noise intensity and the reference ratio is expressed as: Among them, r represents the reference ratio, T represents the number of reference pixels whose pixel values ​​are the expected pixel values, w represents the number of reference pixels in the length direction of the reference pixel row, h represents the number of reference pixels in the height direction of the reference pixel row; s represents the noise intensity.

4. The image bad pixel removal method according to claim 2, characterized in that: The method for calculating the noise estimation value according to the functional relationship between the noise intensity and the reference ratio includes: Obtaining a reference ratio of at least one frame of image; Substituting the obtained reference ratio into the functional relationship between the noise intensity and the reference ratio, the corresponding noise intensity is calculated; A weighted average calculation is performed on all the calculated noise intensities to obtain a noise estimation value.

5. The image bad pixel removal method according to claim 1, characterized in that: The method of screening bad pixels according to the noise estimation value to obtain the bad pixel probability includes: Calculate the number of reference pixel points in the reference pixel row whose pixel values ​​are greater than a first preset multiple of the noise estimation value, and the number of reference pixel points whose pixel values ​​are less than a second preset multiple of the noise estimation value; Counting the ratio of the number of reference pixels whose pixel values ​​are greater than the first preset multiple of the noise estimation value to the total number of reference pixels, so as to obtain the probability of bright bad pixels; The ratio of the number of reference pixels whose pixel values ​​are less than the second preset multiple of the noise estimation value to the total number of reference pixels is counted to obtain the probability of dark pixels.

6. The image bad pixel removal method according to claim 1, characterized in that: The method of calculating the gradient of the current pixel point in the cache window to determine the edge type of the current pixel point includes: Extract a cache window of size M×N, where M and N are odd numbers and the current pixel is located in the middle row and middle column of the cache window; Convolving the cache window with a first convolution kernel to obtain a first gradient window; Obtain the pixel absolute value of each pixel in the first gradient window, and extract the minimum pixel absolute value in each row to obtain M minimum values; Use M minimum values ​​to find the horizontal gradient of the current pixel; Convolving the cache window with a second convolution kernel to obtain a second gradient window; Obtain the pixel absolute value of each pixel point in the second gradient window, and extract the minimum pixel absolute value in each column to obtain N minimum values; Use N minimum values ​​to find the vertical gradient of the current pixel; The horizontal and vertical gradients of the current pixel are used to determine the edge type of the current pixel.

7. The image bad pixel removal method according to claim 6, characterized in that: The first convolution kernel is [-1, 0, 1]; the second convolution kernel is 8. The image bad pixel removal method according to claim 6, characterized in that: The method for obtaining the horizontal gradient using M minimum values ​​includes: obtaining the maximum value among the M minimum values; multiplying the maximum value by the pixel value sign of the current pixel point in the first gradient window to obtain the horizontal gradient; The method for obtaining the vertical gradient using N minimum values ​​includes: obtaining the maximum value among the N minimum values; and multiplying the maximum value by the pixel value sign of the current pixel point in the second gradient window to obtain the vertical gradient.

9. The image bad pixel removal method according to claim 6, characterized in that: The method for determining the edge type of the current pixel point by using the horizontal gradient and the vertical gradient of the current pixel point comprises: Calculate the pixel difference within the cache window; The edge strength ES of the current pixel is calculated using the horizontal gradient and vertical gradient of the current pixel, which is expressed as: Where Dx represents the horizontal gradient of the current pixel, and Dy represents the vertical gradient of the current pixel; If the edge strength is greater than the first edge threshold, it is determined that the current pixel is located in the edge area; If the edge intensity is less than or equal to the first edge threshold, and the pixel difference is less than the second edge threshold, it is determined that the current pixel is located in a flat area; If the edge intensity is less than or equal to the first edge threshold, and the pixel difference is greater than or equal to the second edge threshold, it is determined that the current pixel is located in the texture area.

10. The image bad pixel removal method according to claim 9, characterized in that: The method for calculating the pixel difference in the buffer window comprises: The difference between the ath largest pixel value and the bth smallest pixel value in the green pixels in the cache window is calculated to obtain a pixel difference, wherein a and b are adjustable positive integers.

11. The image bad pixel removal method according to claim 1, characterized in that: The method for removing bad pixels from the current pixel according to the bad pixel probability and the edge type of the current pixel includes: According to the bad pixel probability and the edge type of the current pixel, the upper and lower limits of the pixel value after removing the bad pixels in the cache window are calculated; The upper and lower limits of the pixel values ​​after removing bad pixels in the cache window are used to correct the bad pixels of the current pixel to obtain the corrected pixel value o of the current pixel, which is expressed as: o=max(min(uL,ct),dL) Among them, uL represents the upper limit of the pixel value after bad pixels are removed from the cache window, dL represents the lower limit of the pixel value after bad pixels are removed from the cache window, and ct represents the pixel value of the current pixel before correction.

12. The image bad pixel removal method according to claim 11, characterized in that: The method for calculating the upper and lower limits of pixel values ​​after removing bad pixels from the cache window according to the bad pixel probability and the edge type of the current pixel includes: If the current pixel is located in a flat area, U pixels in the same channel as the current pixel are extracted from the cache window, wherein the extracted U pixels do not include the current pixel; Sort the pixel values ​​of the extracted U pixels from large to small to obtain a sequence pS; Calculate the probability that there are k1 bright bad pixels among the extracted U pixels And the probability that there are k2 dark pixels It is expressed as: Wherein, pL represents the probability of bright bad pixels in the reference pixel row, and pD represents the probability of dark bad pixels in the reference pixel row; the values ​​of k1 and k2 are 1, 2, ..., U-1; If the probability of k1 bright bad pixels is less than the first judgment threshold, it is considered that there are k1 bright bad pixels; If the probability of k2 dark bad pixels is less than the second judgment threshold, it is considered that there are k2 dark bad pixels; Calculate the upper limit uL and lower limit dL of the pixel value after removing bad pixels in the cache window, expressed as: uL=pS[k1] dL=pS[U-1-k2].

13. The image bad pixel removal method according to claim 11, characterized in that: The method for calculating the upper and lower limits of pixel values ​​after removing bad pixels from the cache window according to the bad pixel probability and the edge type of the current pixel includes: If the current pixel is located in the texture area, U pixels of the same channel as the current pixel are extracted from the cache window, wherein the extracted U pixels do not include the current pixel; Sort the pixel values ​​of the extracted U pixels from large to small to obtain a sequence pS; The upper limit uL and lower limit dL of the pixel value after removing bad pixels in the calculation cache window are expressed as: uL=pS[0] dL=pS[U-1].

14. The image bad pixel removal method according to claim 11, characterized in that: The method for calculating the upper and lower limits of pixel values ​​after removing bad pixels from the cache window according to the bad pixel probability and the edge type of the current pixel includes: If the current pixel is located in the edge area, the edge direction θ is calculated using the gradient of the current pixel in the cache window, and the distance vector v1 between the current pixel and the point in the same channel is calculated, which is expressed as: v1=(xx,yy) Where Dx represents the horizontal gradient of the current pixel, and Dy represents the vertical gradient of the current pixel; According to the edge direction and the distance vector, the edge direction rotation distance vector v2 is calculated, which is expressed as: v2=(xx×cos(θ)-yy×sin(θ),xx×sin(θ)+yy×cos(θ)) Calculate the modulus of the edge direction rotation distance vector v2; In the cache window, extract J points whose modulus is less than the modulus threshold; Sort the pixel values ​​of the extracted J pixels from large to small to obtain a sequence pX; Calculate the probability that there are k3 bright bad pixels among the J pixels extracted And the probability of k4 dark pixels remaining It is expressed as: Wherein, pL represents the probability of bright bad pixels in the reference pixel row, and pD represents the probability of dark bad pixels in the reference pixel row; the values ​​of k3 and k4 are 1, 2, ..., J-1; If the probability of k3 bright bad pixels is less than the third judgment threshold, it is considered that there are k3 bright bad pixels; If the probability of k4 dark bad pixels is less than the fourth judgment threshold, it is considered that there are k4 dark bad pixels; Calculate the upper limit uL and lower limit dL of the pixel value after removing bad pixels in the cache window, expressed as: uL=pX[k3] dL=pX[J-1-k4].

15. An image bad pixel removal device, comprising a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image bad pixel removal method according to any one of claims 1 to 14 is executed.

Images