Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus and an image processing method that can accurately correct an abnormal pixel.SOLUTION: An image processing apparatus is provided, comprising: an average value calculation unit that calculates an average value on the basis of pixel values of a plurality of first pixels including a target pixel; a standard deviation calculation unit that calculates the standard deviation of Poisson distribution on the basis of pixel values of a plurality of second pixels including the target pixel; a threshold calculation unit that calculates a threshold on the basis of the average value and the standard deviation; and a correction unit that, when the pixel value of the target pixel exceeds the threshold, corrects the target pixel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Imaging devices such as digital cameras use image sensors that convert incident light into electrical signals. In low-light imaging environments, fluctuations in the number of photons reaching the image sensor, i.e., Poisson noise based on the Poisson distribution, appear in the signal. Furthermore, image sensors may contain pixels with different output characteristics or pixels that output abnormally high or low signals, known as abnormal pixels.

[0003] After the signal is subjected to the Anscombe transform, Poisson noise can be removed using conventional methods such as a bilateral filter or a non-local means filter. However, because the pixel values ​​of abnormal pixels are values ​​that deviate from the Poisson distribution, it is difficult to remove them using the above methods.

[0004] Methods for correcting such abnormal pixels have been proposed. For example, in Patent Document 1, an image containing Poisson noise is divided into multiple segments and high-frequency components in the frequency domain are cut off, thereby reducing noise outside the boundaries in the frequency domain. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6909379 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 only performs processing in the frequency domain, and is not necessarily able to correct abnormal pixels with high precision.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing apparatus and an image processing method that can correct abnormal pixels with high accuracy. [Means for solving the problem]

[0008] According to one disclosure of the present specification, there is provided an image processing device comprising: an average value calculation unit that calculates an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest; a first standard deviation calculation unit that calculates a standard deviation of a Gaussian distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; a second standard deviation calculation unit that calculates the standard deviation of a Poisson distribution based on pixel values ​​of a plurality of third pixels including the pixel of interest; a third standard deviation calculation unit that calculates a mixed standard deviation by mixing the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on the average value; a threshold value calculation unit that calculates a threshold value based on the average value and the mixed standard deviation; and a correction unit that corrects the pixel of interest if the pixel value of the pixel of interest exceeds the threshold value.

[0009] According to one disclosure of the present specification, there is provided a method for detecting a pixel value including a step of calculating an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest, a step of calculating a standard deviation of a Gaussian distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest, a step of calculating a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of third pixels including the pixel of interest, a step of calculating a mixed standard deviation by mixing the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on the average value, and a step of calculating a threshold value based on the average value and the mixed standard deviation. and correcting the pixel of interest if the pixel value of the pixel of interest exceeds the threshold.

[0010] According to one disclosure of this specification, there is provided an image processing device comprising an average value calculation unit that calculates an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest, a standard deviation calculation unit that calculates the standard deviation of a Poisson distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest, a threshold value calculation unit that calculates a threshold value based on the average value and the standard deviation, and a correction unit that corrects the pixel of interest if the pixel value of the pixel of interest exceeds the threshold value.

[0011] According to one disclosure of this specification, there is provided an image processing method comprising the steps of: calculating an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest; calculating a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; calculating a threshold value based on the average value and the standard deviation; and correcting the pixel of interest if the pixel value of the pixel of interest exceeds the threshold value. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide an image processing device and an image processing method that can correct abnormal pixels with high accuracy. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram of an image processing device according to a first embodiment. [Figure 2] 3A to 3C are diagrams showing images to be corrected by the image processing device according to the first embodiment. [Figure 3] 10 is a flowchart of an image processing method according to a second embodiment. [Figure 4] FIG. 10 is a block diagram of an image processing device according to a third embodiment. [Figure 5] FIG. 11 is a diagram illustrating a method for calculating a ratio according to the third embodiment. [Figure 6] 10 is a flowchart of an image processing method according to a fourth embodiment. [Figure 7] FIG. 10 is a block diagram of an image processing system according to a fifth embodiment. [Figure 8]13 is a flowchart of an image processing method according to a sixth embodiment. [Figure 9] FIG. 13 is a block diagram of an image processing system according to a seventh embodiment. [Figure 10] FIG. 13 is a block diagram of an information processing device according to a seventh embodiment. [Figure 11] FIG. 13 is a diagram schematically illustrating an arrangement of pixels according to a seventh embodiment. [Figure 12] 13 is a flowchart of an information processing device according to a seventh embodiment. [Figure 13] 13A to 13C are diagrams schematically showing examples of pixel values ​​and pixel states in correction processing according to the seventh embodiment. [Figure 14] 13A to 13C are diagrams schematically showing examples of pixel values, pixel states, and data holding methods in correction processing according to the seventh embodiment. [Figure 15] 13 is a graph showing an example of threshold values ​​in the correction process according to the seventh embodiment. [Figure 16] 13A to 13C are diagrams schematically showing examples of pixel values ​​and pixel states in correction processing according to the seventh embodiment. [Figure 17] FIG. 13 is a block diagram of a device according to an eighth embodiment. [Figure 18] FIG. 13 is a block diagram of a device according to a ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] [First embodiment] Fig. 1 is a block diagram of an image processing device 1 according to this embodiment. Fig. 2 is a diagram showing an image F to be corrected by the image processing device 1.

[0015] Image F may be, for example, a grayscale image having a single channel, an RGB color image having three channels of RGB, or an image having a single channel in which R, Gb, Gr, and B are arranged in a Bayer array.

[0016] In the case of an RGB color image, the image processing device 1 performs correction processing independently on each of the three channels. In the case of a Bayer array image, the image processing device 1 converts the Bayer array into four channels of R, Gb, Gr, and B, and then performs correction processing independently on each of the four channels.

[0017] Image F is composed of a plurality of pixels P. The plurality of pixels P are arranged in a grid pattern with a plurality of rows and a plurality of columns. That is, the plurality of pixels P are arranged in a grid pattern in the X direction (row direction) and the Y direction (column direction). An abnormality determination is performed for all pixels P included in image F. When performing the abnormality determination, one pixel P is focused on. This pixel P is called the pixel of interest Pa. A region obtained by cutting out the pixel of interest Pa and the pixels P surrounding the pixel of interest Pa from the entire region of image F is called a partial region (window) R. The partial region R is a portion of the entire region of image F and is formed, for example, in a rectangular shape. Here, five pixels P are arranged in each of the X direction and the Y direction, and the partial region R is composed of a total of 25 pixels P. The pixel of interest Pa is located at the center of the partial region R.

[0018] The image processing device 1 performs correction processing on an image F output from, for example, a SPAD (Single Photon Avalanche Diode) sensor. The image F captured by the SPAD sensor has an advantage in terms of image quality in low-illuminance areas compared to images captured by a CMOS sensor. The probability distribution of pixel values ​​of pixels P included in low-illuminance areas can be modeled using a Poisson distribution. Therefore, the image processing device 1 improves the image quality of the image F by performing correction processing based on the Poisson distribution. The correction processing performed by the image processing device 1 will be described in detail below.

[0019] The image processing device 1 includes a line buffer memory 10, area acquisition units 21, 22, and 24, a mean value calculation unit 30, a Poisson standard deviation calculation unit (second standard deviation calculation unit) 40, a threshold calculation unit 50, and a correction unit 60.

[0020] The line buffer memory 10 is connected to the SPAD sensor, the area acquisition units 21, 22, and 24 are connected to the line buffer memory 10, and the average value calculation unit 30 is connected to the area acquisition unit 21. In addition, the Poisson standard deviation calculation unit 40 is connected to the area acquisition unit 22, the threshold value calculation unit 50 is connected to the average value calculation unit 30 and the Poisson standard deviation calculation unit 40, and the correction unit 60 is connected to the area acquisition unit 24 and the threshold value calculation unit 50.

[0021] The line buffer memory 10 reads pixel values ​​of several rows of pixels P from the image F output from the SPAD sensor and stores the pixel values ​​in a buffer. The line buffer memory 10 then enables the pixel values ​​to be read by subsequent area acquisition units 21, 22, and 24. The width of the buffer in the line buffer memory 10 can be the same as the width of the image F, and the height of the buffer in the line buffer memory 10 can be the same as the height of the partial area R.

[0022] The region acquisition unit 21 acquires pixel values ​​of a plurality of pixels P included in a partial region R1 centered on a pixel of interest Pa. The length of the partial region R1 in the X direction is w1, and the length in the Y direction is w1 (R1=w1×w1).

[0023] The region acquisition unit 22 acquires pixel values ​​of a plurality of pixels P included in a partial region R2 centered on the pixel of interest Pa. The length of the partial region R2 in the X direction is w2, and the length in the Y direction is w2 (R2=w2×w2).

[0024] The region acquisition unit 24 acquires pixel values ​​of multiple pixels P included in a partial region R3 centered on the target pixel Pa. The length of the partial region R3 in the X direction is w3, and the length in the Y direction is w3 (R3=w3×w3).

[0025] Here, w1, w2, and w3 are hyperparameters. The user can freely set the lengths w1, w2, and w3 of the partial regions within the range of the width and height of the line buffer memory 10. The lengths w1, w2, and w3 of the partial regions may all be the same length or may be different lengths. If all lengths are the same, the process for acquiring the partial regions can be standardized, which is advantageous in terms of processing speed and memory usage. Furthermore, the length in the X direction and the length in the Y direction of the partial regions may be the same length or may be different lengths.

[0026] The average value calculation unit 30 calculates an average value m based on the pixel values ​​of a plurality of pixels P included in the partial region R1, and outputs the average value m to the threshold value calculation unit 50. When the pixel value of the pixel P included in the partial region R1 is x1 (1) ,x2 (1) ,…,x N1 (1) In this case, the formula for calculating the average value m is the following formula (1).

number

[0027] The Poisson standard deviation calculation unit 40 calculates the standard deviation σ of the Poisson distribution based on the pixel values ​​of the plurality of pixels P included in the partial region R2. p and outputs it to the threshold calculation unit 50. If the pixel value of pixel P included in the partial region R2 is x1 (2) ,x2 (2) ,…,x N2 (2) If , the standard deviation of the Poisson distribution is σ p The calculation formula is the following formula (2).

number

[0028] The threshold calculation unit 50 calculates the mean value m output from the mean value calculation unit 30 and the standard deviation σ of the Poisson distribution output from the Poisson standard deviation calculation unit 40. p The threshold is calculated based on the mean value m and output to the correction unit 60. The threshold is calculated by multiplying the mean value m by the standard deviation σ of the Poisson distribution. pThe upper threshold t is calculated by adding u , and the mean value m and the standard deviation σ of the Poisson distribution p The lower threshold t is calculated by subtracting l The upper threshold t u The calculation formula is the following formula (3), and the lower threshold t l The calculation formula is the following formula (4).

number

number

[0029] Here, the coefficient α is a hyperparameter that the user can freely set. The smaller the coefficient α, the more likely the pixel of interest Pa is to be determined to be an abnormal pixel, but the more likely it is to be falsely detected. On the other hand, the larger the coefficient α, the less likely it is to be falsely detected, but the more unlikely it is that the pixel of interest Pa is to be determined to be an abnormal pixel.

[0030] The correction unit 60 calculates the pixel value of the pixel P included in the partial region R3 and the upper threshold value t u and the lower threshold t l The correction unit 60 outputs a correction value for the pixel of interest Pa based on the lower threshold t l If the pixel value of the target pixel Pa is less than the upper threshold t u If the pixel value of the pixel of interest Pa is equal to or greater than the upper threshold t, the correction unit 60 corrects the pixel value of the pixel of interest Pa. For example, the correction unit 60 outputs the median value of the pixel values ​​of the pixels P included in the partial region R3. In other words, since the pixel of interest Pa is an abnormal pixel, the correction unit 60 replaces the pixel value of the pixel of interest Pa included in the partial region R3 with the median value. On the other hand, the correction unit 60 l (lower threshold or more) and the pixel value of the target pixel Pa is equal to or greater than the upper threshold t u If it is less than the upper threshold, the pixel of interest Pa is not corrected. That is, since the pixel of interest Pa is a normal pixel, the correction unit 60 outputs the pixel value of the pixel of interest Pa.

[0031] Here, since the pixel values ​​of the abnormal pixels are values ​​that deviate from the Poisson distribution, it has been difficult to correct the abnormal pixels using conventional methods for correcting Poisson noise (for example, bilateral filters, non-local means filters, etc.). In contrast, the image processing device 1 according to this embodiment uses an upper limit threshold t calculated based on the standard deviation of the Poisson distribution. u and the lower threshold t l Since abnormal pixels are corrected based on the above, the abnormal pixels can be corrected with high accuracy. As a result, by correcting image F by combining a conventional method with the image processing device 1, both Poisson noise and abnormal pixels can be effectively corrected. Furthermore, since the image processing device 1 corrects abnormal pixels without converting image F into a frequency spectrum, the processing load can be reduced. Note that when a bilateral filter or a non-local means filter is used to treat Poisson noise, image F must be subjected to the Anscombe transform.

[0032] [Second embodiment] FIG. 3 is a flowchart of an image processing method according to this embodiment. Each step of the flowchart is executed by a program. In this embodiment, the functions of each component of the image processing device 1 are realized by the program. The program can be executed by an information processing device such as a computer. The information processing device includes a calculation unit, a storage unit that stores a program for operating the calculation unit, and an input / output unit that inputs and outputs an image F. The image F is, for example, an image captured by a SPAD sensor.

[0033] In step S100, the calculation unit acquires the pixel value of pixel P included in partial region R centered on pixel of interest Pa. Here, a common partial region R is acquired, rather than acquiring different regions individually like the partial regions R1, R2, and R3 described above.

[0034] In step S101, the calculation unit calculates an average value m of pixel values ​​based on the pixel values ​​of a plurality of pixels P included in the partial region R. For example, the average value m is calculated using the above-mentioned formula (1).

[0035] In step S102, the calculation unit calculates the standard deviation σ of the Poisson distribution based on the pixel values ​​of the plurality of pixels P included in the partial region R. p For example, the standard deviation σ of the Poisson distribution can be calculated using the above formula (2). p Calculate.

[0036] In step S103, the calculation unit calculates the mean value and the standard deviation σ of the Poisson distribution. p Based on the upper threshold t u and the lower threshold t l For example, the upper threshold t u and use the above formula (4) to find the lower threshold t l Calculate.

[0037] In step S104, the calculation unit determines whether the pixel value of the target pixel Pa exceeds the lower threshold t l is less than the upper threshold t u It is determined whether the pixel value of the target pixel Pa is equal to or greater than the lower threshold t l is less than the upper threshold t u If so, the process proceeds to step S105.

[0038] In step S105, the calculation unit calculates the median value of the pixel values ​​of the pixels P included in the partial region R as the correction value of the pixel of interest Pa.

[0039] In step S106, the calculation unit outputs the median value as the correction value for the pixel of interest Pa, and then ends the process.

[0040] In the above-mentioned step S104, the calculation unit calculates whether the pixel value of the target pixel Pa is lower than the lower threshold t l is equal to or greater than the upper threshold t u If it is less than 100, the process proceeds to step S107.

[0041] In step S107, the calculation unit outputs the pixel value of the pixel of interest Pa, and then ends the process.

[0042] 3 is a correction process for one pixel of interest Pa. When processing all pixels P in an image F, the correction process is performed on the pixel of interest Pa in each partial region R while the partial region R is moved using a sliding window.

[0043] As described above, according to the image processing method of this embodiment, the standard deviation σ of the Poisson distribution p The upper threshold t calculated based on u and the lower threshold t l Since the abnormal pixels are corrected based on the above, the abnormal pixels can be corrected with high accuracy.

[0044] [Third embodiment] 4 is a block diagram of an image processing device 1A according to this embodiment. In this embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof will be omitted as appropriate.

[0045] The image processing device 1A includes a line buffer memory 10, area acquisition units 21, 22, 23, and 24, an average value calculation unit 30, a Poisson standard deviation calculation unit 40, and a Gaussian standard deviation calculation unit (first standard deviation calculation unit) 70. The image processing device 1A further includes a weighting coefficient calculation unit (third standard deviation calculation unit) 80, a mixed standard deviation calculation unit (third standard deviation calculation unit) 90, a threshold calculation unit 50A, and a correction unit 60.

[0046] The line buffer memory 10 is connected to a photoelectric conversion device (not shown), the area acquisition units 21, 22, 23, and 24 are connected to the line buffer memory 10, and the average value calculation unit 30 is connected to the area acquisition unit 21. Furthermore, the Poisson standard deviation calculation unit 40 is connected to the area acquisition unit 22, the Gaussian standard deviation calculation unit 70 is connected to the area acquisition unit 23, and the weighting coefficient calculation unit 80 is connected to the average value calculation unit 30. Furthermore, the mixed standard deviation calculation unit 90 is connected to the Poisson standard deviation calculation unit 40, the Gaussian standard deviation calculation unit 70, and the weighting coefficient calculation unit 80. Furthermore, the threshold calculation unit 50A is connected to the average value calculation unit 30 and the mixed standard deviation calculation unit 90, and the correction unit 60 is connected to the area acquisition unit 24 and the threshold calculation unit 50A.

[0047] The region acquisition unit 21 acquires pixel values ​​of a plurality of pixels P included in a partial region R1 centered on a pixel of interest Pa. The length of the partial region R1 in the X direction is w1, and the length in the Y direction is w1 (R1=w1×w1).

[0048] The region acquisition unit 22 acquires pixel values ​​of a plurality of pixels P included in a partial region R2 centered on the pixel of interest Pa. The length of the partial region R2 in the X direction is w2, and the length in the Y direction is w2 (R1=w2×w2).

[0049] The region acquisition unit 23 acquires pixel values ​​of a plurality of pixels P included in a partial region R3 centered on the pixel of interest Pa. The length of the partial region R3 in the X direction is w3, and the length in the Y direction is w3 (R3=w3×w3).

[0050] The region acquisition unit 24 acquires pixel values ​​of multiple pixels P included in a partial region R4 centered on the target pixel Pa. The length of the partial region R4 in the X direction is w4, and the length in the Y direction is w4 (R4=w4×w4).

[0051] The average value calculation unit 30 calculates the average value m using the above-mentioned formula (1) based on the pixel values ​​of the plurality of pixels P included in the partial region R1, and outputs it to the weighting coefficient calculation unit 80 and the threshold value calculation unit 50A.

[0052] The Poisson standard deviation calculation unit 40 calculates the standard deviation σ of the Poisson distribution using the above-mentioned formula (2) based on the pixel values ​​of the plurality of pixels P included in the partial region R2. p and outputs it to the mixed standard deviation calculation unit 90.

[0053] The Gaussian standard deviation calculation unit 70 calculates the standard deviation σ of the Gaussian distribution based on the pixel values ​​of the plurality of pixels P included in the partial region R3. g and outputs it to the mixed standard deviation calculation unit 90. If the pixel value of pixel P included in the partial region R3 is x1 (3) ,x2 (3) ,…,x N3 (3) If , the standard deviation of the Gaussian distribution is σ gThe calculation formula is the following formula (5).

number

number

[0054] The weighting coefficient calculation unit 80 calculates a ratio p when adding the standard deviation of the Poisson distribution and the standard deviation of the Gaussian distribution based on the mean value m output from the mean value calculation unit 30. The weighting coefficient calculation unit 80 calculates the standard deviation σ of the Gaussian distribution based on the ratio p. g and the standard deviation of the Poisson distribution σ p The weighting coefficients for each of these are calculated and output to the mixed standard deviation calculation unit 90. The method for calculating the ratio p will be described later.

[0055] The mixed standard deviation calculation unit 90 calculates the standard deviation σ of the Poisson distribution according to the weighting coefficients. p and the standard deviation of the Gaussian distribution σ g The mixed standard deviation σ is calculated by adding the weighted values. t and outputs the mixed standard deviation σ to the threshold calculation unit 50A. t The calculation formula is the following formula (7).

number

[0056] Here, the standard deviation of the Poisson distribution is σ p and the standard deviation of the Gaussian distribution σ g Adding the mixed standard deviation σ t is calculated, it is possible to correct abnormal pixels taking into consideration both Gaussian noise and Poisson noise.

[0057] The threshold calculation unit 50A calculates the mean value m output from the mean value calculation unit 30 and the mixed standard deviation σ output from the mixed standard deviation calculation unit 90. tThe threshold is calculated based on the average value m and output to the correction unit 60. The threshold is calculated by multiplying the average value m by a constant and then multiplying the average value m by a mixed standard deviation σ t The upper threshold t is calculated by adding u , and the mixed standard deviation σ multiplied by a constant to the mean value m t The lower threshold t is calculated by subtracting l The upper threshold t u The calculation formula is the following formula (8), and the lower threshold t l The calculation formula is the following formula (9).

number

number

[0058] Here, the coefficient α for the constant multiplication is a hyperparameter, and the user can freely set the coefficient α. The smaller the coefficient α, the more likely the pixel of interest Pa is to be determined to be an abnormal pixel, but the more likely it is to be falsely detected. On the other hand, the larger the coefficient α, the fewer false detections there are, but the more unlikely it is that the pixel of interest Pa is to be determined to be an abnormal pixel.

[0059] The correction unit 60 calculates the pixel value of the pixel P included in the partial region R4 and the upper limit threshold t u and the lower threshold t l The correction unit 60 outputs a correction value of the pixel of interest Pa based on the lower threshold t l If it is less than the upper threshold t u If it is equal to or greater than the lower threshold t, the correction unit 60 corrects the pixel value of the pixel of interest Pa. For example, the correction unit 60 outputs the median value of the pixel values ​​of the pixels P included in the partial region R4. That is, since the pixel of interest Pa is an abnormal pixel, the correction unit 60 replaces the pixel value of the pixel of interest Pa included in the partial region R4 with the median value. On the other hand, the correction unit 60 also corrects the pixel value of the pixel of interest Pa if it is greater than the lower threshold t l is equal to or greater than the upper threshold t u If it is less than this, the pixel of interest Pa is not corrected. That is, since the pixel of interest Pa is a normal pixel, the corrector 60 outputs the pixel value of the pixel of interest Pa.

[0060] Next, a method for calculating the ratio p by the weighting coefficient calculation unit 80 will be described. The ratio p is a function of the mean value m (hereinafter referred to as the ratio function), and the value range of the ratio function can be from "0" to "1." When the mean value m is low and the illuminance is low, the ratio p of the standard deviation of the Poisson distribution is increased, and when the mean value m is high and the illuminance is high, the ratio p of the standard deviation of the Gaussian distribution is increased. In other words, as the mean value m becomes higher (larger), the standard deviation σ of the Gaussian distribution g For example, the ratio function can use the exponential curve formula shown in the following formula (10). Here, e is Napier's constant, and the constant c is a hyperparameter that the user can freely set.

number

[0061] Figure 5(a) is a graph of the ratio function of equation (10). The horizontal axis is the average value m, and the vertical axis is the ratio p. When the average value m is low, the ratio p is small, and when the average value m is high, the ratio p is large.

[0062] The ratio function may use the Plateau curve formula. Figure 5(b) is a graph of the ratio function of formula (11). The horizontal axis is the mean value m, and the vertical axis is the ratio p. Compared to the ratio function of formula (10), the ratio function of formula (11) has a larger ratio p when the mean value m is low. This places emphasis on the Gaussian distribution, making it advantageous when Gaussian distribution noise is included even in low illuminance.

number

[0063] The ratio function may use a linear clip curve formula. Figure 5(c) is a graph of the ratio function of formula (12). The horizontal axis is the average value m, and the vertical axis is the ratio p. The ratio function of formula (12) has an advantage in implementation compared to the ratio function of formula (11), because the slope of the linear portion can be set by changing the constant c, which is a hyperparameter.

number

[0064] The method for calculating the ratio p from the average value m may be a method other than using the ratio function as described above. For example, it may be a method of approximation using a lookup table that associates the average value m with the ratio p.

[0065] Since the pixel values ​​of abnormal pixels are values ​​that deviate from the Poisson distribution and Gaussian distribution, it has been difficult to correct abnormal pixels using conventional methods for correcting Poisson noise and Gaussian noise (for example, bilateral filters, non-local means filters, etc.). In contrast, the image processing device 1A according to this embodiment uses the standard deviation σ of the Poisson distribution to correct the abnormal pixels. p and the standard deviation of the Gaussian distribution σ g The mixed standard deviation σ t The upper threshold t calculated based on u and the lower threshold t l Since abnormal pixels are corrected based on the above, the abnormal pixels can be corrected with high accuracy. As a result, by correcting image F by combining a conventional method with the image processing device 1, Gaussian noise, Poisson noise, and abnormal pixels can be effectively corrected. Furthermore, since the image processing device 1 corrects abnormal pixels without converting image F into a frequency spectrum, the processing load can be reduced.

[0066] [Fourth embodiment] FIG. 6 is a flowchart of an image processing method according to this embodiment. Each step of the flowchart is executed by a program. In this embodiment, the functions of each component of the image processing device 1A are realized by the program. The program can be executed by an information processing device such as a computer. The information processing device includes a calculation unit, a memory unit that stores a program for operating the calculation unit, and an input / output unit that inputs and outputs an image F.

[0067] In step S200, the calculation unit acquires the pixel value of pixel P included in partial region R centered on pixel of interest Pa. Here, instead of acquiring different regions individually like the partial regions R1, R2, R3, and R4 described above, a common partial region R is acquired.

[0068] In step S201, the calculation unit calculates an average value m based on the pixel values ​​of a plurality of pixels P included in the partial region R. For example, the average value m is calculated using the above-mentioned formula (1).

[0069] In step S202, the calculation unit calculates the standard deviation σ of the Poisson distribution based on the pixel values ​​of the plurality of pixels P included in the partial region R. p For example, the standard deviation σ of the Poisson distribution can be calculated using the above formula (2). p Calculate.

[0070] In step S203, the calculation unit calculates the standard deviation σ of the Gaussian distribution based on the pixel values ​​of the plurality of pixels P included in the partial region R. g For example, the standard deviation of the Gaussian distribution σ is calculated using the above formula (5). g Calculate.

[0071] In step S204, the calculation unit calculates a ratio p when adding the standard deviation of the Poisson distribution and the standard deviation of the Gaussian distribution based on the mean value m. For example, the calculation unit calculates the ratio p using the above-mentioned formula (10). Then, the calculation unit calculates the standard deviation σ of the Gaussian distribution based on the ratio p. g and the standard deviation of the Poisson distribution σ p Calculate the weighting coefficients for each of the above.

[0072] In step S205, the calculation unit calculates the standard deviation σ of the Poisson distribution. p and the standard deviation of the Gaussian distribution σ g are summed based on the weighting coefficients to obtain the mixed standard deviation σ t For example, the mixed standard deviation σ is calculated using the above formula (7). t Calculate.

[0073] In step S206, the calculation unit calculates the mean value m and the mixed standard deviation σ t Using the upper threshold t u and the lower threshold t l For example, the upper threshold value t u and use the above formula (9) to find the lower threshold t l Calculate.

[0074] In step S207, the calculation unit determines whether the pixel value of the target pixel Pa exceeds the lower threshold t l is less than the upper threshold t u It is determined whether the pixel value of the target pixel Pa is equal to or greater than the lower threshold t l is less than the upper threshold t u If so, the process proceeds to step S208.

[0075] In step S208, the calculation unit calculates the median value of the pixel values ​​of the pixels P included in the partial region R as the correction value of the pixel of interest Pa.

[0076] In step S209, the calculation unit outputs the median value as the correction value for the pixel of interest Pa, and then ends the process.

[0077] In step S207, the calculation unit calculates whether the pixel value of the target pixel Pa is lower than the lower threshold t l and is equal to or greater than the upper threshold t u If it is less than 1, the process proceeds to step S210.

[0078] In step S210, the calculation unit outputs the pixel value of the pixel of interest Pa, and then ends the process.

[0079] 6 is a correction process for one pixel of interest Pa. When processing all pixels P in an image F, the correction process is performed on the pixel of interest Pa in each partial region R while the partial region R is moved using a sliding window.

[0080] As described above, according to the image processing method of this embodiment, the standard deviation σ of the Poisson distribution p and the standard deviation of the Gaussian distribution σ g The mixed standard deviation σ t The upper threshold t calculated based on u and the lower threshold t l The image processing method corrects the abnormal pixels based on the above equation.

[0081] [Fifth embodiment] 7 is a block diagram of an image processing system 1B according to this embodiment. In this embodiment, the same components as those in other embodiments are denoted by the same reference numerals, and detailed descriptions thereof will be omitted as appropriate. The image processing system 1B according to this embodiment includes image processing devices 1Ba and 1Bb.

[0082] The image processing device 1Ba has the same configuration as the image processing device 1A described above. The values ​​of the hyperparameters are also the same as those of the image processing device 1A. The image processing device 1Ba outputs the corrected image F to the image processing device 1Bb.

[0083] The image processing device 1Bb performs correction processing on the image F output from the image processing device 1Ba. The image processing device 1Bb has the same configuration as the image processing device 1Ba described above. However, the values ​​of the hyperparameters differ from those of the image processing device 1Ba. For example, by changing the hyperparameters that set the partial regions, the size of the partial regions acquired by the region acquisition units 21-24 of the image processing device 1Bb is made different from the size of the partial regions acquired by the region acquisition units 21-24 of the image processing device 1Ba. By making the size of the partial regions of the image processing device 1Ba larger than the size of the partial regions of the image processing device 1Bb, the image processing device 1Ba can refer to a wide range and correct abnormal pixels that clearly deviate from the pixel values ​​of normal pixels. On the other hand, the image processing device 1Bb can refer to a narrower range and correct abnormal pixels that do not deviate too much from the pixel values ​​of normal pixels. This allows more abnormal pixels to be corrected.

[0084] Furthermore, the hyperparameter (coefficient α) used when calculating the threshold value may be different between the image processing devices 1Ba and 1Bb. For example, by making the coefficient α of the threshold calculation unit 50A of the image processing device 1Ba larger than the coefficient α of the threshold calculation unit 50A of the image processing device 1Bb, the image processing device 1Ba can correct abnormal pixels that clearly deviate from the pixel values ​​of normal pixels. On the other hand, the image processing device 1Bb can correct abnormal pixels that do not deviate too much from the pixel values ​​of normal pixels. This allows more abnormal pixels to be corrected.

[0085] Furthermore, in the image processing devices 1Ba and 1Bb, the hyperparameter (constant c) used to calculate the ratio p may be set to a different value, and further, the ratio function may be set to a different function.

[0086] When multiple adjacent abnormal pixels exist in the same partial region, it may be difficult to determine whether a pixel is abnormal using only one image processing device. According to the image processing system 1B of this embodiment, correction processing is repeated multiple times for all pixels P in an image F using image processing devices 1Ba and 1Bb, whose partial regions have different sizes. As a result, even when multiple adjacent abnormal pixels exist, the sizes of the partial regions are different, so adjacent abnormal pixels can be separated and determined, and the abnormal pixels can be corrected with high accuracy.

[0087] The image processing devices 1Ba and 1Bb may have the same hyperparameter values. Although the image processing system 1B has been described as including two image processing devices 1Ba and 1Bb, the present invention is not limited to this and may be configured to include three or more image processing devices.

[0088] [Sixth embodiment] FIG. 8 is a flowchart of an image processing method according to this embodiment. Each step of the flowchart is executed by a program. In this embodiment, the functions of each component of the image processing system 1B are realized by the program. The program can be executed by an information processing device such as a computer. The information processing device includes a calculation unit, a memory unit that stores a program for operating the calculation unit, and an input / output unit that inputs and outputs an image F.

[0089] In step S300, the calculation unit performs a first correction process, specifically steps S200 to S210 shown in FIG.

[0090] In step S301, the calculation unit performs a second correction process. Specifically, the calculation unit performs the same correction process as in step S300 on image F that has been subjected to the correction process in step S300. When performing the correction process in step S301, hyperparameters are set to values ​​different from those used in the correction process in step S300. For example, the size of the partial region, the constant c, and the coefficient α are set to values ​​different from those used in the correction process in step S300. Furthermore, a ratio function different from that used in the correction process in step S300 is used. Note that the hyperparameters and ratio function in step S301 may be the same as those used in step S300. Furthermore, although the correction process is repeated twice, in steps S300 and S301, the present invention is not limited to this and may be repeated three or more times.

[0091] [Seventh embodiment] FIG. 9 is a block diagram of an image processing system 1C according to this embodiment.

[0092] The image processing system 1C includes an information processing device 1D capable of correcting an image F, and the above-described image processing device 1A. Note that the image processing system 1C may include the above-described image processing devices 1 and 1B instead of the image processing device 1A.

[0093] The information processing device 1D performs correction processing based on pixel value concatenation on the image F and outputs the corrected image F to the image processing device 1A. The image processing device 1A performs correction processing on the corrected image F output from the information processing device 1D. The information processing device 1D will be described in detail below.

[0094] 10 is a block diagram of an information processing device 1D according to this embodiment. The information processing device 1D includes a pixel data storage unit 131, a correction data generation unit 132, a first pixel value determination unit 133, a first pixel state storage unit 134, a first connection unit 135, a correction unit 136, and a first connection number determination unit 137.

[0095] 11 to 16(b), the correction process executed by the information processing device 1D based on pixel value concatenation will be described. First, prior to describing the specific processing procedure, the configuration of pixel data acquired by the photoelectric conversion device and held in the pixel data holding unit 131 will be described.

[0096] Fig. 11 is a diagram schematically showing a pixel arrangement according to this embodiment. Fig. 11 schematically shows a plurality of pixels P11 to P56 constituting image data held in the pixel data holding unit 131, and the pixel values ​​of each pixel. The plurality of pixels P11 to P56 are arranged to form a plurality of rows and a plurality of columns. For the sake of simplicity, Fig. 11 shows only five rows and six columns of pixels, but in reality, more pixels may be arranged.

[0097] In the symbols assigned to pixels, the two numbers after "P" indicate the row number and column number, respectively. In Figure 11, the values ​​written in the circles representing pixels P11 to P56 indicate pixel values. For example, "20" is written in the circle for pixel P11, indicating that the pixel value of pixel P11 is 20.

[0098] However, a pixel having an abnormal pixel value is not necessarily a single, independent pixel. In other words, two or more abnormal pixels may be connected, so the connected abnormal pixels must be taken into consideration when correcting the abnormal pixels. Here, two abnormal pixels being connected means that the two abnormal pixels are in a positional relationship that is continuous in either the vertical, horizontal, or diagonal direction in Figure 11.

[0099] The following equation (13) indicates the probability that at least one of eight pixels in the vicinity of a certain abnormal pixel is an abnormal pixel.

number

[0100] In this case, the expected number of connected abnormal pixels is roughly expressed by the following equation (14).

number

[0101] For example, in an image with 4096 pixel columns and 2048 pixel rows, even if the percentage P is a very small value of 0.1%, the expected number of anomalous pixels calculated using equation (14) is approximately 67. Assuming that all of these anomalous pixels are pairs of two anomalous pixels and that no two anomalous pixels are connected, there will be approximately 33 pairs of connected anomalous pixels in the image. This corresponds to a situation where, if an image is divided into 32 8x4 blocks, each divided image will have approximately one pair of consecutive anomalous pixels. In other words, the probability of anomalous pixels being adjacent to each other cannot be ignored.

[0102] Next, we will discuss the probability that two or more abnormal pixels exist among the eight pixels surrounding an abnormal pixel, i.e., the probability that three or more abnormal pixels are connected. Equation (13) shows the probability that one or more abnormal pixels exist among the eight pixels surrounding an abnormal pixel. Therefore, by subtracting the probability that there is only one abnormal pixel among the eight pixels surrounding an abnormal pixel from equation (13), we can calculate the probability that two or more abnormal pixels exist among the eight pixels surrounding an abnormal pixel.

[0103] The probability that there is only one abnormal pixel among the eight pixels neighboring an abnormal pixel is expressed by the following equation (15).

number

[0104] Therefore, the expected number of occurrences of three or more connected abnormal pixels is roughly expressed by the following equation (16).

number

[0105] As an example, in an image with 4096 pixel columns and 2048 pixel rows, the expected value of the number of abnormal pixels with three or more connected pixels is calculated. When P=0.1%, the expected value of the number of abnormal pixels with three or more connected pixels is 0.234. When P=0.5%, the expected value of the number of abnormal pixels with three or more connected pixels is 28.8. When P=1.0%, the expected value of the number of abnormal pixels with three or more connected pixels is 226.

[0106] As described above, even if the ratio P is a small value of 1% or less, the possibility of three or more connected abnormal pixels cannot be ignored. Furthermore, depending on the values ​​of P, H, and W, it may be necessary to consider four or more connected abnormal pixels. As described above, taking into account the case where multiple connected abnormal pixels exist, the information processing device 1D based on pixel value connection performs correction processing for N or fewer connected abnormal pixels (N is a positive integer).

[0107] 12 is a flowchart showing the correction process executed by the information processing device 1D based on pixel value concatenation. The correction process method of this embodiment will be described with reference to the flowchart of FIG.

[0108] The process of Fig. 12 detects abnormal pixels from an image captured by a photoelectric conversion device and corrects the detected abnormal pixels. The process of Fig. 12 is started, for example, after the photoelectric conversion device captures an image and the image is stored in the pixel data storage unit 131. Note that the process of this embodiment does not use previously acquired position information of abnormal pixels (so-called flaw map) or an image captured in a light-shielded state (so-called dark image). The pixel data storage unit 131 is assumed to have a storage capacity sufficient to simultaneously store pixel values ​​for each of the multiple pixels in an entire image for one frame. Similarly, the first pixel state storage unit 134 is assumed to have a storage capacity sufficient to simultaneously store pixel states for an entire image for one frame.

[0109] In step S11, the first connecting unit 135 acquires the pixel values ​​of each of the multiple pixels that make up the image from the pixel data storage unit 131. Then, if the pixel values ​​of two adjacent pixels are close to each other (the difference between the pixel values ​​of the two adjacent pixels is within a first range), the first connecting unit 135 determines that the two pixels belong to the same first connected pixel group (first connecting process). The first pixel state storage unit 134 stores information indicating the determination result of the first connecting process. In this process, for each pixel, the pixel values ​​of a pixel region of three rows and three columns that includes the pixel to be processed (pixel of interest) and its eight neighboring pixels are referenced, and the pixel values ​​are compared and determined. The determination result is stored in the data region of the first pixel state storage unit 134 that corresponds to the referenced pixel region of three rows and three columns.

[0110] Hereinafter, the process in which the first connecting unit 135 determines that two pixels belong to the same first connected pixel group and the first pixel state holding unit 134 holds the pixel states of these pixels may be referred to as "connecting." After the connection process is complete, pixels with a connection count of N or less (N is a positive integer) are referred to as isolated points. Here, the connection count is the number of pixels that can be reached from a given pixel by tracing the connected pixels, i.e., the number of pixels that belong to the same first connected pixel group. For example, if a given pixel is not connected to any other pixel, the connection count is 1. The value of N indicates the upper limit of the number of abnormal pixel connections that are considered, and can be set appropriately depending on the required accuracy of abnormal pixel detection, image quality, etc.

[0111] First, an example of the first connection process when the threshold value N for the number of connections in determining isolated points is 1, i.e., when a pixel that is not connected to other pixels is considered an isolated point, will be explained with reference to Figures 13(a) to 15(d).

[0112] FIG. 13(a) is a diagram schematically showing an example of pixel values ​​in the correction process according to this embodiment. The notation in FIG. 13(a) is the same as in FIG. 11. FIG. 13(b) is a diagram schematically showing an example of pixel states in the correction process according to this embodiment. FIG. 13(b) schematically shows pixel state data stored in the first pixel state storage unit 134 for each pixel. The connecting line N1 indicates the relationship between two adjacent pixels P11 and P12, and indicates that the pixel value "20" of pixel P11 and the pixel value "22" of pixel P12 are determined to be close values ​​and are therefore connected.

[0113] In Figure 13(b), the numerical values ​​written in the eight outermost positions within the circle corresponding to each pixel are linked data indicating whether the pixel value of the pixel in question and the pixel values ​​of the adjacent pixels are similar. In the linked data, "0" indicates that the pixel value of the pixel in question and the pixel values ​​of the adjacent pixels in that direction are not similar and are not linked. Also, "1" indicates that the pixel value of the pixel in question and the pixel values ​​of the adjacent pixels in that direction are similar and are linked. Furthermore, for pixels at the edges, there are fewer than eight adjacent pixels. Thus, if there are no adjacent pixels, the value of the linked data is 0.

[0114] FIG. 14(a) is a diagram schematically illustrating an example of pixel values ​​in the vicinity of pixel P11. FIG. 14(b) is a diagram schematically illustrating an example of pixel states in the vicinity of pixel P11. As shown in FIG. 14(b), the right, bottom, and lower right sides of the connected data for pixel P11 are 1, and pixel P11 is connected to each of its adjacent pixels by connecting lines N1. Specifically, pixel P11 has pixel values ​​similar to those of pixels P12, P21, and P22, and therefore pixel P11 is connected to pixels P12, P21, and P22. Pixel P11 is a pixel located at an edge, and there are no adjacent pixels in any other direction. Therefore, the connected data for pixel P11 is 0 except for the right, bottom, and lower right sides.

[0115] Figures 14(c) and 14(d) are diagrams showing a method for storing concatenated data. The eight values ​​included in the concatenated data can be stored as 8-bit data by arranging them in a clockwise direction, for example, as shown in Figure 14(c), based on a value in a certain direction. Figure 14(d) shows the 8-bit data "00011100" ("1C" in hexadecimal) obtained in this way.

[0116] 13(a) and 13(b), the eight values ​​included in the connected data of the hatched pixels P15, P43, and P45 are all 0. In this way, pixels P15, P43, and P45, whose eight values ​​are all 0, are not connected to other pixels. Because the number of connections of such pixels is 1, i.e., N or less, pixels P15, P43, and P45 are isolated points.

[0117] 15(a) to 15(d) are graphs showing examples of thresholds used to determine whether the pixel values ​​of two adjacent pixels in the first connecting portion 135 are close to each other. Note that in comparisons of magnitude using thresholds described below, "greater than or equal to" and "greater than" can be interpreted interchangeably, and "less than or equal to" and "under" can also be interpreted interchangeably.

[0118] 15(a) to 15(d), the horizontal axis represents the pixel value of the pixel of interest, and the vertical axis represents the pixel values ​​of the neighboring pixels adjacent to the pixel of interest. In Figures 15(a) to 15(d), line L1 has a slope of 1 and an intercept of 0 (i.e., y = x), and represents the lower limit threshold of the pixel values ​​of the neighboring pixels to be connected.

[0119] In FIG. 15(a), line L2 indicates the upper threshold of the pixel values ​​of neighboring pixels to be connected. The slope of line L2 is greater than 1, and the intercept of line L2 is greater than 0. Furthermore, on line L2, when x = x1, y = y2. When the pixel value of the pixel of interest is x1, the pixel of interest and the neighboring pixel are connected if their pixel values ​​are greater than or equal to y1 (= x1) and less than or equal to y2. That is, the first connecting unit 135 connects two pixels when the pixel values ​​of the pixel of interest and the neighboring pixel are in the region between lines L1 and L2 (the difference between the pixel values ​​of the pixel of interest and the neighboring pixel is within a first range determined by the upper and lower thresholds). At this time, the first pixel state holding unit 134 holds a pixel state representing the connection state by holding "1" as the value of the corresponding bit in the connected data of the pixel of interest and the neighboring pixel. That is, a "1" is held in each of the bit indicating the direction of the neighboring pixel of the linked data corresponding to the pixel of interest and the bit indicating the direction of the pixel of interest of the linked data corresponding to the neighboring pixel.

[0120] The function for determining the threshold is not limited to that shown in Fig. 15(a). For example, the upper threshold may be determined by a broken line L3 composed of multiple line segments, as shown in Fig. 15(b). On the line L3, when x = x1, y = y3. When the pixel value of the pixel of interest is x1, the pixel of interest and the neighboring pixel are connected if the pixel value of the neighboring pixel is greater than or equal to y1 (= x1) and less than or equal to y3.

[0121] Alternatively, for example, the upper threshold may be given by a curve L4 as shown in Figure 15(c). On the line L4, when x = x1, y = y4. When the pixel value of the pixel of interest is x1, the pixel of interest and the neighboring pixel are connected if the pixel value of the neighboring pixel is greater than or equal to y1 (= x1) and less than or equal to y4.

[0122] Alternatively, for example, a lower threshold value may be given by a straight line L5 different from the straight line L1 as shown in FIG. 15(d). The slope of the straight line L5 is smaller than 1, and the intercept of the straight line L5 is smaller than 0. Also, on the straight line L5, when x = x1, y = y5. When the pixel value of the target pixel is x1, the target pixel and the neighboring pixel are connected when the pixel value of the neighboring pixel is y5 (<x1) or more and y2 or less.

[0123] In the example of FIG. 15(d), one or both of the straight line L2 and the straight line L5 may be replaced with a broken line or may be replaced with a curve. Also, the straight line L2 and the straight line L5 may be replaced with different types of lines.

[0124] A method for obtaining a threshold value (upper threshold value or lower threshold value) will be described. When the graph shape of the threshold value is simple, the first connection part 135 may calculate the threshold value by a function that takes the pixel value of the target pixel as an input and outputs the value of the threshold value. In this case, it is desirable to be able to adjust the function with the coefficients of the terms included in the function as parameters. Also, it is not essential that the mathematical formula of this function is the same throughout the entire graph. For example, this function may be a piecewise linear function. Also, the first connection part 135 may include a lookup table in which the pixel value of the target pixel and the threshold value are associated. In this case, the first connection part 135 refers to the pixel value of the target pixel as an index and obtains the threshold value. Here, the lookup table may hold values for all indexes, or may hold values only for indexes of some representative points. When the lookup table holds values only for indexes of representative points, the first connection part may calculate the threshold value for indexes other than the representative points by interpolation.

[0125] Referring again to FIG. 12, the processing after the first connection processing in step S11 will be described. In step S12 after the first connection processing in step S11 is completed, the first pixel value determination unit 133 determines whether the pixel value of a pixel of interest among the multiple pixels is unique (determination of first condition). If it is determined that the pixel value of the pixel of interest is unique (YES in step S12), the processing proceeds to step S13. If it is determined that the pixel value of the pixel of interest is not unique (NO in step S12), the processing proceeds to step S15. In this case, the pixel of interest is not a correction target.

[0126] The determination in step S12 will be explained in more detail. Note that this explanation also assumes that the value of N is 1, that is, that a pixel that is not connected to other pixels is an isolated point.

[0127] The first pixel value determination unit 133 compares pixel values ​​of a pixel region (neighborhood pixel group) of three rows and three columns including the pixel of interest and its eight neighboring pixels. The eight neighboring pixels are arranged to surround the pixel of interest. If the pixel value of the pixel of interest is the largest among the pixel values ​​of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the pixel of interest is a convex singular value. If the pixel value of the pixel of interest is the smallest among the pixel values ​​of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the pixel of interest is a concave singular value. If the pixel value of the pixel of interest is neither the largest nor the smallest among the pixel values ​​of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the pixel of interest is not singular. The first pixel state storage unit 134 stores information indicating these determination results in a data region corresponding to the pixel of interest.

[0128] In FIG. 13(b), the two-bit value written in the center of the circle corresponding to each pixel is convexity / concave data indicating whether the pixel value of the pixel (pixel of interest) is maximum or minimum in a pixel region of three rows and three columns including the pixel (pixel of interest) and its eight neighboring pixels. If the left bit of the convexity / concave data is 1, the pixel value of the pixel is maximum in the pixel region of three rows and three columns. If the right bit of the convexity / concave data is 1, the pixel value of the pixel is minimum in the pixel region of three rows and three columns. For example, for pixel P11, the convexity / concave data value is "01," indicating that the pixel value of pixel P11 is minimum (i.e., a concave singular value) in the pixel region of three rows and three columns. Also, for pixel P45, the convexity / concave data value is "10," indicating that the pixel value of pixel P45 is maximum (i.e., a convex singular value) in the pixel region of three rows and three columns.

[0129] The first connectivity determination unit 137 determines whether the pixel of interest is a convex or concave isolated point based on the 8-bit connected data and 2-bit concave / convex data (i.e., a total of 10 bits of pixel state data) stored in the first pixel state storage unit 134 (determination of the second condition). That is, in step S13 when the pixel value of the pixel of interest is singular, the first connectivity determination unit 137 determines whether the pixel of interest is an isolated point based on whether the number of connections calculated from the connected data is equal to or less than a threshold N (equal to or less than a first threshold). If it is determined that the pixel value of the pixel of interest is an isolated point (YES in step S13), the process proceeds to step S14. In this case, the pixel of interest is to be corrected. If it is determined that the pixel value of the pixel of interest is not an isolated point (NO in step S13), the process proceeds to step S15. In this case, the pixel of interest is not to be corrected.

[0130] For example, when pixel P45, whose concave / convex data value is "10" and which has a convex singular value, is referenced, all of the values ​​of the 8-bit concatenated data are 0. Therefore, the connectivity number of pixel P45 is 1, and pixel P45 is determined to be a convex isolated point in the processing of step S13. Therefore, pixel P45 is a correction target.

[0131] Furthermore, when pixel P15, whose concave / convex data value is "01" and which has a concave singular value, is referenced, the values ​​of all 8-bit concave data are 0. Therefore, the connectivity number of pixel P15 is 1, and pixel P15 is determined to be a concave isolated point in the processing of step S13. Therefore, pixel P15 is also a correction target.

[0132] Note that pixel P43 is an isolated point because all values ​​of the 8-bit concatenated data are 0, but it is not a singular value because the value of the unevenness data is "00." Since the pixel value is determined not to be singular in the processing of step S12 and the process proceeds to step S15, pixel P43 is not a target for correction.

[0133] 13(b), if all nine pixels in a pixel region of three rows and three columns that includes the pixel of interest and eight neighboring pixels have the same pixel value, the pixel value of the pixel of interest is both the maximum and minimum, and therefore the value of the unevenness data may be "11." In this case, the pixel values ​​of the pixel of interest and the eight neighboring pixels are the same, and therefore the connectivity of the pixel of interest is nine or more, so the pixel of interest is not an isolated point and is not subject to correction.

[0134] In step S14, the correction unit 136 corrects the pixel value of the pixel of interest by replacing the pixel value of the pixel of interest with the correction value generated by the correction data generation unit 132 and outputting the result. In step S15, the correction unit 136 does not correct the correction value of the pixel of interest. This process may involve outputting the pixel value held in the pixel data holding unit 131 as is.

[0135] The correction process in the correction data generation unit 132 is not particularly limited, but may, for example, refer to nine pixel values ​​in a pixel region of three rows and three columns including one pixel of interest and eight neighboring pixels, and use the median of these nine pixel values ​​as the correction value. Alternatively, instead of the median, the average or weighted average of the pixel region may be used as the correction value. Furthermore, the median, average, or weighted average may be calculated using pixel values ​​of the nine pixel values ​​of the pixel region excluding singular values.

[0136] The processes from step S12 to step S15 are performed for each of the plurality of pixels, and may be performed in parallel for each of the plurality of pixels, or may be performed sequentially for each of the plurality of pixels.

[0137] In addition, in the process of step S12, nine pixels in a pixel area of ​​three rows and three columns including the pixel of interest and its neighboring eight pixels are referenced, but the range of the pixel area to be referenced is not limited to this. For example, 25 pixels in a pixel area of ​​five rows and five columns including the pixel of interest and its neighboring 24 pixels may be referenced.

[0138] Furthermore, in determining whether a singular value is convex or concave in step S12, the criterion is whether the singular value is the maximum or minimum value in the pixel region, but the criterion for determining whether the singular value is convex or concave is not limited to this. In step S12, a singular value may be determined to be convex if the pixel value of the pixel of interest in the pixel region is ranked higher than the upper limit of a predetermined range (second range), or may be determined to be concave if the pixel value of the pixel of interest in the pixel region is ranked lower than the lower limit of the predetermined range. For example, if this predetermined range is from third to seventh, a singular value of the pixel of interest that is ranked first (maximum) or second in the pixel region is determined to be convex, and a singular value of the pixel of interest that is ranked ninth (minimum) or eighth in the pixel region is determined to be concave. If the upper limit of this predetermined range is set to second and the lower limit is set to the number of pixels in the pixel region minus one, the same processing is performed as when the criterion for determining whether the singular value is the maximum or minimum value in the pixel region. In other words, if the pixel value of the pixel of interest is ranked first (maximum) within the pixel region, it is determined to be convex, and if the pixel value of the pixel of interest is ranked last (minimum) within the pixel region, it is determined to be concave.

[0139] In the above example, for simplicity of explanation, the case where the threshold N for the number of connections in determining isolated points is 1 is described, i.e., pixels that are not connected to other pixels are considered isolated points; however, the value of threshold N may be 2 or greater. By setting the value of threshold N to 2 or greater, it is possible to make a determination that takes connected abnormal pixels into consideration. Below, the correction process of this embodiment will be more generalized and explained, taking into account the case where the value of threshold N is set to a value other than 1. In this case, pixels included in the first group of connected pixels, whose number of connections is N or less, are isolated points.

[0140] First, with reference to Figures 16(a) and 16(b), the isolated point determination process in step S13 of Figure 12 will be described. In this description, it is assumed that the value of the isolated point threshold N is 3.

[0141] FIG. 16(a) is a diagram schematically showing an example of pixel values ​​in the correction process according to this embodiment. FIG. 16(b) is a diagram schematically showing an example of pixel states in the correction process according to this embodiment. In FIGS. 16(a) and 16(b), the notation methods used in these diagrams are the same as those used in FIGS. 13(a) and 13(b). The pixel values, link data, and unevenness data shown in FIGS. 16(a) and 16(b) are also the same as those used in FIGS. 13(a) and 13(b).

[0142] 16(a) and 16(b), the hatched pixels P15, P43, and P45 are not connected to other pixels and have a connectivity count of 1, i.e., the connectivity count is equal to or less than the threshold value (N=3), so pixels P15, P43, and P45 are isolated points. This is similar to what is shown in FIGS. 13(a) and 13(b).

[0143] 16(b), the first connected pixel group G1 indicated by the dashed line includes pixels P42, P52, and P53. Referring to the connection data of pixels P42, P52, and P53, it is found that pixels P42, P52, and P53 are connected to one another. Therefore, the number of connections of pixels P42, P52, and P53 in first connected pixel group G1 is 3. In other words, because the number of connections is equal to or less than the threshold value (N=3), pixels P42, P52, and P53 are also isolated points. This number of connections can be determined by tracing the connection relationships by referring to the connection data of each pixel.

[0144] Next, the process of determining the correction target in steps S12 and S13 in FIG. 12 when the threshold N of the number of connections is 2 or more will be described.

[0145] The determination of whether a pixel of interest is a convex or concave isolated point is the same as when the threshold N is 1. However, when the connection count threshold N is 2 or greater, there may be multiple pixels in the first connected pixel group, and in such cases, it may not be determined that all pixels in the first connected pixel group are the same type of isolated point. For example, in the example of FIG. 16(b), of pixels P42, P52, and P53, the pixel value of pixel P52 is convex. However, the pixel values ​​of pixels P42 and P53 are not convex because pixel P52 is nearby.

[0146] In this way, when the number of connections is two or more, if the first connected pixel group determined to be an isolated point contains at least one pixel determined to be convex, all of the pixels in the first connected pixel group are considered to be convex isolated points. Also, if the first connected pixel group determined to be an isolated point contains at least one pixel determined to be concave, all of the pixels in the first connected pixel group are considered to be concave isolated points. That is, in either case, all pixels in the first connected pixel group are subject to correction (step S14 in FIG. 12). Also, if the first connected pixel group determined to be an isolated point does not contain either a pixel determined to be convex or a pixel determined to be concave, all of the pixels in the first connected pixel group are not subject to correction (step S15 in FIG. 12).

[0147] Note that there may be cases where the first connected pixel group determined to be an isolated point includes both pixels determined to be convex and pixels determined to be concave. It is desirable to define an exception process to handle such cases. A specific example of the exception process may be that all pixels in the first connected pixel group are convex isolated points, or that all pixels in the first connected pixel group are concave isolated points. Another specific example of the exception process may be that all pixels in the first connected pixel group are neither convex nor concave isolated points, and are therefore excluded from correction.

[0148] In the above example, it is assumed that the same correction process is performed on pixels in the first connected pixel group that are determined to be isolated points, but this is not limited to this. For example, it is also possible to perform correction on only pixels determined to be convex or concave among the first connected pixel group that are determined to be isolated points. The threshold N for the number of connections may be set to different values ​​for convex isolated points and concave isolated points. Also, the value of the threshold N for the number of connections may be changed for each imaging frame period of the photoelectric conversion device.

[0149] The data reference range in the process of determining the number of connections in step S13 will now be described in more detail. The first connection number determination unit 137 determines the number of connections by referring to a data area of ​​pixels in a range of (2N-1) × (2N-1) pixels centered on the pixel of interest, among the data stored in the first pixel state storage unit 134.

[0150] The reason for setting the reference range of the data area as described above will be explained. If it can be determined that the number of pixels reachable by tracing the connected pixels starting from the pixel of interest is N or less, including the pixel of interest, then the connectivity of the first connected pixel group including the pixel of interest can be said to be N or less. Also, if N pixels are connected in a straight line in the same direction from the pixel of interest and an adjacent pixel, then the connectivity can be said to be N+1 or more. Therefore, to determine whether the connectivity is (N+1) or more, it is sufficient to refer to the pixel states within a range of (N-1) pixels above, below, left, and right from the pixel of interest. In other words, it is possible to determine whether the connectivity is N or less by referring to the data area of ​​pixels within a (2N-1) x (2N-1) range centered on the pixel of interest.

[0151] As an example, a case will be described where the connectivity threshold N is 2. In this case, the first connectivity determination unit 137 determines the connectivity number by referring to a 3×3 pixel area centered on the pixel of interest, i.e., a data area including the pixel of interest and its surrounding eight pixels. If the pixel of interest and its surrounding pixels are not connected, the connectivity number of the first connected pixel group including the pixel of interest is 1. If the pixel of interest and two or more surrounding pixels are connected, the connectivity number of the first connected pixel group including the pixel of interest is greater than 2. If the pixel of interest is connected to one of its surrounding pixels and the connecting pixel is connected only to the pixel of interest, the connectivity number of the first connected pixel group including the pixel of interest is 2. If the pixel of interest is connected to one of its surrounding pixels and the connecting pixel is connected to pixels other than the pixel of interest, the connectivity number of the first connected pixel group including the pixel of interest is greater than 2. By calculating the connectivity number in this manner, it is possible to determine whether the connectivity number is 2 or less.

[0152] When the connectivity number is determined, the first connection process by first connection unit 135 and the determination of convex or concave singular values ​​by first pixel value determination unit 133 have been completed for at least the pixels corresponding to the above-mentioned data area. First connection unit 135 and first pixel value determination unit 133 refer to pixel values ​​of pixels in a 3 × 3 range centered on the pixel of interest. Therefore, the range of pixel values ​​related to the determination of the connectivity number by first connection number determination unit 137 is a (2N+1) × (2N+1) range.

[0153] In this embodiment, for simplicity of explanation, the pixel data storage unit 131 and the first pixel state storage unit 134 are assumed to store data for one frame. However, as described above, the first connection number determination unit 137 references a pixel data area in a (2N-1) × (2N-1) range centered on the pixel of interest, and therefore data from the (N-1)th row before the pixel of interest to the (N-1)th row after the pixel of interest is sufficient. However, since the processing by the first connection unit 135 compares data from the (N-1)th row after the pixel of interest with data from the Nth row after the pixel of interest, data from the Nth row after the pixel of interest is also required. Therefore, the first pixel state storage unit 134 can also be configured as a band memory having a storage capacity capable of storing data from the (N-1)th row before the pixel of interest to the Nth row after the pixel of interest.

[0154] Furthermore, first connecting unit 135, first pixel value determination unit 133, first connection number determination unit 137, and correction data generation unit 132 refer to a pixel data area of ​​a 3 × 3 range centered on the pixel of interest from pixel data storage unit 131. Here, the pixel of interest for first connecting unit 135 and first pixel value determination unit 133 is (N-1) rows behind the pixel of interest in the processing of correction data generation unit 132. Therefore, pixel data storage unit 131 can also be configured using a band memory having a storage capacity capable of storing data for approximately (N+1) rows.

[0155] Furthermore, an algorithm for determining the number of connections, such as Union-Find, may be used to determine the connection state in the first connection number determination unit 137. Even in this case, it is sufficient to refer to the data area of ​​pixels in the range of (2N-1) x (2N-1) centered on the pixel of interest in order to determine whether the number of connections is N or less, and therefore the band memory described above can be applied.

[0156] As described above, in the information processing device 1D based on pixel value concatenation, in the correction process for abnormal pixels, it is possible to detect and correct connected abnormal pixels that are equal to or smaller than the threshold value N. The value of N can be set appropriately depending on the occurrence status of abnormal pixels, etc.

[0157] Furthermore, in the information processing device 1D based on pixel value concatenation, position information of abnormal pixels acquired in advance and images captured under light-blocking conditions are not required. Therefore, abnormal pixels can be corrected more easily. Furthermore, since there is no need for memory capacity to hold this data, the memory capacity of the information processing device can be reduced.

[0158] [Eighth embodiment] The image processing device and image processing method in the above-described embodiments can be applied to various devices. Examples of such devices include digital still cameras, digital camcorders, camera heads, copiers, fax machines, mobile phones, vehicle-mounted cameras, observation satellites, and surveillance cameras. Fig. 17 shows a block diagram of a digital still camera as an example of such a device. Fig. 17 shows an example in which the image processing device and image processing method are applied to a digital still camera.

[0159] The device 70 shown in FIG. 17 includes a barrier 706, a lens 702, an aperture 704, and an image capture device 700 (an example of a photoelectric conversion device). The image capture device 700 includes a plurality of photoelectric conversion elements arranged in a plurality of rows and a plurality of columns, and the above-mentioned image processing device to which pixel values ​​based on signals output from the plurality of photoelectric conversion elements are input. The device 70 further includes a signal processing unit (processing device) 708, a timing generating unit 720, an overall control / calculation unit 718 (control device), a memory unit 710 (storage device), a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. At least one of the barrier 706, the lens 702, and the aperture 704 is an optical device corresponding to the device. The barrier 706 protects the lens 702, and the lens 702 forms an optical image of a subject on the image capture device 700. The aperture 704 adjusts the amount of light passing through the lens 702. The imaging device 700 converts an optical image formed by the lens 702 into image data (image signals). The signal processing unit 708 performs various corrections, data compression, etc. on the imaging data output from the imaging device 700. The timing generation unit 720 outputs various timing signals to the imaging device 700 and the signal processing unit 708. The overall control / calculation unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to or from the recording medium 714, which is a removable recording medium such as a semiconductor memory for recording or reading imaging data. The external I / F unit 712 is an interface for communicating with an external computer, etc. Timing signals, etc., may be input from outside the device. The device 70 may further include a display device (monitor, electronic viewfinder, etc.) for displaying information obtained by the photoelectric conversion device. The device includes at least a photoelectric conversion device. Furthermore, the equipment 70 includes at least one of an optical device, a control device, a processing device, a display device, a storage device, and a mechanical device that operates based on information obtained by the photoelectric conversion device. The mechanical device is a movable part (e.g., a robot arm) that operates in response to a signal from the photoelectric conversion device.

[0160] Each pixel may include a plurality of photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit). The signal processing unit 708 may be configured to process a pixel signal based on the charge generated in the first photoelectric conversion unit and a pixel signal based on the charge generated in the second photoelectric conversion unit, and acquire information about the distance from the image capturing device 700 to the subject.

[0161] [Ninth embodiment] 18(a) and 18(b) are block diagrams of devices related to an in-vehicle camera according to this embodiment. FIG. 18 illustrates an example in which the image processing device and image processing method according to the above-described embodiment are applied to a moving body such as a vehicle. The device 80 includes an image capturing device 800 (an example of a photoelectric conversion device) and a signal processing device (processing device) that processes signals from the image capturing device 800. The device 80 includes an image processing unit 801 that performs image processing on multiple pieces of image data acquired by the image capturing device 800, and a parallax calculation unit 802 that calculates parallax (phase difference between parallax images) from the multiple pieces of image data acquired by the device 80. The device 80 also includes a distance measurement unit 803 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of a collision based on the calculated distance. Here, the parallax calculation unit 802 and the distance measurement unit 803 are examples of distance information acquisition means that acquire distance information to an object. That is, the distance information includes information on the parallax, the amount of defocus, the distance to the object, etc. The collision determination unit 804 may determine the possibility of a collision using any of these distance information. The distance information acquisition means may be realized by dedicated hardware, a software module, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination of these.

[0162] The device 80 is connected to a vehicle information acquisition device 810 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The device 80 is also connected to a control ECU 820, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the determination result of the collision determination unit 804. The device 80 is also connected to an alarm device 830 that issues an alarm to the driver based on the determination result of the collision determination unit 804. For example, if the determination result of the collision determination unit 804 indicates a high collision possibility, the control ECU 820 performs vehicle control to avoid the collision and mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 830 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., or vibrating the seat belt or steering wheel. The device 80 functions as a control means for controlling the operation of controlling the vehicle as described above.

[0163] In this embodiment, the device 80 captures images of the surroundings of the vehicle, for example, the front or rear. Fig. 18(b) shows the device when capturing an image of the area in front of the vehicle (image capturing range 850). A vehicle information acquisition device 810, which serves as an image capturing control means, sends an instruction to the device 80 or the image capturing device 800 to perform an image capturing operation. This configuration can further improve the accuracy of distance measurement.

[0164] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from a lane, etc. Furthermore, the present invention is not limited to vehicles such as automobiles, but can be applied to moving objects (moving devices) such as ships, aircraft, artificial satellites, industrial robots, and consumer robots. In addition, the present invention can be applied to a wide range of devices that use object recognition or biometric recognition, such as intelligent transport systems (ITS) and surveillance systems, without being limited to moving objects.

[0165] [Modified embodiment] The present invention is not limited to the above-described embodiments and can be modified in various ways. For example, an example in which part of the configuration of one embodiment is added to another embodiment, or an example in which part of the configuration of another embodiment is replaced with another embodiment, is also an embodiment of the present invention.

[0166] For example, in the above description, when the pixel of interest Pa is an abnormal pixel, the correction unit 60 replaces the pixel value of the pixel of interest Pa included in the partial region R4 with the median value. However, the present invention is not limited to this. For example, the correction unit 60 may replace the pixel value of the pixel of interest Pa included in the partial region R4 with the average value of the pixel values ​​of the pixels included in the partial region R4. In this case, the average value may be the average value of the pixel values ​​of all pixels included in the partial region R4, or may be the average value of the pixel values ​​of only normal pixels (normal pixels) among the pixels included in the partial region R4.

[0167] The correction unit 60 may replace the pixel value of the target pixel Pa included in the partial region R4 with the maximum or minimum value of the pixel values ​​of the pixels included in the partial region R4. In this case, the ... l If the pixel value of the pixel of interest Pa is less than the upper threshold t u If it is equal to or greater than this, the pixel value of the pixel of interest Pa is replaced with the maximum value of the pixel values ​​of the pixels included in the partial region R4.

[0168] In addition, the pixel value of the target pixel Pa is lower than the lower threshold t l If the pixel value of the target pixel Pa is less than the upper threshold t u In this case, the maximum value determined from the pixel values ​​of only normal pixels may be used as the correction value. Since the correction value is calculated from only normal pixels, the influence of abnormal pixels on the calculation of the correction value can be reduced, and the correction value can be calculated with high accuracy.

[0169] Mean value m, standard deviation σ of Poisson distribution p , the standard deviation of the Gaussian distribution σ gWhen calculating, the calculation may be performed without including the pixel of interest Pa. This makes it possible to more accurately grasp the characteristics of the average value, Poisson distribution, and Gaussian distribution when the pixel of interest Pa is an abnormal pixel.

[0170] The disclosure of the above embodiment includes the following configurations and methods. (Configuration 1) an average value calculation unit that calculates an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest; a first standard deviation calculation unit that calculates a standard deviation of a Gaussian distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; a second standard deviation calculation unit that calculates a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of third pixels including the pixel of interest; a third standard deviation calculation unit that calculates a mixed standard deviation by mixing the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on the average value; a threshold calculation unit that calculates a threshold based on the average value and the mixed standard deviation; a correction unit that corrects the pixel of interest when the pixel value of the pixel of interest exceeds the threshold value. (Configuration 2) The image processing device according to configuration 1, wherein the third standard deviation calculation unit calculates weighting coefficients for the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on a ratio according to the average value. (Configuration 3) 3. The image processing device according to configuration 1 or 2, wherein the third standard deviation calculation unit increases the weighting coefficient of the standard deviation of the Gaussian distribution as the average value increases. (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the correction unit replaces the pixel value of the target pixel with a median value of pixel values ​​of a plurality of fourth pixels including the target pixel. (Configuration 5) 4. The image processing device according to any one of configurations 1 to 3, wherein the correction unit replaces the pixel value of the target pixel with an average value of pixel values ​​of a plurality of fourth pixels including the target pixel. (Configuration 6) 4. The image processing device according to any one of configurations 1 to 3, wherein the correction unit replaces the pixel value of the target pixel with the maximum or minimum value among pixel values ​​of a plurality of fourth pixels including the target pixel. (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, wherein the thresholds comprise an upper threshold calculated by adding the mixed standard deviation multiplied by a constant to the average value, and a lower threshold calculated by subtracting the mixed standard deviation multiplied by a constant from the average value. (Configuration 8) The image processing device according to Configuration 7, wherein the correction unit corrects the pixel value of the target pixel when the pixel value of the target pixel is less than the lower threshold or when the pixel value of the target pixel is equal to or greater than the upper threshold, and does not correct the target pixel when the pixel value of the target pixel is equal to or greater than the lower threshold and less than the upper threshold. (Configuration 9) 9. The image processing device according to configuration 7 or 8, wherein the correction unit replaces the pixel of interest with the minimum value among the pixel values ​​of a plurality of fourth pixels including the pixel of interest when the pixel value of the pixel of interest is less than the lower threshold, and replaces the pixel value of the pixel of interest with the maximum value among the pixel values ​​of the plurality of fourth pixels including the pixel of interest when the pixel value of the pixel of interest is equal to or greater than the upper threshold. (Configuration 10) 10. The image processing device according to any one of configurations 1 to 9, wherein correction processing is repeated multiple times for all pixels of an image including the first pixel, the second pixel, and the third pixel. (Configuration 11) an information processing device capable of correcting an image; an image processing device according to any one of configurations 1 to 10, capable of correcting the image corrected by the information processing device; The information processing device includes: a first connection unit that determines that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when a difference in pixel values ​​between the two adjacent pixels is within a first range; a pixel value determination unit that determines that the first condition is satisfied when a ranking of the pixel value of the pixel of interest in a neighborhood pixel group including the pixel of interest and neighborhood pixels arranged near the pixel of interest is higher than an upper limit of a second range or lower than a lower limit of the second range; a first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold; a correction unit that corrects the pixel value of the pixel of interest when at least the first condition and the second condition are satisfied, and does not correct the pixel value of the pixel of interest when the first condition is not satisfied or the second condition is not satisfied. 1. An image processing device comprising: (Configuration 12) an average value calculation unit that calculates an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest; a standard deviation calculation unit that calculates a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; a threshold calculation unit that calculates a threshold based on the average value and the standard deviation; a correction unit that corrects the pixel of interest when the pixel value of the pixel of interest exceeds the threshold value. (Configuration 13) a plurality of photoelectric conversion elements arranged in a plurality of rows and a plurality of columns; 13. The image processing device according to any one of configurations 1 to 12, to which pixel values ​​based on signals output from the plurality of photoelectric conversion elements are input; A photoelectric conversion device comprising: (Configuration 14) The photoelectric conversion device according to Configuration 13, an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device. (Configuration 15) 15. The device according to claim 14, wherein the processing device acquires distance information from the photoelectric conversion device to the subject. (Method 1) calculating an average value based on pixel values ​​of a plurality of first pixels including the pixel of interest; calculating a standard deviation of a Gaussian distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; calculating a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of third pixels including the pixel of interest; calculating a mixed standard deviation by mixing the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on the mean value; calculating a threshold based on the mean value and the blended standard deviation; and correcting the pixel of interest if the pixel value of the pixel of interest exceeds the threshold value. (Method 2) A program for causing a computer to execute the image processing method described in Method 1. (Method 3) calculating an average value based on pixel values ​​of a plurality of first pixels including the pixel of interest; calculating a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; calculating a threshold value based on the mean value and the standard deviation; and correcting the pixel of interest if the pixel value of the pixel of interest exceeds the threshold value. (Method 4) A program for causing a computer to execute the image processing method according to Method 3. [Explanation of symbols]

[0171] 1, 1A, 1B...Image processing device 30...Average value calculation section 40...Poisson standard deviation calculation section 50,50A...Threshold calculation section 60...Correction section 70...Gaussian standard deviation calculation section 90...Mixed standard deviation calculation section

Claims

1. an average value calculation unit that calculates an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest; a first standard deviation calculation unit that calculates a standard deviation of a Gaussian distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; a second standard deviation calculation unit that calculates a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of third pixels including the pixel of interest; a third standard deviation calculation unit that calculates a mixed standard deviation by mixing the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on the average value; a threshold calculation unit that calculates a threshold based on the average value and the mixed standard deviation; a correction unit that corrects the pixel of interest when the pixel value of the pixel of interest exceeds the threshold value.

2. 2 . The image processing device according to claim 1 , wherein the third standard deviation calculation unit calculates weighting coefficients for the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on a ratio according to the average value.

3. 3. The image processing device according to claim 2, wherein the third standard deviation calculation unit increases the weighting coefficient of the standard deviation of the Gaussian distribution as the average value increases.

4. The image processing device according to claim 1 , wherein the correction unit replaces the pixel value of the target pixel with a median value of pixel values ​​of a plurality of fourth pixels including the target pixel.

5. The image processing device according to claim 1 , wherein the correction unit replaces the pixel value of the target pixel with an average value of pixel values ​​of a plurality of fourth pixels including the target pixel.

6. The image processing device according to claim 1 , wherein the correction unit replaces the pixel value of the target pixel with a maximum or minimum value among pixel values ​​of a plurality of fourth pixels including the target pixel.

7. 2. The image processing device according to claim 1, wherein the thresholds comprise an upper threshold calculated by adding the mixed standard deviation multiplied by a constant to the average value, and a lower threshold calculated by subtracting the mixed standard deviation multiplied by a constant from the average value.

8. The image processing device described in claim 7, characterized in that the correction unit corrects the pixel value of the target pixel when the pixel value of the target pixel is less than the lower threshold or when the pixel value of the target pixel is greater than or equal to the upper threshold, and does not correct the target pixel when the pixel value of the target pixel is greater than or equal to the lower threshold and less than the upper threshold.

9. 9. The image processing device according to claim 8, wherein the correction unit replaces the pixel of interest with the minimum value among the pixel values ​​of a plurality of fourth pixels including the pixel of interest when the pixel value of the pixel of interest is less than the lower threshold, and replaces the pixel value of the pixel of interest with the maximum value among the pixel values ​​of a plurality of fourth pixels including the pixel of interest when the pixel value of the pixel of interest is equal to or greater than the upper threshold.

10. The image processing device according to claim 1 , wherein the correction process is repeated a plurality of times for all pixels of the image including the first pixel, the second pixel, and the third pixel.

11. an information processing device capable of correcting an image; and the image processing device according to claim 1 , capable of correcting the image corrected by the information processing device, The information processing device includes: a first connection unit that determines that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when a difference between the pixel values ​​of the two adjacent pixels is within a first range; a pixel value determination unit that determines that a first condition is satisfied when a ranking of the pixel value of the pixel of interest in a neighborhood pixel group including the pixel of interest and neighborhood pixels arranged near the pixel of interest is higher than an upper limit of a second range or lower than a lower limit of the second range; a first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold; a correction unit that corrects the pixel value of the pixel of interest when at least the first condition and the second condition are satisfied, and does not correct the pixel value of the pixel of interest when the first condition is not satisfied or the second condition is not satisfied.

1. An image processing device comprising:

12. an average value calculation unit that calculates an average value based on pixel values ​​of a plurality of first pixels including a pixel of interest; a standard deviation calculation unit that calculates a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; a threshold calculation unit that calculates a threshold based on the average value and the standard deviation; a correction unit that corrects the pixel of interest when the pixel value of the pixel of interest exceeds the threshold value.

13. a plurality of photoelectric conversion elements arranged in a plurality of rows and a plurality of columns; an image processing device according to claim 1 , wherein pixel values ​​based on signals output from the plurality of photoelectric conversion elements are input; A photoelectric conversion device comprising:

14. The photoelectric conversion device according to claim 13; an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device.

15. The device according to claim 14 , wherein the processing device acquires distance information from the photoelectric conversion device to the subject.

16. calculating an average value based on pixel values ​​of a plurality of first pixels including the pixel of interest; calculating a standard deviation of a Gaussian distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; calculating a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of third pixels including the pixel of interest; calculating a mixed standard deviation by mixing the standard deviation of the Gaussian distribution and the standard deviation of the Poisson distribution based on the mean value; calculating a threshold based on the mean value and the blended standard deviation; and correcting the pixel of interest if the pixel value of the pixel of interest exceeds the threshold value.

17. A program for causing a computer to execute the image processing method according to claim 16.

18. calculating an average value based on pixel values ​​of a plurality of first pixels including the pixel of interest; calculating a standard deviation of a Poisson distribution based on pixel values ​​of a plurality of second pixels including the pixel of interest; calculating a threshold value based on the mean value and the standard deviation; and correcting the pixel of interest if the pixel value of the pixel of interest exceeds the threshold value.

19. A program for causing a computer to execute the image processing method according to claim 18.

Images