Image processing device, image processing method, and program

The image processing device addresses image quality differences by increasing the noise of interpolated pixels to match non-defective pixels' variance, reducing blurring and artifacts through targeted noise adjustment.

JP2024161707AInactive Publication Date: 2024-11-20CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023076639
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-20
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Image quality differences occur between regions of an image due to noise reduction processing applied after interpolation on defective pixels, leading to issues like blurring and artifacts.

Method used

An image processing device that applies interpolation processing to defective pixels using a different pixel as a reference and then increases the noise of the interpolated pixel to match the noise variance of non-defective pixels, using methods such as adding random numbers from a predetermined probability distribution.

Benefits of technology

Reduces image quality differences by aligning the noise variance of interpolated pixels with non-defective pixels, minimizing blurring and artifacts, even when subsequent noise reduction or compression encoding is applied.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161707000001_ABST
    Figure 2024161707000001_ABST
Patent Text Reader

Abstract

To provide an image processing device, an image processing method, and a program that are capable of reducing the occurrence of image quality differences.SOLUTION: An image processing method applies an interpolation process to a defective pixel of an acquired image by using a pixel other than the defective pixel as a reference pixel, and increases noise in the interpolated pixel to which the interpolation process has been applied.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for processing images obtained by imaging. [Background technology]

[0002] Imaging elements such as CCD and CMOS may contain defective pixels caused by damage, dust, etc. If left as is, defective pixels will be observed as so-called white or black defects, so correction processing is performed to prevent this. Generally, the correction processing uses a method based on an interpolation process in which the pixel value of the defective pixel is replaced with a pixel value obtained by taking a weighted average of other pixel values ​​(see Patent Document 1). Other image processing techniques include techniques that increase noise to achieve a certain effect, and Patent Document 2 discloses a technique that adds noise to an image in order to suppress tone gaps that tend to occur in gradation areas of the image. The technique disclosed in Patent Document 2 aims to suppress degradation of image quality such as tone gaps. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-177792 A [Patent Document 2] JP 2010-206703 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when noise reduction processing or the like is further applied to an image after the defective pixels have been subjected to the interpolation processing, a difference in image quality may occur between the pixels after the defective pixels have been interpolated and the area consisting of other pixels (normal pixels that were not defective to begin with).

[0005] Therefore, an object of the present invention is to make it possible to reduce the occurrence of differences in image quality. [Means for solving the problem]

[0006] The image processing device of the present invention is characterized by having an interpolation means for applying an interpolation process to a defective pixel of an image by using a pixel other than the defective pixel as a reference pixel, and a noise increasing means for increasing noise in the interpolated pixel to which the interpolation process has been applied. Effect of the Invention

[0007] According to the present invention, it is possible to reduce the occurrence of image quality differences. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 2] 1A and 1B are diagrams used to explain defective pixels, reference pixels, and interpolation processing; [Diagram 3] 5 is a flowchart showing a flow of image processing according to the first embodiment. [Figure 4] FIG. 11 is a diagram illustrating an example of a functional configuration of an image processing device according to a second embodiment. [Diagram 5] 10 is a flowchart showing a flow of image processing according to a second embodiment. [Figure 6] FIG. 13 is a diagram illustrating an example of a functional configuration of an image processing device according to a third embodiment. [Figure 7] 13 is a flowchart showing the flow of image processing according to the third embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the embodiment may be configured by appropriately combining parts of each embodiment described below. In the following embodiments, the same configurations are described with the same reference symbols.

[0010] <First embodiment> 1 is a diagram showing an example of the functional configuration of an image processing device 1 according to the first embodiment. The image processing device 1 of this embodiment applies an interpolation process to a defective pixel, and then performs image processing such as increasing noise in the pixel after the interpolation process (hereinafter, referred to as an interpolated pixel). Before explaining the configuration and processing of the image processing device 1 of this embodiment, we will now explain the image quality difference that may occur when, for example, noise reduction processing is applied to an image after interpolation processing has been performed on defective pixels.

[0011] When an image sensor has a defective pixel due to damage or dust, as described above, an interpolation process is often performed in which the pixel value of the defective pixel is replaced with a pixel value obtained by the weighted average of other pixel values ​​(hereinafter referred to as reference pixels). However, when an interpolation process using a weighted average is performed, due to the nature of the weighted average, the noise variance of the interpolated pixel becomes smaller than the noise variance of other pixels (reference pixels) used for the weighted average. In general, the reference pixel is selected from a continuous area near the defective pixel where the pixel value changes as little as possible, so that the noise variance of the defective pixel when it is assumed that the pixel defect does not occur can be approximated by the noise variance of the reference pixel. Therefore, it can be said that the noise variance of the interpolated pixel becomes smaller than the noise variance when it is assumed that the pixel defect does not occur. In other words, in the image after the interpolation process, pixels with smaller noise variance than the original one are scattered.

[0012] For example, when a noise reduction process is performed on an image after an interpolation process, the noise reduction process may be performed on the interpolated pixels assuming a noise variance that is greater than the actual one. In other words, in this case, excessive noise reduction is applied to the interpolated pixels, which may cause image quality degradation such as blurring. In addition to the noise reduction process, when a lossy compression encoding process is performed on the interpolated image, the noise of the interpolated pixels may be locally reduced, which may cause image quality degradation such as artifacts after the compression encoding process.

[0013] Depending on the type of imaging element, each pixel value of the captured image may contain noise due to statistical fluctuations in the number of photons, heat, etc., in addition to the signal value derived from the brightness of the subject. In particular, when photographing at night or in a dark place, a high gain is set to make the brightness of the subject observable, but in this case, the noise increases along with the signal value. In this case, the noise can be reduced by applying noise reduction processing, but as described above, applying noise reduction processing may cause image quality degradation.

[0014] Hereinafter, the weighted average interpolation process and the principle by which the noise variance of the pixel values ​​of the interpolated pixels becomes smaller than the noise variance of the pixel values ​​of the reference pixels will be described in detail with reference to Fig. 2. After that, the image processing according to this embodiment for compensating for the noise variance reduced by the interpolation process will be described.

[0015] In FIG. 2, each square shown in a grid pattern in the figure represents a pixel, with each black square representing a defective pixel 200 and each white square represented by a solid line representing a reference pixel. In the example of FIG. 2, a case will be described assuming that an interpolation process is applied to a defective pixel 200x. That is, for the defective pixel 200x, a pixel other than the defective pixel 200x is used as a reference pixel 201 to perform an interpolation process on the defective pixel 200x, thereby calculating a pixel value x of an interpolated pixel. However, to simplify the explanation, it is approximated that the noise variance of the reference pixels 201 is constant. Here, the number of reference pixels 201 is set to M, and each pixel value of the reference pixels 201 is set to vm Let us express it as:

[0016] The pixel value of the captured image is a combination of the signal value (true value) and noise, so the pixel value v m is expressed by the following formula (1). m is the signal value, n m represents noise.

[0017] v m =s m +n m Formula (1)

[0018] Pixel value v m The weight coefficient of the weighted average corresponding to a m Then, the pixel value x of the interpolated pixel is expressed by the following equation (2). Here, the weighting coefficients a1, a2, ..., a M are all positive constants less than or equal to 1, and the sum of the M numbers is 1.

[0019]

number

[0020] In addition, the variance of the pixel value x of the interpolated pixel is σ 2 Let the noise variance of the reference pixel 201 be σ 2 ref Then, the formula for the sum of variances gives us the following equation (3).

number

[0021] Here, the variance σ of the pixel value x of the interpolated pixel is calculated by the following formula (4): 2 is the noise variance σ of the reference pixel 201. 2 ref It can be seen that the noise is reduced by the interpolation process based on the weighted average, i.e., M is smaller than the noise. However, the necessary and sufficient condition for the equality in equation (4) to hold is M=1, and the equality does not hold when M≧2.

[0022]

number

[0023] As described above, the noise variance in the defective pixel on the assumption that no pixel defect occurred can be approximated by the noise variance of the reference pixel 201. Therefore, it can be said that the noise variance of the interpolated pixel after the interpolation process is performed on the defective pixel 200x is smaller than the variance on the assumption that no pixel defect occurred.

[0024] Therefore, in the image processing device of this embodiment, in order to reduce the difference between the actual noise variance of the interpolated pixel and the noise variance assuming that the pixel defect did not occur, the following noise variance compensation process is performed. As described above, since the noise variance of the interpolated pixel is reduced due to the effect of the interpolation process, the image processing device of this embodiment compensates for the difference in noise variance by increasing the noise of the interpolated pixel. In particular, in this embodiment, the noise is increased by using the noise that would be obtained if the pixel defect had not occurred as a target noise.

[0025] In this embodiment, the method of determining the variance of the target noise is based on the pixel values ​​of the interpolated pixels and a mathematical model that represents the relationship between the pixel values ​​and the noise variance. Here, the mathematical model can be obtained, for example, by acquiring images with uniform brightness at multiple levels in advance, calculating the average and variance of the pixel values ​​of each image, and expressing the relationship as a broken line. As described above, since the target noise can be approximated by the noise variance of the reference pixels, a method of calculating the variance of the reference pixels and using the obtained value as the variance of the target noise may also be used.

[0026] In this embodiment, a specific method for increasing noise is, for example, a method of adding random numbers. For example, if a random number that is generated from a predetermined probability distribution and is uncorrelated with the noise contained in the image is added, the variance of the pixel value after the random number is added will be equal to the sum of the variance of the pixel value before the random number is added and the variance of the predetermined probability distribution. Therefore, in this embodiment, a probability distribution that includes a variance as a distribution parameter, either positive or negative, is used as the predetermined probability distribution, and the variance of the probability distribution is made to approximately match the difference between the variance of the target noise and the noise variance of the interpolated pixel. This makes it possible to make the noise variance of the interpolated pixel by adding the random number approximately match the variance of the target noise.

[0027] As described above, the noise variance of the interpolated pixel can be calculated by equation (3). However, the noise variance of the reference pixel is required for the calculation by equation (3). In this embodiment, the noise variance of the reference pixel is calculated by a method of estimation based on the pixel value of the reference pixel and the above-mentioned mathematical model. Note that, since the noise generated in the imaging system can be approximated by a Gaussian distribution, and the Gaussian distribution has reproducibility, it is preferable to use the Gaussian distribution as the probability distribution.

[0028] Hereinafter, a description will be given of an example of the functional configuration of the image processing device according to the first embodiment shown in Fig. 1. Here, an example will be described in which an image is acquired from an external imaging device (not shown), and the image is subjected to image processing by the image processing device 1 of this embodiment, and the image is then output to an external display device (not shown).

[0029] In the image processing device 1 according to this embodiment, the image acquisition unit 11 is a communication interface that acquires an image from an external imaging device. For example, in the imaging device, a white balance adjustment process and a debayer process are performed on an image acquired by an imaging element, and the image acquisition unit 11 acquires the image after these processes.

[0030] The defect information acquisition unit 12 is a communication interface that acquires defective pixel information from an external imaging device. For example, the imaging device holds defective pixel information indicating the position of a defective pixel on an imaging element, the type of defect, etc., and the defect information acquisition unit 12 acquires the defective pixel information from the imaging device. Note that instead of the defect information acquisition unit 12 acquiring the defective pixel information, the image processing device 1 may include a defective pixel detection unit, and the defective pixel detection unit may perform processing to detect defective pixels from the image acquired by the image acquisition unit 11.

[0031] The interpolation unit 13 performs an interpolation process on the image acquired by the image acquisition unit 11 based on the defective pixel information acquired by the defect information acquisition unit 12. As described above, the interpolation process is a process in which the pixel value of the defective pixel is replaced with a pixel value obtained by the weighted average of other pixel values ​​(pixel values ​​of reference pixels). As a specific method for determining the weight coefficient of the weighted average, the method described in the above-mentioned Patent Document 1 can be used, for example.

[0032] The noise increase unit 14 executes noise increase processing to increase the noise of the pixel value of the interpolated pixel in the image after the interpolation processing is applied by the interpolation unit 13. The noise increase unit 14 determines a target noise based on the pixel value of the interpolated pixel and a mathematical model that represents the relationship between the pixel value and the noise variance. Then, the noise increase unit 14 performs noise increase processing on the interpolated pixel based on the target noise, for example, as described above, using a method of adding random numbers generated from a predetermined probability distribution.

[0033] In this embodiment, an example is given in which the noise increasing unit 14 includes a function for determining the target noise, but a noise determining unit for determining the target noise may be provided separately. In addition, in this embodiment, an example is given in which a random number is generated from a predetermined probability distribution, but the random number may be obtained by other methods. For example, the random number may be obtained by selecting from one or more pseudo-random numbers stored in the ROM 104.

[0034] The processing unit 15 executes various predetermined processes such as noise reduction, compression encoding, gamma correction, super-resolution, and sharpening on the image after the noise increase process by the noise increase unit . The image output unit 16 is a communication interface that outputs the image after various predetermined processes by the processing unit 15 to an external display device (not shown) or the like.

[0035] 1 is merely an example and does not limit the functional configuration of the image processing device 1 according to this embodiment. For example, the image processing device 1 may be an image processing device that also serves as an imaging device, including an image acquisition unit 11, a defect information acquisition unit 12, an interpolation unit 13, a noise increase unit 14, a processing unit 15, an image output unit 16, and an imaging unit including an optical system and an imaging element.

[0036] 3 is a flowchart showing the flow of processing in the image processing device 1 according to this embodiment. In the following flowcharts, the reference character S represents a processing step (process).

[0037] First, in the process of S1, the image acquisition unit 11 of the image processing device 1 acquires an image from an external imaging device. Next, in the process of S2, the defect information acquisition unit 12 acquires defective pixel information from an external imaging device. Next, in the image processing device 1, the loop process of S3 and S4 is performed on the image acquired by the image acquisition unit 11 for each position of a defective pixel included in the defective pixel information acquired by the defect information acquisition unit 12.

[0038] In S3, the interpolation unit 13 applies an interpolation process to the defective pixel by using a pixel other than the defective pixel as a reference pixel. In S4, the noise increasing section 14 increases noise in the pixel values ​​of the interpolated pixels after the interpolation process by the interpolation section 13. Then, when the loop processing of S3 and S4 for each position of the defective pixel is completed, the process of the image processing device 1 proceeds to S5.

[0039] When proceeding to S5, the processing unit 15 performs various predetermined processes such as noise reduction, compression encoding, gamma correction, super-resolution, and sharpening on the image after the loop processing of S3 and S4 is completed, that is, the image after the noise increase processing is performed on each interpolated pixel. Thereafter, in the process of S6, the image output unit 16 outputs the image after the processing by the processing unit 15 has been applied to an external display device (not shown) or the like.

[0040] As described above, the image processing device 1 of the first embodiment performs a process to increase the noise of the interpolated pixel after applying an interpolation process to a defective pixel. That is, according to this embodiment, it is possible to reduce the difference between the actual noise variance of the interpolated pixel and the noise variance in the case where it is assumed that the pixel defect does not occur. Therefore, even if, for example, a noise reduction process is subsequently performed in the processing unit 15, image quality degradation caused by the difference between the noise variance assumed in the noise reduction process and the actual noise variance is reduced. Even if a lossy compression encoding process is performed in the processing unit 15, the occurrence of image quality degradation such as denaturation of spatial frequency characteristics and artifacts can be reduced. Furthermore, according to this embodiment, the occurrence of image quality degradation that may occur due to application of a noise reduction process that reduces noise caused by statistical fluctuations in the number of photons, heat, etc. can also be reduced.

[0041] <Second embodiment> In the above-described first embodiment, an example has been given in which noise on the assumption that no pixel defect occurs is used as the target noise. In contrast to this, in the following second embodiment, a configuration example will be described in which the target noise is determined using an image after noise reduction processing is performed on an image including interpolated pixels. Fig. 4 is a diagram showing an example of the functional configuration of an image processing device 41 according to the second embodiment. In the configuration of Fig. 4, explanations of functional units that are generally similar to those in the configuration of Fig. 1 of the first embodiment described above will be omitted as appropriate. The image processing device 41 according to the second embodiment includes an interpolated pixel noise reduction unit 17 in addition to the same configuration as that of Fig. 1 described above.

[0042] The interpolated pixel noise reduction unit 17 performs noise reduction processing on the interpolated pixels in the image after the interpolation processing is applied by the interpolation unit 13. The target of the noise reduction processing here is not limited to only the interpolated pixels, and may be, for example, a local region including the interpolated pixels, or the entire image.

[0043] In the second embodiment, the noise increase unit 14 estimates the variance of the target noise based on the pixel value of the interpolated pixel after the noise reduction process is performed by the interpolated pixel noise reduction unit 17 and the above-mentioned mathematical model. Then, based on the estimated variance of the target noise, the noise increase unit 14 performs noise increase processing such that the noise variance of the pixel value of the interpolated pixel after the interpolation process by the interpolation unit 13 becomes larger. Note that, although the second embodiment also shows an example in which the noise increase unit 14 includes a function of determining the target noise, the target noise may be determined by a noise determination unit provided as a functional unit separate from the noise increase unit 14.

[0044] Fig. 5 is a flowchart showing the flow of processing in an image processing device 41 according to the second embodiment. In the flowchart of Fig. 5, explanations of processing steps that are generally similar to those in the flowchart shown in Fig. 3 of the first embodiment will be omitted as appropriate. In the case of the flowchart according to the second embodiment, the loop processing is such that processing S7 is added between S3 and S4.

[0045] In the second embodiment, as the processing of S7, the interpolated pixel noise reduction unit 17 performs noise reduction processing on the image after the interpolation processing by the interpolation unit 13 in S3, as described above, targeting the interpolated pixels, or a local area including the interpolated pixels, or the entire image.

[0046] Next, in S4, the noise increase unit 14 estimates the variance of the target noise based on the pixel value of the interpolated pixel after the noise reduction process by the interpolated pixel noise reduction unit 17 and the mathematical model. Then, based on the estimated variance of the target noise, the noise increase unit 14 makes the noise variance of the pixel value of the interpolated pixel after the interpolation process by the interpolation unit 13 larger. That is, in the case of the second embodiment, in S4, the noise increase unit 14 performs noise increase processing of the interpolated pixel based on the variance of the target noise estimated using the pixel value of the interpolated pixel after the noise reduction process and the mathematical model. Note that the processing from S5 onwards after the end of this hoop processing is the same as described above.

[0047] According to the second embodiment, by using the pixel values ​​of the interpolated pixels after the noise reduction process is performed, the estimation accuracy of the noise variance of the interpolated pixels based on the mathematical model is improved, thereby realizing a more suitable noise increase.

[0048] In the second embodiment, the interpolated pixel noise reduction unit 17 performs noise reduction processing on the image after the interpolation processing by the interpolation unit 13, but the order of the interpolation processing and the noise reduction processing does not matter. That is, for example, the noise reduction processing may be performed on the image acquired by the image acquisition unit 11, and then the interpolation processing may be performed.

[0049] <Third embodiment> Next, as a third embodiment, similar to the first embodiment described above, the increase in noise for interpolated pixels is achieved by adding a random number, but when calculating the random number, an image in which noise reduction processing has been performed on reference pixels is used. Fig. 6 is a diagram showing an example of the functional configuration of an image processing device 61 according to the third embodiment. Note that in the configuration of Fig. 6, explanations of functional units that are generally similar to those in the configuration of Fig. 1 of the first embodiment described above will be omitted as appropriate. The image processing device 61 according to the third embodiment has a reference pixel noise reduction unit 18 in addition to the same configuration as that of Fig. 1 described above, and has a random number addition unit 19 instead of the noise increase unit 14 of Fig. 1.

[0050] The reference pixel noise reduction unit 18 performs noise reduction processing on the reference pixels in the image after the interpolation processing has been applied by the interpolation unit 13. The target of the noise reduction processing here is not limited to only the reference pixels, and may be, for example, a local region including the reference pixels, or the entire image.

[0051] The random number addition unit 19 increases noise in the interpolated image by executing a process of adding random numbers to the interpolated pixels described above in the image after the interpolation process by the interpolation unit 13. That is, in the case of the third embodiment, the random number addition unit 19 increases noise in the same manner as the noise increase unit in the above-mentioned embodiment by adding random numbers to the interpolated pixels. Also, in the case of the third embodiment, the random number addition unit 19 determines the target noise using the pixel value after the noise reduction process is executed on the reference pixel and a mathematical model. Note that, even in the third embodiment, the target noise may be determined by a noise determination unit provided as a functional unit separate from the random number addition unit 19.

[0052] In the third embodiment, the random number adder 19 generates random numbers from a predetermined probability distribution in the same manner as described in the first embodiment, and the variance of the probability distribution is the difference between the variance of the target noise and the noise variance of the interpolated pixel as described above. The noise variance of the interpolated pixel can be calculated based on the above-mentioned formula (3), but the noise variance of the reference pixel is required for this calculation. The random number adder 19 estimates the noise variance of the reference pixel using the pixel value of the reference pixel after the noise reduction process by the reference pixel noise reduction unit 18 and a mathematical model.

[0053] Fig. 7 is a flowchart showing the flow of processing in an image processing device 61 according to the third embodiment. In the flowchart of Fig. 7, explanations of processing steps that are generally similar to those in the flowchart of Fig. 3 will be omitted as appropriate. In the case of the flowchart of the third embodiment, after processing S3, processing of S8 and S9 is performed, and these steps from S3 to S9 form a loop process.

[0054] In the third embodiment, as the processing of S8, the reference pixel noise reduction unit 18 performs noise reduction processing on the image after the interpolation processing by the interpolation unit 13 in S3, targeting the reference pixels, or a local area including those reference pixels, or the entire image. Next, in the process of S9, the random number addition unit 19 executes random number addition processing on the interpolated pixels in the image after the interpolation processing by the interpolation unit 13. In the image processing device 61 of the third embodiment, when the loop process from S3 to S9 for each defective pixel position is completed, the process proceeds to S5. The process from S5 onwards is the same as described above.

[0055] According to the third embodiment, by using the pixel values ​​of the reference pixels after the noise reduction process, the estimation accuracy of the noise variance of the reference pixels using the above-mentioned mathematical model is improved, so that the variance of the probability distribution can be determined with higher accuracy. As a result, it is possible to realize a more suitable random number addition process (noise increase process).

[0056] <Hardware configuration> FIG. 8 is a diagram showing an example of a hardware configuration that can be applied to any of the image processing devices according to the first to third embodiments described above. The image processing device of each embodiment includes a CPU (Central Processing Unit) 101, a communication I / F unit 102, a RAM (Random Access Memory) 103, and a ROM (Read Only Memory) 104. The CPU 101, the communication I / F unit 102, the RAM 103, and the ROM 104 are connected to each other via a data bus 105 so as to be able to transmit and receive information to and from each other.

[0057] The CPU 101 performs various controls by reading out a control program stored in the ROM 104, loading the control program into the RAM 103, and executing the control program. The RAM 103 is used as a work area and a temporary storage area for loading the program executed by the CPU 101.

[0058] Moreover, the CPU 101 reads out an image processing program according to any one of the first to third embodiments stored in the ROM 104, expands it in the RAM 103, and executes the image processing program to perform processing according to the functional configuration of each of the above-mentioned embodiments. That is, the image processing program according to this embodiment includes an interpolation processing program, and the above-mentioned interpolation processing in the interpolation unit 13 can be realized by the CPU 101 executing the interpolation processing program stored in the ROM 104. The image processing program also includes a noise increase processing program, and the above-mentioned noise increase processing in the noise increase unit 14 can be realized by the CPU 101 executing the noise increase processing program stored in the ROM 104. The image processing program also includes various processing programs, and the above-mentioned various processing in the processing unit 15 can be realized by the CPU 101 executing the various processing programs stored in the ROM 104. The image processing program also includes an interpolated pixel noise reduction processing program, and the above-mentioned interpolated pixel noise reduction processing in the interpolated pixel noise reduction unit 17 can be realized by the CPU 101 executing the interpolated pixel noise reduction processing program stored in the ROM 104. The image processing program also includes a reference pixel noise reduction processing program, and the noise reduction processing in the reference pixel noise reduction unit 18 described above can be realized by the CPU 101 executing the reference pixel noise reduction processing program stored in the ROM 104. The image processing program also includes a random number addition processing program, and the random number addition processing in the random number addition unit 19 described above can be realized by the CPU 101 executing the random number addition processing program stored in the ROM 104.

[0059] The communication I / F unit 102 is a communication I / F for connecting the image processing device of each of the above-mentioned embodiments to an external imaging device, display device, etc., and transmits and receives various information to and from other devices via the communication I / F. Image data and the like received from an external device via the communication I / F unit 102 is transmitted and received to the CPU 101, RAM 103, and ROM 104 via a data bus 105. The image acquisition unit 11, defect information acquisition unit 12, and image output unit 16 in the above-mentioned embodiments can be realized by the communication I / F unit 102.

[0060] In the first to third embodiments described above, an interpolation process is applied to a defective pixel using a pixel other than the defective pixel as a reference pixel, and the noise of the interpolated pixel after the interpolation process is applied is increased. Here, as a technique for obtaining a predetermined effect by increasing noise, there is a technique for adding noise to an image to suppress tone steps that tend to occur in a gradation area of ​​an image, as described in the above-mentioned Patent Document 2. However, the technique described in Patent Document 2 is intended to suppress a predetermined degradation of image quality, which is tone steps. In contrast, the image processing of this embodiment compensates for the noise that has been reduced by the interpolation process, and the image processing of this embodiment makes it possible to optimize the noise itself. Thus, the technique described in Patent Document 2 and the image processing of this embodiment are different not only in configuration but also in purpose and effect.

[0061] The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-mentioned embodiments are merely examples of concrete implementations of the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0062] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) an interpolation means for applying an interpolation process to a defective pixel of an image by using a pixel other than the defective pixel as a reference pixel; a noise increasing means for increasing noise of the interpolated pixels to which the interpolation process has been applied; 13. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein at least one of noise reduction processing and compression encoding processing is applied to an image including interpolated pixels in which the noise has been increased by the noise increasing means. (Configuration 3) 3. The image processing device according to claim 1, further comprising: a noise determination means for determining a target noise when increasing the noise of the interpolated pixel, based on the pixel value of the interpolated pixel and a mathematical model representing a relationship between the pixel value and a noise variance. (Configuration 4) a first noise reduction means for applying a first noise reduction process to the interpolated pixels; The image processing device according to configuration 3, wherein the noise determination means determines the target noise by using a pixel value obtained after the first noise reduction process is applied to the interpolated pixel and the mathematical model. (Configuration 5) 5. The image processing device according to configuration 4, wherein the first noise reduction means applies the first noise reduction process to a local region including the interpolated pixels in the image, or to the image. (Configuration 6) a second noise reduction unit that applies a second noise reduction process to the reference pixels; The image processing device according to configuration 3, wherein the noise determination means determines the target noise by using a pixel value of the interpolated pixel, a pixel value of the reference pixel after the second noise reduction process is applied, and the mathematical model. (Configuration 7) 7. The image processing device according to configuration 6, wherein the second noise reduction means applies the second noise reduction process to a local region including the reference pixel in the image, or to the image. (Configuration 8) 8. The image processing device according to any one of configurations 1 to 7, wherein the noise increasing means increases the noise by adding a random number to the interpolated pixels. (Configuration 9) 9. The image processing device according to claim 8, wherein the noise increasing means adds the random number selected from one or more pseudo-random numbers. (Configuration 10) 9. The image processing device according to configuration 8, wherein the noise increasing means generates the random numbers based on a probability distribution that explicitly or implicitly includes a variance as a distribution parameter. (Configuration 11) 11. The image processing device according to claim 10, wherein the probability distribution is a Gaussian distribution. (Configuration 12) 11. The image processing device according to claim 10, wherein the variance in the distribution parameter is approximately equal to the difference between the variance of a target noise and the variance of noise of the interpolated pixel obtained by adding the random number. (Configuration 13) the interpolation process is a process of setting a pixel value obtained by a weighted average of pixel values ​​of the reference pixels as a pixel value of the interpolated pixel, 13. The image processing device according to claim 12, wherein the noise increasing means obtains the noise variance of the interpolated pixel by using a weighting coefficient of the weighted average and the noise variance of the reference pixel. (Configuration 14) 14. The image processing device according to configuration 13, wherein the noise increasing means determines the noise variance of the reference pixels using a mathematical model that represents the relationship between pixel values ​​and noise variance. (Configuration 15) 15. The image processing device according to claim 14, wherein the noise increasing means calculates a variance of noise of the reference pixels by using pixel values ​​after noise reduction processing is performed on the reference pixels and the mathematical model. (Configuration 16) 16. The image processing device according to any one of configurations 3 to 7, 14, and 15, wherein the mathematical model is a model that represents a relationship between an average and a variance of pixel values ​​of a plurality of images having uniform brightness levels. (Method 1) an interpolation step of applying an interpolation process to a defective pixel of an image by using a pixel other than the defective pixel as a reference pixel; a noise increasing step of increasing noise of the interpolated pixels to which the interpolation process has been applied; 13. An image processing method comprising: (Program 1) A program for causing a computer to function as the image processing device according to any one of configurations 1 to 16. [Explanation of symbols]

[0063] 1, 41, 61: image processing device, 13: interpolation unit, 14: noise increase unit, 15: processing unit, 17: interpolation pixel noise reduction unit, 18: reference pixel noise reduction unit, 19: random number addition unit

Claims

1. an interpolation means for applying an interpolation process to a defective pixel of an image by using a pixel other than the defective pixel as a reference pixel; a noise increasing means for increasing noise of the interpolated pixels to which the interpolation process has been applied; 13. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, further comprising: applying at least one of a noise reduction process and a compression encoding process to an image including the interpolated pixels in which the noise has been increased by the noise increasing means.

3. 2. The image processing device according to claim 1, further comprising: a noise determining means for determining a target noise when increasing the noise of the interpolated pixel, based on the pixel value of the interpolated pixel and a mathematical model representing a relationship between the pixel value and a noise variance.

4. a first noise reduction means for applying a first noise reduction process to the interpolated pixels; 4. The image processing apparatus according to claim 3, wherein the noise determining means determines the target noise by using a pixel value obtained after the first noise reduction process is applied to the interpolated pixel and the mathematical model.

5. 5 . The image processing apparatus according to claim 4 , wherein the first noise reduction means applies the first noise reduction process to a local region including the interpolated pixels in the image, or to the image itself.

6. a second noise reduction unit that applies a second noise reduction process to the reference pixels; 4. The image processing device according to claim 3, wherein the noise determination means determines the target noise using a pixel value of the interpolated pixel, a pixel value obtained by applying the second noise reduction process to the reference pixel, and the mathematical model.

7. 7. The image processing apparatus according to claim 6, wherein the second noise reduction means applies the second noise reduction process to a local region including the reference pixel in the image, or to the image itself.

8. 2. The image processing apparatus according to claim 1, wherein said noise increasing means increases said noise by adding a random number to said interpolated pixels.

9. 9. An image processing apparatus according to claim 8, wherein said noise increasing means adds said random number selected from one or more pseudo-random numbers.

10. 9. The image processing apparatus according to claim 8, wherein said noise increasing means generates said random numbers based on a probability distribution that explicitly or implicitly includes a variance as a distribution parameter.

11. 11. The image processing apparatus according to claim 10, wherein the probability distribution is a Gaussian distribution.

12. 11. The image processing apparatus according to claim 10, wherein the variance in the distribution parameter is approximately equal to a difference between a variance of a target noise and a variance of noise of the interpolated pixel obtained by adding the random number.

13. the interpolation process is a process of setting a pixel value obtained by a weighted average of pixel values ​​of the reference pixels as a pixel value of the interpolated pixel, 13. The image processing apparatus according to claim 12, wherein said noise increasing means obtains the noise variance of said interpolated pixel by using a weighting coefficient of said weighted average and the noise variance of said reference pixel.

14. 14. The image processing apparatus according to claim 13, wherein said noise increasing means determines the noise variance of the reference pixels using a mathematical model that represents the relationship between pixel values ​​and noise variance.

15. 15. The image processing apparatus according to claim 14, wherein the noise increasing means calculates the variance of the noise of the reference pixels by using pixel values ​​after noise reduction processing is performed on the reference pixels and the mathematical model.

16. 16. The image processing apparatus according to claim 3, wherein the mathematical model is a model that represents a relationship between an average and a variance of pixel values ​​of a plurality of images having uniform brightness levels.

17. an interpolation step of applying an interpolation process to a defective pixel of an image by using a pixel other than the defective pixel as a reference pixel; a noise increasing step of increasing noise of the interpolated pixels to which the interpolation process has been applied; 13. An image processing method comprising:

18. Computer, an interpolation means for applying an interpolation process to a defective pixel of an image by using a pixel other than the defective pixel as a reference pixel; a noise increasing means for increasing noise of the interpolated pixels to which the interpolation process has been applied; A program that causes the image processing device to function as an image processing device having the above-mentioned configuration.

Citation Information

Patent Citations

  • Image processing unit and imaging device

    JP2009177792A

  • Image processing apparatus and image processing program

    JP2010206703A