Image processing device, image processing program, and image processing method

Abstract

To solve a problem in which there is a demand for an image processing device that can detect and remove each defective pixel even when white or black dots of the same color are adjacent to each other.SOLUTION: An image processing device for correcting defective pixels in image data includes an image processing unit that sets pixels of the same color surrounding a target pixel to be processed as a surrounding pixel group, sets normal pixels among the surrounding pixel group as a normal pixel group for determination, uses the normal pixel group for determination in a determination process, and determines whether the target pixel to be processed is a normal pixel or not.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing program, and an image processing method. [Background technology]

[0002] Conventionally, there has been a known method for correcting defective pixels such as white spots (white defects) and black spots (black defects) that occur in single-chip color image sensors that have a color filter array such as a Bayer array by applying Max and Min filters to eight pixels of the same color surrounding the pixels contained in a 5 x 5 pixel area directly on the Bayer plane.

[0003] Furthermore, in Patent Document 1, when applying the Max and Min filters, not only the surrounding pixel signals of the same color but also the pixel signals of different colors that exist closer to the inside are simultaneously referenced. This makes it possible to correct protruding dot-like defective pixels while minimizing the destruction of the image structure. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-135566 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technique described in Patent Document 1 requires reference to pixels of multiple color components, which increases the amount of calculation and memory usage. In addition, when defective pixels of the same color exist adjacent to the pixel of interest and the reference pixel to be processed, the defective pixels remain without being removed, resulting in a deterioration of image quality.

[0006] Therefore, there is a demand for an image processing device, an image processing program, and an image processing method that are capable of detecting and removing each defective pixel even when white or black dots of the same color are adjacent to each other. [Means for solving the problem]

[0007] The first invention is an image processing device that corrects defective pixels in image data, characterized in having an image processing unit that sets pixels of the same color surrounding a target pixel to be processed as a surrounding pixel group, sets normal pixels from the surrounding pixel group as a normal pixel group for judgment, uses the normal pixel group for judgment in a judgment process, and judges whether the target pixel to be processed is a normal pixel or not.

[0008] The second image processing program of the present invention is characterized in that it causes a computer mounted on an image processing device that corrects defective pixels in image data to function as an image processing unit that determines whether the target pixel to be processed is a normal pixel or not by treating pixels of the same color surrounding the target pixel to be processed as a surrounding pixel group, treating normal pixels among the surrounding pixel group as a normal pixel group for judgment, using the normal pixel group for judgment in the judgment process.

[0009] The third invention of the present invention is an image processing method used in an image processing device that corrects defective pixels in image data, comprising an image processing unit, which sets pixels of the same color surrounding a target pixel to be processed as a surrounding pixel group, sets normal pixels among the surrounding pixel group as a normal pixel group for judgment, uses the normal pixel group for judgment in a judgment process, and judges whether the target pixel to be processed is a normal pixel or not. Effect of the Invention

[0010] According to the present invention, even when white or black dots of the same color exist adjacent to each other, each defective pixel can be detected and removed. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an image capturing device according to an embodiment; [Diagram 2]4 is a flowchart showing the overall operation of the image capturing device according to the embodiment. [Diagram 3] 5 is a flowchart showing the operation (RGB image calculation process) of an image signal processing unit in the image capturing device according to the embodiment. [Figure 4] 5A and 5B are explanatory diagrams illustrating target pixels for defective pixel detection processing in the image signal processing unit according to the embodiment. [Diagram 5] 5 is a flowchart showing the operation of defective pixel detection processing (white point detection processing) in the image signal processing unit according to the embodiment. [Figure 6] 5 is a flowchart showing the operation of defective pixel detection processing (black point detection processing) in the image signal processing unit according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (A) Main embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of an image processing device, an image processing program, and an image processing method according to the present invention will be described in detail with reference to the drawings.

[0013] (A-1) Configuration of the embodiment FIG. 1 is a block diagram showing the configuration of an image capturing device according to an embodiment.

[0014] In FIG. 1, the image capturing device 1 has a capturing section 11, an image signal processing section 12, an image output section 13, a capturing control section 14, and an operation input section 15.

[0015] The image capturing device 1 can be realized by a circuit device having a CPU, a ROM, a RAM, an EEPROM, etc. The ROM stores an image processing program that can be read by the CPU. The image processing program causes the CPU (computer) to function as an image signal processing unit 12. The functions of the image capturing device 1 are realized by the CPU executing the image signal processing unit 12, etc. stored in the ROM.

[0016] The image capturing device 1 is a device that determines shooting parameters based on input from a user or a program, captures an image of a subject using the shooting parameters, performs image processing such as demosaic processing and defective pixel correction processing on the captured data, and outputs the processed captured data.

[0017] The photographing unit 11 may be, for example, a photographing device such as a camera, and has an image sensor, a lens, an aperture, a shutter, etc. The photographing unit 11 photographs an image of a photographing object using photographing parameters (exposure amount, etc.) determined by the photographing control unit 14, and outputs the image data to the image signal processing unit 12. The aperture of the photographing unit 11 may be omitted. In this embodiment, a Bayer array color filter is provided as an example of the function of the image sensor, and the photographing data to be output is also Bayer array data. The image sensor (image sensor) may be provided with a color filter other than the Bayer array, and the format of the output data is not particularly limited.

[0018] The image signal processing unit 12 acquires a signal output from the imaging element of the photographing unit 11 and performs a predetermined signal processing based on the signal from the imaging element. The image signal processing unit 12 converts the signal into data in a desired image data format and outputs the data. The image data format to which the signal is converted may be any commonly used format.

[0019] The image output unit 13 outputs the image data output from the image signal processing unit 12 to the outside. For example, the image output unit 13 may be a communication unit that forms and transmits a packet including image data according to a communication protocol of a network (not shown). Alternatively, the image output unit 13 may output the image data to an SD card or the like, or to a display device such as a liquid crystal display.

[0020] The imaging control unit 14 determines imaging parameters in response to an input from the operation input unit 15, and outputs the parameters to the imaging unit 11. The imaging parameters may be determined by any method.

[0021] The operation input unit 15 can specify parameters for shooting by a user or a program, and can receive a command to execute shooting. The above is an example, and it is assumed that any operation or information for operating the image shooting device 1 can be input.

[0022] (A-2) Operation of the embodiment The operation of the image capturing device 1 in this embodiment will be described with reference to the drawings.

[0023] (A-2-1) Overall operation FIG. 2 is a flowchart showing the overall operation of the image capturing device according to the embodiment.

[0024] <Step S1> In the image capturing device 1, an operation input unit 15 receives information input, a capture execution command, and the like from a user or a program, and outputs them to a capture control unit .

[0025] <Step S2> The photographing control unit 14 calculates photographing parameters from the information input from the operation input unit 15, and outputs the photographing parameters to the photographing unit 11. Similarly, the photographing control unit 14 outputs a photographing execution command input from the operation input unit 15 to the photographing unit 11.

[0026] <Step S3> The photographing unit 11 photographs an object based on the input photographing parameters and photographing execution command, and outputs image raw data. The image raw data is different from commonly used file formats such as bitmap and JPEG, and is raw data output by an imaging element. In this embodiment, the image raw data is image data in a Bayer array.

[0027] <Step S4> The image signal processing unit 12 performs image processing on the raw image data input from the photographing unit 11 to calculate an RGB image. The details of the RGB image calculation process will be described later.

[0028] <Step S5> The image output unit 13 outputs the output image data received from the image signal processing unit 12 to the outside of the image capturing device 1.

[0029] (A-2-2) Image processing (detailed operation of step S4) FIG. 3 is a flowchart showing the operation (calculation of an RGB image) of the image signal processing unit in the image capturing device according to the embodiment.

[0030] <Step S41> The image signal processing unit 12 acquires Bayer image data (data input from the imaging unit 11) to be processed.

[0031] <Step S42> The image signal processing unit 12 performs white balance processing on the Bayer image data (raw data). Bayer image data generally includes data for each color channel of RGB, which are the three primary colors of light. In the case of raw data output from the imaging unit 11 (image sensor), the sensitivity of each color channel often differs. In order to match the sensitivity of each color, white balance processing is performed. Any commonly used method may be used for the white balance processing.

[0032] <Step S43> The image signal processing unit 12 performs defective pixel correction processing. An image sensor is composed of many pixels, and some pixels may output large or small values ​​regardless of the amount of light they receive. A pixel in this state is called a defective pixel. If defective pixel correction is not performed, pixels that are significantly brighter (white dots) or darker (black dots) than the surrounding pixels will appear in the image, leading to degradation of image quality. This processing is performed to suppress degradation of image quality. Details of this defective pixel correction processing will be described later.

[0033] <Step S44> The image signal processor 12 converts the Bayer data that has been subjected to the defective pixel correction process into RGB data. This process may be performed using any commonly known method.

[0034] <Step S45> The image signal processing unit 12 performs color matrix correction processing on the converted RGB data. The response spectrum of each RGB color channel of the image sensor does not necessarily match the RGB response spectrum of the human eye. This processing is performed to mix the three RGB colors to create a color close to what the human senses. The specific processing content performed in this processing may be any commonly known method.

[0035] <Step S46> The image signal processing unit 12 outputs the RGB image data that has been subjected to the color matrix correction processing to the image output unit 13 .

[0036] (A-2-3) Defective pixel correction process (details of operation of step S43) The defective pixel correction process consists of two processes: defective pixel detection process and defective pixel removal process.

[0037] The defective pixel detection process determines whether or not each pixel in the Bayer data is a defective pixel. The defective pixel detection process will be described in detail later.

[0038] The defective pixel removal process corrects the pixel values ​​of pixels that are determined to be defective in the defective pixel detection process described above (pixels that are marked as defective in steps S107 and S205 described below). The specific content of the defective pixel removal process may be any commonly known method. As an example, a method may be used in which the average value of the surrounding pixels is set as the pixel value of the pixel of interest.

[0039] The detailed operation of the defective pixel detection process will be described with reference to FIGS.

[0040] Note that the processing explanation below is for one pixel, so by applying this processing to all pixels of the Bayer data, it is possible to detect defective pixels in the entire image. Figure 4(A) is a diagram showing an example of a Bayer array (a range of 5 x 5 pixels). In the following, as shown in Figure 4(B), the pixel that is the subject of defective pixel detection will be called the pixel of interest (marked with a circle), and the pixels of the same color (R in the example in Figure 4(B)) surrounding the pixel of interest will be called surrounding pixels (marked with a cross).

[0041] FIG. 5 is a flowchart showing the process of detecting defective pixels such as white dots.

[0042] <s101> The image signal processor 12 obtains the pixel values ​​of the surrounding pixels.

[0043] <s102> The image signal processing unit 12 sorts the acquired pixel values ​​of the surrounding pixels in ascending or descending order of pixel values.

[0044] <s103> The image signal processing unit 12 compares the maximum value of the surrounding pixel values ​​with the sum of the second largest pixel value and a threshold value (first threshold value for detecting defective pixels of white dots) and determines whether the maximum value of the surrounding pixels is greater. If there is one white dot among the surrounding pixels, the difference between the maximum value of the surrounding pixels and the second pixel value will be large. If the difference between the above-mentioned maximum value and the second pixel value is greater than the threshold value, the image signal processing unit 12 determines that the above-mentioned maximum value is a white dot and executes step S104. If not, it determines that the above-mentioned maximum value is a normal pixel and executes step S105.

[0045] <s104> When the maximum value of the surrounding pixel values ​​is determined to be a white point in step S103, the image signal processing unit 12 sets the second largest pixel value of the surrounding pixels as the judgment value, which is the largest pixel value among the surrounding pixels that are likely to be normal.

[0046] <s105> On the other hand, if it is determined in step S103 that the maximum value of the surrounding pixel values ​​is a normal pixel, the image signal processing unit 12 sets the maximum value of the surrounding pixels as the determination value.

[0047] <s106> The image signal processing unit 12 compares the pixel value of interest with the sum of the judgment value and the threshold value, and determines whether the pixel value of interest is greater. If the pixel of interest is a white point, the difference between the pixel value of interest and the judgment value will be large. If the pixel value of interest is greater than the sum of the judgment value and the threshold value, the image signal processing unit 12 determines the pixel of interest to be a defective pixel and executes step S107. If not, the pixel of interest is determined to be a normal pixel and executes step S108.

[0048] <s107> If the pixel of interest is determined to be a defective pixel in step S106 described above, the image signal processing unit 12 records the pixel of interest as a defective pixel.

[0049] <s108> If the pixel of interest is determined to be a normal pixel in step S106 described above, the image signal processing unit 12 records the pixel of interest as a normal pixel.

[0050] FIG. 6 is a flowchart showing the process of detecting defective pixels such as black dots.

[0051] The processing of the same steps in Fig. 6 as Fig. 5 (steps S101 and S102) is omitted because it is the same as Fig. 5. Also, when the processing of Fig. 6 is performed following the processing of Fig. 5, the acquired data of surrounding pixel values ​​and the like are held, so there is no need to perform the processing of steps S101 and S102 again.

[0052] <s201> The image signal processing unit 12 compares the minimum value of the surrounding pixel values ​​with the difference between the second smallest pixel value and a threshold value (a second threshold value for detecting defective pixels of black dots) and determines whether the minimum value of the surrounding pixels is smaller. If there is one black dot among the surrounding pixels, the difference between the second smallest pixel value of the surrounding pixels and the minimum value will be large. If the difference between the second smallest pixel value and the above-mentioned minimum value is greater than the threshold value, the image signal processing unit 12 determines that the above-mentioned minimum value is a black dot and executes step S202. If not, it determines that the above-mentioned minimum value is a normal pixel and executes step S203.

[0053] <s202> When the minimum value of the surrounding pixel values ​​is determined to be a black point in step S201, the image signal processing unit 12 sets the second smallest pixel value of the surrounding pixels as the judgment value, which is the smallest pixel value among the surrounding pixels that are likely to be normal.

[0054] <s203> On the other hand, if it is determined in step S201 that the minimum value of the surrounding pixel values ​​is a normal pixel, the image signal processing unit 12 sets the minimum value of the surrounding pixels as the determination value.

[0055] <s204> The image signal processing unit 12 compares the pixel value of interest with the difference between the judgment value and the threshold value, and determines whether the pixel value of interest is smaller. If the pixel value of interest is smaller than the difference between the judgment value and the threshold value, the image signal processing unit 12 determines the pixel of interest as a defective pixel and executes step S205. If not, the pixel of interest is determined as a normal pixel and executes step S206.

[0056] <s205> If the pixel of interest is determined to be a defective pixel in step S204 described above, the image signal processing unit 12 records the pixel of interest as a defective pixel.

[0057] <s206> If the pixel of interest is determined to be a normal pixel in step S204 described above, the image signal processing unit 12 records the pixel of interest as a normal pixel.

[0058] (A-3) Effects of the embodiment According to this embodiment, even when white or black dots of the same color exist adjacent to each other in Bayer image data, it is possible to detect and remove each defective pixel.

[0059] (B) Although various modified embodiments have been mentioned in the above embodiment, the same can also be applied to the following modified embodiments.

[0060] (B-1) In the above-described embodiment, an example was shown in which processing for detecting defective pixels of white dots was performed after processing for detecting defective pixels of black dots. However, the order of execution may be reversed, and only one of the processes may be performed depending on the application.

[0061] (B-2) In the above-described embodiment, the defective pixel correction process is performed on Bayer image data that has been subjected to white balance correction process. However, the defective pixel correction process may also be performed on Bayer image data that has not been subjected to white balance correction process.

[0062] (B-3) In the above-described embodiment, an example was shown in which the image capturing device 1 calculates an RGB image including defective pixel correction processing for the Bayer image data. However, the image capturing device 1 may output the Bayer image data (raw data) to another information processing device, and the other information processing device (having functions similar to those of the image signal processing unit 12) may calculate an RGB image from the Bayer image data.

[0063] (B-4) In the above embodiment, in steps S103, S104, and S105 of the process for detecting defective white dot pixels, the maximum value and the second pixel value of the surrounding pixels are processed, but subsequent to the above process, the second and third pixel values ​​may be processed, or the above process may be repeated any number of times (the same applies to the process for detecting defective black dot pixels). In this case, there is a possibility that defective pixels contained in the surrounding pixels will be detected as many times as the process is repeated. [Explanation of symbols]

[0064] 1...image capturing device, 11...capturing section, 12...image signal processing section, 13...image output section, 14...capturing control section, 15...operation input section.

Claims

1. 1. An image processing device that corrects defective pixels in image data, comprising: An image processing device comprising: an image processing unit that determines whether the target pixel to be processed is a normal pixel or not by using, in a judgment process, pixels of the same color surrounding the target pixel to be processed as a surrounding pixel group, normal pixels among the surrounding pixel group as a normal pixel group for judgment, and using the normal pixel group for judgment in a judgment process.

2. 2. The image processing apparatus according to claim 1, wherein the normal pixel group for judgment is a pixel group obtained by excluding a pixel having an extremely large pixel value from the surrounding pixel group when the pixel has such a pixel.

3. 3. The image processing apparatus according to claim 1, wherein the normal pixel group for judgment is a pixel group obtained by excluding a pixel having an extremely small pixel value from the surrounding pixel group when the pixel has such a pixel.

4. A computer installed in an image processing device that corrects defective pixels in image data, An image processing program characterized by functioning as an image processing unit that determines whether the target pixel to be processed is a normal pixel or not by setting pixels of the same color surrounding the target pixel to a surrounding pixel group, setting normal pixels among the surrounding pixel group to a normal pixel group for judgment, using the normal pixel group for judgment in a judgment process, and

5. 1. An image processing method for use in an image processing device that corrects defective pixels in image data, comprising: An image processing unit is provided. The image processing method is characterized in that the image processing unit sets pixels of the same color surrounding the target pixel to be processed as a surrounding pixel group, sets normal pixels among the surrounding pixel group to a normal pixel group for judgment, uses the normal pixel group for judgment in the judgment process, and judges whether the target pixel to be processed is a normal pixel or not.

Images