Imaging device, control method for imaging device, and program

Abstract

To provide an imaging apparatus that can accurately correct a pixel group to be corrected.SOLUTION: An imaging apparatus is provided which has an imaging unit A102 and an imaging unit B106 having different structures. The imaging apparatus determines pixels corresponding to a pixel group including a plurality of adjacent defective pixels as pixels to be corrected based on an image related to output from the imaging unit A102, and calculates a correlation value between the pixels to be corrected and pixels around the pixels to be corrected based on an image related to output from the imaging unit B106 and the difference in characteristics between the imaging unit A and the imaging unit B. The imaging apparatus calculates a correction value of the pixel value of the pixels to be corrected based on the calculated correlation value.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method for the imaging device, and a program.

Background Art

[0002] In recent years, a SPAD (Single Photon Avalanche Diode) sensor has been proposed as a new image sensor. A general image sensor, a CMOS (Complementary Metal Oxide Semiconductor) sensor, has a mechanism for measuring the amount of light accumulated in pixels during a certain period when reading light as an electrical signal. Therefore, a CMOS sensor is easily affected by noise under low illuminance where the amount of light is small. On the other hand, in a SPAD sensor, when photons are input to a pixel and converted into charges, the charges are amplified by avalanche amplification and then counted as the number of photons, so it is less affected by noise even under low illuminance. In addition, the SPAD sensor has high time resolution and also has a global shutter function, so it is expected to be mounted not only on digital cameras and digital video cameras but also on imaging devices for monitoring and industrial applications.

[0003] However, generally, a SPAD sensor needs to keep a certain distance between the sensitivity regions of adjacent pixels in terms of structure, and the aperture ratio decreases as the size of one pixel becomes smaller. Therefore, it is difficult to increase the number of pixels, and an imaging device equipped with both a SPAD sensor and a CMOS sensor that is easy to increase the number of pixels has been studied. Also, in industrial applications, an imaging device equipped with both a sensor for imaging the infrared region such as SWIR (Short wavelength infra-Red) and a SPAD sensor has been studied. Further, Patent Document 1 discloses an apparatus for estimating the output of a defective pixel of a CCD (Charge Coupled Device) element from the outputs of other CCD elements. A defective pixel is a pixel whose output does not satisfy a predetermined condition.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Application Publication No. 6-165044 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, in SPAD sensors, for example, the charge generated not only in normal pixels but also in defective pixels is avalanche amplified, and this effect extends to surrounding pixels, resulting in the creation of defective pixel groups containing multiple adjacent defective pixels. If there is only one defective pixel, it is possible to easily correct it by calculating correlation information such as the edge direction from surrounding pixels. However, in the case of defective pixel groups containing multiple adjacent defective pixels, it becomes difficult to accurately calculate correlation information, so correction marks tend to be noticeable when the pixel values ​​of the defective pixel group are corrected. The present invention aims to provide an imaging device that can accurately correct pixel groups that are subject to correction, as in the example above. [Means for solving the problem]

[0006] An imaging apparatus according to one embodiment of the present invention comprises a first imaging means and a second imaging means having different structures; a determination means for determining, based on the image output of the first imaging means, pixels that correspond to a group of adjacent pixels whose output does not satisfy predetermined conditions and are to be corrected; a first calculation means for calculating a correlation value between the pixels to be corrected and the pixels surrounding the pixels to be corrected, based on the image output of the second imaging means and the difference in characteristics between the first imaging means and the second imaging means; and a second calculation means for calculating a correction value for the pixel value of the pixels to be corrected based on the calculated correlation value. The first calculation means calculates the correlation value based on the pixel values ​​of the reference pixels in the image output of the second imaging means, which are located at the same relative positions as the pixels to be corrected in the image output of the first imaging means and the surrounding pixels. do. [Effects of the Invention]

[0007] According to the imaging device of the present invention, it is possible to correct the pixel group to be corrected with high accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the configuration of the imaging device. [Figure 2] This is a block diagram showing an example configuration of the defective pixel correction unit A. [Figure 3] This is a flowchart illustrating an example of the operation process of the defective pixel correction unit A. [Figure 4] This is a block diagram showing an example of the configuration of the defective pixel group correction unit. [Figure 5] This figure shows an example of a RAW image 1 containing a group of defective pixels. [Figure 6] This figure shows an example of RAW image 2. [Figure 7] This figure shows an example of RAW image 2. [Figure 8] This figure shows an example of a RAW image 1 containing a group of defective pixels. [Figure 9] This figure shows an example of a RAW image 2 input to the correlation value calculation unit. [Figure 10] This figure shows an example of a RAW image 2 input to the correlation value calculation unit. [Figure 11] This is a block diagram showing an example of the configuration of the correlation value calculation unit. [Figure 12] This figure shows the relationship between the brightness value of RAW image 2 and the confidence level α. [Modes for carrying out the invention]

[0009] (Example 1) Figure 1 shows the configuration of the imaging device according to this embodiment. The imaging device shown in Figure 1 comprises a system control unit 100, an optical system A101, an imaging unit A102, a defective pixel correction unit A103, a signal processing unit A104, and an optical system B105. The imaging device also comprises an imaging unit B106, a defective pixel correction unit B107, a signal processing unit B108, a recording processing unit 109, and a user input unit 110. The optical system A101 through the user input unit 110 are connected to the system control unit 100.

[0010] The system control unit 100 has a processor such as a CPU (Central Processing Unit) and controls the entire imaging device. The system control unit 100 detects a user's operation via, for example, the user input unit 110, and controls the operations of the optical system A101 to the recording processing unit 109.

[0011] The optical system A101 includes a lens group having a zoom lens and a focus lens, an aperture adjusting device, etc., and forms a subject image on the imaging unit A102. In this example, the imaging unit A102 is a SPAD sensor and has a plurality of pixels arranged in a two-dimensional matrix. Each of the plurality of pixels has a filter corresponding to one of the three colors of a Bayer pattern color filter (red (R), green (G), blue (B)). The light beam of the subject that has passed through the optical system A101 is converted into an electrical signal by the SPAD sensor, and a RAW image, which is a digital image, is output.

[0012] The defective pixel correction unit A103 corrects defective pixels in the RAW image obtained by imaging by the imaging unit A102. A defective pixel is a pixel whose output does not satisfy a predetermined condition (for example, a pixel whose output value is outside a predetermined range), and is generally also called a damaged pixel. Details of the defective pixel correction unit A103 will be described later.

[0013] The signal processing unit A104 performs signal processing such as white balance (WB) processing, de-Bayer processing, color reproduction processing, sharpness processing, tone mapping processing, gamma correction processing, etc. on the RAW image output by the defective pixel correction unit A103. The image data generated by the signal processing unit A104 is output to the recording processing unit 109.

[0014] Optical system B105, like optical system A101, is equipped with a lens group having a zoom lens and a focus lens, an aperture adjustment device, etc., and forms an image of the subject on the imaging unit B106. Imaging unit B106 is an imaging means with a different structure from imaging unit A102. Imaging unit B106 has an image sensor such as a CCD or CMOS sensor and an A / D conversion element that converts the electrical signal output from the image sensor into a digital image, which is a RAW image. Imaging unit B106 employs a photoelectric conversion method that measures the amount of light, while imaging unit A, which is a SPAD sensor, employs a photoelectric conversion method that counts the number of photons. In other words, imaging unit A (first imaging means) 102 and imaging unit B (second imaging means) 106 have at least different photoelectric conversion principles.

[0015] The image sensor in the imaging unit B106 has multiple pixels arranged in a two-dimensional matrix, and each of the multiple pixels has a Bayer pattern color filter. Hereafter, a sensor having a Bayer pattern color filter will also be referred to as a "Bayer array sensor". As the image sensor of the imaging unit B106, for example, an image sensor with a larger number of pixels than a SPAD sensor is used. The defective pixel correction unit B107 corrects defective pixels in the RAW image obtained by imaging by the imaging unit B106. The defective pixel correction unit B107 extracts edges by performing filtering such as a bandpass filter on the pixels surrounding the defective pixel, for example, and determines the direction of the edge. The defective pixel correction unit B107 calculates a correction value for the pixel value of the defective pixel using the pixel value of a pixel that has a color filter of the same color as the defective pixel and is located in the direction of the edge of the defective pixel. The defective pixel correction unit B107 corrects the pixel value of the defective pixel with the calculated correction value. As described above, the imaging device can perform defective pixel correction with minimal correction marks by referring to correlation information such as edge direction for defective pixels.

[0016] The signal processing unit B108 performs signal processing such as white balance processing, debayering, color reproduction processing, sharpness processing, tone mapping processing, and gamma correction processing on the RAW image output from the defective pixel correction unit B107. The image data generated by the signal processing unit B108 is output to the recording processing unit 109.

[0017] The recording processing unit 109 records the image data output from the signal processing unit A 104 and the signal processing unit B 108 in a recording device such as a solid state drive (SSD). The recording processing unit 109 may record the image data selected by the user via the user input unit 110 among the image data output from the signal processing unit A 104 and the signal processing unit B 108. Further, the recording processing unit 109 may record the image data output from the signal processing unit A 104 and the signal processing unit B 108 in an associated manner, or may record them in a superimposed manner as one piece of image data. Further, the recording device does not necessarily have to be built in the imaging device, and the image data may be recorded in a server or the like via a network. With respect to the imaging device as described above, the defective pixel correction unit A 103, which is a part related to the present invention, will be described below.

[0018] The defective pixel correction unit A 103 includes a defective pixel determination unit 200, a defective pixel group determination unit 201, a defective pixel correction unit 202, a defective pixel group correction unit 203, a selector A 204, and a selector B 205.

[0019] The defective pixel determination unit 200 determines whether each pixel of the RAW image (RAW image 1) input from the imaging unit A 102 is a defective pixel specified in advance, for example. The defective pixels are detected in advance by, for example, defective pixel inspection or the like. Position data of the defective pixels specified in advance is stored in a predetermined storage unit (not shown). The defective pixel determination unit 200 refers to the position data in the storage unit and determines whether each pixel of the RAW image 1 is a defective pixel specified in advance.

[0020] The defective pixel group determination unit 201 determines whether a pixel in the RAW image 1 belongs to, for example, a predetermined group of defective pixels. A defective pixel group is a group of pixels that includes multiple adjacent defective pixels. Each pixel included in the defective pixel group is a pixel to which at least one or more defective pixels are connected (or adjacent). In other words, a defective pixel group may include pixels where inter-pixel crosstalk occurs. For example, position data of defective pixels included in a predetermined group of defective pixels is stored in the predetermined storage unit mentioned above. The defective pixel group determination unit 201 refers to the position data in the storage unit and determines whether a pixel in the RAW image 1 belongs to a predetermined group of defective pixels.

[0021] The defective pixel correction unit 202 corrects the pixel values ​​of defective pixels in RAW image 1 using the same processing as the defective pixel correction unit B107. The defective pixel group correction unit 203 corrects the pixel values ​​of the defective pixel group in RAW image 1 based on the correlation value calculated based on the RAW image 2 input via the system control unit 100.

[0022] Selector A204 outputs the corrected pixel value from the defective pixel correction unit 202 or the corrected pixel value from the defective pixel group correction unit 203 to Selector B205, according to the determination result of the defective pixel group determination unit 201. Selector B205 outputs the output from Selector A or the pixel value input from the imaging unit A102 to the signal processing unit A104, according to the determination result of the defective pixel determination unit 200. The signal processing unit A104 performs various signal processing based on the output from Selector B205.

[0023] Figure 3 is a flowchart illustrating an example of the operation process of the defective pixel correction unit A. In Figure 3, S indicates the step number corresponding to each process according to this flowchart. In SS301, the defective pixel correction unit A103 reads the RAW image. As a result, RAW image 1 is input from the imaging unit A102 to the defective pixel determination unit 200, the defective pixel group determination unit 201, the defective pixel correction unit 202, and the defective pixel group correction unit 203, respectively. Also, via the system control unit 100, RAW image 2, in which the pixel values ​​of defective pixels have been corrected by the defective pixel correction unit B107, is input to the defective pixel group correction unit 203.

[0024] Next, in S302, the defective pixel determination unit 200 determines whether each pixel of the RAW image 1 input from the imaging unit A102 is a predetermined defective pixel. If a pixel in the RAW image 1 is a defective pixel, the process proceeds to S303. In the processing from S303 onward, the selector B205 outputs the output of the selector A204, i.e., the pixel value corrected by either the defective pixel correction unit 202 or the defective pixel group correction unit 203, to the signal processing correction unit A104. If a pixel in the RAW image 1 is not a defective pixel, the selector B205 outputs the pixel value input from the imaging unit A102 directly to the signal processing unit A104. Note that the determination of whether a pixel is defective is not limited to a method based on whether it corresponds to a predetermined defective pixel. For example, isolated point detection may be performed using a bandpass filter or the like to determine whether a pixel is defective from the RAW image 1.

[0025] Next, in S303, the defective pixel group determination unit 201 determines whether each pixel of the RAW image 1 input from the imaging unit A102 belongs to a predetermined defective pixel group. If the pixels of the RAW image 1 do not belong to a defective pixel group, the process proceeds to S304. In S304, the defective pixel correction unit 202 corrects the pixel value of the pixel in question by performing the same processing as the defective pixel correction unit B107, and the selector A204 outputs the corrected pixel value to the selector B205. The selector B205 outputs the pixel value input from the selector A204 to the signal processing unit A104.

[0026] If a pixel in RAW image 1 corresponds to a group of defective pixels, the process proceeds to S305. In S305, the defective pixel group correction unit 203 corrects the pixel value of the pixel based on a correlation value calculated based on the RAW image 2 input from the defective pixel correction unit B107. Then, selector A204 outputs the corrected pixel value to selector B205. Selector B205 outputs the pixel value input from selector A to signal processing unit A104. According to the imaging device of this embodiment, by correcting a group of defective pixels including multiple adjacent defective pixels based on a correlation value calculated based on the image output by imaging unit B106, which has a different structure from imaging unit A102, it is possible to obtain a correction result with reduced correction marks.

[0027] Figure 4 is a block diagram showing an example of the configuration of the defective pixel group correction unit. The defective pixel group correction unit 203 includes an alignment processing unit 400, a correlation value calculation unit 401, and a correction processing unit 402.

[0028] The alignment processing unit 400 aligns the RAW image 1 input from the imaging unit A102 with the RAW image 2 input from the defective pixel correction unit B107 via the system control unit 100 (alignment processing). The alignment processing is performed taking into account the difference in the number of pixels of each RAW image. Specifically, a brightness image is calculated for each RAW image, and alignment is performed by feature point matching. The RAW image 2 after the alignment processing is output to the correlation value calculation unit 401. The correlation value calculation unit 401 calculates a correlation value to correct the defective pixel group of RAW image 1 based on the RAW image 2 input from the alignment processing unit 400.

[0029] Figure 5 shows an example of RAW image 1 containing a group of defective pixels. The hatched pixels in Figure 5 are the defective pixels. For G22, the pixel to be corrected (the corrected pixel), the surrounding pixels G02, G11, B12, G13, G20, R21, R23, G24, G31, B32, G33, and G42 are also defective pixels and form a group of defective pixels.

[0030] Figure 6 shows an example of a RAW image 2 input to the correlation value calculation unit. In this embodiment, the number of pixels in RAW image 2 is twice the number of pixels in RAW image 1, both horizontally and vertically. Therefore, g44, r45, b54, and g55 are located at the same relative positions as the correction target pixel G22 in RAW image 1. The correlation value calculation unit 401 calculates the correlation value between G22, the correction target pixel in RAW image 1, and G00, G04, G40, and G44, which are non-defective pixels around G22 and are the same color as G22 (same-color pixels), from the pixel values ​​of the reference pixels in RAW image 2. The correlation value calculation unit 401 determines the reference pixels based on the position of the correction target pixel G22 in RAW image 1. More specifically, in RAW image 2, g44 and g55, which are pixels (G pixels) of the same color as G22 and are located at the same relative positions as the correction target pixel G22, are determined to be the reference pixels. In addition, in RAW image 2, the surrounding pixels of the same color as g44 and g55 are determined to be the reference pixels. In this example, the pixels g00 and g11, g08 and g19, g80 and g91, and g88 and g99, which are in the same relative positions as G00, G04, G40, and G44, are determined to be the reference pixels. In other words, the pixels whose positional relationship with g44 and g55 in RAW image 2 is the same as the positional relationship between G22 and G00, G04, G40, and G44 in RAW image 1 are determined to be the reference pixels.

[0031] The correlation value COR can be calculated, for example, by formula (1). COR = AVE1 / AVE2 AVE1 = (g44 + g55) / 2 AVE2=(g00+g11+g08+g19+g80+g91+g88+g99) / 8 ...(Formula 1)

[0032] AVE1 corresponds to the average value of G22 in RAW image 1. AVE2 corresponds to the average value of non-defective same-color pixels G00, G04, G40, and G44 surrounding G22. COR is a correlation value that shows the ratio of AVE1 to AVE2. The correlation value COR calculated by the correlation value calculation unit 401 is input to the correction processing unit 402.

[0033] The correction processing unit 402 corrects the pixel values ​​of the RAW image 1 input from the imaging unit A102 based on the correlation value COR input from the correlation value calculation unit 401. For example, the correction processing unit 402 calculates the corrected pixel value (corrected value) of the pixel to be corrected, G22, using the correlation value COR with the surrounding non-defective pixels of the same color G00, G04, G40, and G44, using equation (2). G22=COR×(G00+G04+G40+G44) / 4...(Formula 2)

[0034] The corrected pixel value of G22 is output to selector A204, and correction is performed using the corrected value of the G22 pixel. The correlation value calculation unit 401 similarly calculates the correlation value between the pixels to be corrected and surrounding non-defective pixels of the same color, including pixels other than G22 (such as R23) included in the defective pixel group of RAW image 1, based on the pixel value of the corresponding reference pixel in RAW image 2. Then, the correction processing unit 402 calculates the corrected pixel value of the pixels to be corrected based on the calculated correlation value.

[0035] In Example 1, the correlation value between the pixel to be corrected and surrounding non-defective pixels of the same color is calculated based on the difference in characteristics between imaging unit A102 and imaging unit B106. Specifically, the correlation value is calculated from the pixel value of a reference pixel determined based on the difference in the number of pixels between imaging unit A102 and imaging unit B106. This makes it possible to accurately calculate the correlation value between the pixel to be corrected and surrounding pixels of the same color, even for groups of defective pixels that include multiple adjacent defective pixels, and to obtain good correction results with reduced correction marks.

[0036] In this embodiment, we have explained an example where RAW image 2 is larger than RAW image 1, and the number of pixels in the horizontal and vertical directions is a constant multiple (the same integer multiple in the examples of Figures 5 and 6), but the correlation value can be calculated similarly in other cases as well. The imaging device calculates the correlation value after converting RAW image 2 to the same number of pixels as RAW image 1 by enlarging or reducing it beforehand. Specifically, the imaging device first performs debayer processing on RAW image 2 to generate R, G, and B images with the same number of pixels as RAW image 2. Next, the imaging device enlarges or reduces the R, G, and B images using interpolation methods such as bilinear interpolation so that they have the same number of pixels as RAW image 1. Then, the imaging device generates RAW image 3 with the same Bayer pattern as RAW image 1 by extracting one pixel each from the R, G, and B images so that they have the same Bayer pattern as RAW image 1. Then, the imaging device calculates the correlation value from each pixel of RAW image 3 that is in the same position as the corrected pixel of RAW image 1 and the surrounding same-colored pixels that are not defective pixels. This makes it possible to handle cases where RAW image 2 is smaller than RAW image 1, or where the number of pixels in the horizontal and vertical directions is not a constant multiple of the number of pixels in RAW image 1.

[0037] (Example 2) The imaging apparatus of Example 2 will be described in which the imaging unit B106 uses an image sensor (monochromatic sensor) such as a monochromatic CCD or CMOS sensor, which has a color filter of the same color for all pixels. The configuration of the imaging unit A102 is the same as in Example 1. The configuration of the entire imaging apparatus, the defective pixel correction unit A103, and the defective pixel group correction unit 203 are the same as in Example 1, so a detailed explanation will be omitted. In addition, the defective pixel group in RAW image 1 in Example 2 is assumed to occur at the hatched locations shown in Figure 5 above.

[0038] Figure 7 shows an example of a RAW image 2 input to the correlation value calculation unit in Example 2. The number of pixels in RAW image 2 is twice the number of pixels in RAW image 1, both horizontally and vertically, and l44, l45, l54, and l55 are located at the same relative positions as the correction target pixel G22 in RAW image 1. The correlation value calculation unit 401 calculates the correlation value between the correction target pixel G22 in RAW image 1 and the non-defective same-color pixels G00, G04, G40, and G44 surrounding G22, similar to the first example, from the pixel values ​​of the reference pixels determined from RAW image 2. The correlation value calculation unit 401 determines the reference pixels based on the difference in characteristics between the imaging unit A102 and the imaging unit B106 (in the second example, the difference in color filter sampling). In RAW image 2, the pixels located at the same relative position as the correction target pixel and the pixels surrounding that pixel are determined as reference pixels.

[0039] When the pixel to be corrected is G22, in RAW image 2 of Figure 7, I44, I45, I54, and I56, which are in the same position relative to G22, are determined as reference pixels. In RAW image 2, I00, I01, I10, and I11, which are in the same position relative to G00, are determined as reference pixels. Also, I08, I09, I18, and I19, which are in the same position relative to G04, are determined as reference pixels. Furthermore, I80, I81, I90, and I91, which are in the same position relative to G40, are determined as reference pixels. Furthermore, I88, I89, I98, and I99, which are in the same position relative to G44, are determined as reference pixels.

[0040] The correlation coefficient COR can be calculated, for example, by formula (3). COR = AVE1 / AVE2 AVE1 = (l44 + l45 + l54 + l55) / 4 AVE2=(l00+l01+l10+l11 +l08+l09+l18+l19 +l80+l81+l90+l91 (+l88+l89+l98+l99) / 16 ...(Formula 3)

[0041] AVE1 corresponds to the average value of G22 in RAW image 1. AVE2 corresponds to the average value of non-defective same-color pixels G00, G04, G40, and G44 surrounding G22. The correction processing unit 402 corrects the pixel values ​​of the pixels to be corrected based on the calculated correlation value COR, similar to Example 1.

[0042] The correlation value calculation unit 401 calculates the correlation value between a defective pixel in the imaging unit A102 and surrounding non-defective pixels of the same color, taking into account the difference in pixel count and color filter sampling from the image output from imaging unit B106, which has a different pixel count and color filter sampling. This makes it possible to calculate the correlation between the pixel to be corrected and surrounding pixels of the same color with high accuracy, even if multiple defective pixels are in adjacent groups of defective pixels, and to calculate a good correction result with reduced correction marks. In this embodiment, an example is given where the number of pixels differs between imaging unit A102 and imaging unit B106, but imaging unit B106 with the same number of pixels but different color filter sampling may also be used.

[0043] Furthermore, although this embodiment describes an example where the imaging unit A102 is a Bayer array sensor and the imaging unit B106 is a monochrome sensor, it is also possible to handle cases where, for example, the imaging unit A102 is a monochrome sensor and the imaging unit B106 is a Bayer pattern sensor. Specifically, the correlation value calculation unit 401 refers to the color filter of a pixel in RAW image 2 that is in the same position relative to the pixel to be corrected in RAW image 1. The correlation value calculation unit 401 can correct the pixel value of the pixel to be corrected by calculating a correlation value based on the pixel of RAW image 2 and the pixel values ​​of surrounding pixels that are the same color as the referenced color filter.

[0044] Furthermore, the present invention is also applicable when the imaging unit A102 and the imaging unit B106 each have different color filter arrangements. The correlation value calculation unit 401 determines a pixel in RAW image 2 that is the same color as the pixel to be corrected in RAW image 1 and is in the same relative position as the pixel to be corrected. The correlation value calculation unit 401 can perform correction by calculating a correlation value based on the pixel value of the determined pixel and the pixel values ​​of surrounding pixels that are the same color as the color filter of that pixel. By correcting the pixel value in this way, correction can be performed not only on sensors with Bayer arrays but also on sensors with complementary color filters or RGBW filters.

[0045] (Example 3) In the imaging device of Example 3, the imaging unit A102 and the imaging unit B106 have different corresponding wavelength bands, and the correlation value is calculated based on the difference in wavelength bands. Specifically, the imaging device calculates the correlation value based on edge information calculated based on the RAW image 2 related to the output of imaging unit B. In this example, imaging unit B106 is equipped with an image sensor that images the infrared region, such as SWIR. Note that the overall configuration of the imaging device, the defective pixel correction unit A103, and the defective pixel group correction unit 203 are the same as in Example 1, so a detailed explanation is omitted.

[0046] Figure 8 shows an example of RAW image 1 containing a group of defective pixels. The hatched pixels in Figure 5 are the defective pixels. For the pixel G22, which is the target of correction, the surrounding pixels G11, B12, G13, R21, R23, G24, G31, B32, and G33 are also defective.

[0047] Figure 9 shows an example of a RAW image 2 input to the correlation value calculation unit. In this embodiment, each pixel of the RAW image 2 outputs a pixel value corresponding to the SWIR wavelength band. This makes it possible to image subject features that cannot be detected in the visible light wavelength band. However, since the imaging results for the subject differ between the SWIR wavelength band and the visible light wavelength band, the ratio of pixel values ​​cannot be used directly as a correlation value, as in Embodiments 1 and 2. Therefore, the correlation value calculation unit 401 calculates a correlation value related to the direction of the subject edge at IR22, which is located at the same position relative to the correction target pixel G22, from the RAW image 2 input from the alignment processing unit 400.

[0048] The correlation value HVDiff can be calculated, for example, by equation (3). HVDiff = HDiff - VDiff HDiff=|2×IR22-IR20-IR24|+|IR21-IR23| VDiff=|2×IR22-IR02-IR42|+|IR12-IR32| ...(Formula 3)

[0049] HDiff represents the correlation value of the intensity of horizontal edges, and will be a large value when the subject has vertical linear edges. IR22, IR20, IR24, IR21, and IR23 are reference pixels for calculating HDiff. VDiff represents the correlation value of the intensity of vertical edges, and will be a large value when the subject has horizontal linear edges. IR22, IR02, IR42, IR12, and IR32 are reference pixels for VDiff. HVDiff is the difference value between HDiff and VDiff. The reason why HDiff and VDiff are calculated based on the difference between every other pixel is that the imaging unit A102 has a Bayer array, and pixels of the same color are sampled every other pixel.

[0050] The correlation value HVDiff calculated by the correlation value calculation unit 401 is input to the correction processing unit 402. The correction processing unit 402 uses the correlation value HVDiff input from the correlation value calculation unit 401 to correct the pixel values ​​of the pixels to be corrected on the RAW image 1 input from the imaging unit A102.

[0051] The correction processing unit 402 determines that if HVDiff is positive, vertical linear edges exist in the subject, meaning there is a strong vertical correlation. If HVDiff is negative, the correction processing unit 402 determines that horizontal linear edges exist in the subject, meaning there is a strong horizontal correlation. Based on the above determination, the correction processing unit 402 calculates the correction value for G22. Specifically, according to equation (4), if HVDiff is positive, the correction value for G22 is calculated using the non-defective, surrounding same-color pixels G02 and G42 in the vertical direction. If HVDiff is negative, the corrected pixel value (correction value) for G22 is calculated using the non-defective, surrounding same-color pixels G20 and G24 in the horizontal direction. The corrected pixel value for G22 is output to selector A204.

[0052] if(HVDiff>0){ G22 = (G02 + G42) / 2 else{ G22 = (G20 + G24) / 2 } ... (Formula 4) Furthermore, the correction processing unit 402 may not only separate the processing of the correlation value HVDiff into positive and negative values, but may also calculate the correction value from the average value of same-color pixels in both the vertical and horizontal directions by setting a dead zone based on the intensity of the signal value.

[0053] According to the imaging device of Example 3, even when image sensors that detect different wavelength bands are used in imaging unit A102 and imaging unit B106, the correlation value between the pixel to be corrected and surrounding pixels of the same color can be calculated with high accuracy based on the edge information of the subject.

[0054] (Example 4) The imaging device of Example 4 calculates a correlation value based on the difference in noise characteristics between imaging unit A102 and imaging unit B106. In this example, imaging unit B106 is equipped with an image sensor such as a CCD or CMOS sensor that has different noise characteristics from imaging unit A102. Generally, CCDs and CMOS sensors are noisier than SPAD sensors, especially in low-light environments. Below, a correlation value calculation method that reduces the influence of noise will be described, even when the correlation value is calculated based on the output of an image sensor such as a CCD or CMOS sensor that is noisy. Note that the overall configuration of the imaging device, the defective pixel correction unit A103, and the defective pixel group correction unit 203 are the same as in Example 1, so a detailed explanation will be omitted. Also, the defective pixel group in RAW image 1 in Example 4 is assumed to occur at the hatched locations shown in Figure 5 above.

[0055] Figure 10 shows an example of a RAW image 2 input to the correlation value calculation unit. In this embodiment, the number of pixels and color filter of RAW image 2 are the same as those of RAW image 1, but RAW image 2 is assumed to have more noise than RAW image 1.

[0056] Figure 11 is a block diagram showing an example of the configuration of the correlation value calculation unit in Example 4. The correlation value calculation unit 401 includes a pixel ratio calculation unit 1100, a reliability calculation unit 1101, and a pixel ratio correction unit 1102. The pixel ratio calculation unit 1100 determines the reference pixel of the RAW image 2 input from the alignment processing unit 400, and calculates the pixel ratio of the correction target pixel of RAW image 1 to the non-defective same-color pixels surrounding the correction target pixel based on the pixel value of the reference pixel. If the correction target pixel is G22, the non-defective same-color pixels surrounding it are G00, G04, G40, and G44. The pixel ratio calculation unit 1100 determines g22, which is the pixel of RAW image 2 that is in the same position relative to G22 of RAW image 1, as the reference pixel. The pixel ratio calculation unit 1100 also determines g00, g04, g40, and g44, which are the pixels of RAW image 2 that are in the same position relative to G00, G04, G40, and G44 of RAW image 1, as the reference pixels.

[0057] The pixel ratio (RATIO) is calculated, for example, by equation (5). RATIO=g22 / AVE1 AVE1=(g00+g04+g40+g44) / 4...(Formula 5) The pixel ratio RATIO calculated by the pixel ratio calculation unit 1100 is input to the pixel ratio correction unit 1102.

[0058] The reliability calculation unit 1101 calculates the reliability of the pixel ratio RATIO based on noise according to brightness for the RAW image 2 input from the alignment processing unit 400. The pixel ratio RATIO is used to calculate the correlation value between G22 and surrounding non-defective pixels of the same color. Therefore, the reliability of the pixel ratio RATIO indicates the reliability of the pixel value correction. The calculated reliability is input to the pixel ratio correction unit 1102.

[0059] Figure 12 shows the relationship between the brightness value of RAW image 2 and the confidence level α. In the graph shown in Figure 12, the vertical axis represents the confidence level α, and the horizontal axis represents the luminance value. The relationship between the confidence level α and the luminance value shown in this graph is pre-stored in a predetermined memory unit as, for example, a conversion table for calculating the confidence level α from the luminance value.

[0060] The reliability calculation unit 1101 outputs a reliability of 0.0 as α if the luminance value is less than threshold Th1, indicating a dark region, and RAW image 2 has a lot of noise relative to the signal value, making the calculated pixel ratio RATIO unreliable. Furthermore, the reliability calculation unit 1101 outputs a reliability of 1.0 as α if the luminance value is greater than threshold Th2, indicating a bright region, and RAW image 2 has little noise relative to the signal value, making the calculated pixel ratio RATIO reliable. Threshold Th2 is pre-set to a value greater than threshold Th1.

[0061] As shown in Figure 12, when the brightness value of the RAW image 2 is greater than or equal to the threshold Th1 and less than or equal to the threshold Th2, the confidence level α changes linearly according to the brightness value. Therefore, when the brightness value is greater than or equal to the threshold Th1 and less than or equal to the threshold Th2, the confidence level calculation unit 1101 calculates a confidence level α between 0.0 and 1.0 corresponding to the magnitude of the brightness value. The confidence level α calculated by the confidence level calculation unit 1101 is input to the pixel ratio correction unit 1102.

[0062] Returning to the explanation of Figure 11, the pixel ratio correction unit 1102 calculates a correlation value based on the pixel ratio RATIO input from the pixel ratio calculation unit 1100 and the confidence level α input from the confidence level calculation unit 1101. The correlation coefficient COR can be calculated, for example, by formula (6). COR=α×RATIO+(1.0-α)×1.0 (Equation 6)

[0063] The pixel ratio correction unit 1102 corrects the pixel ratio RATIO using the confidence level α, as shown in equation (6). If the RAW image 2 has a lot of noise relative to the signal value, the pixel ratio RATIO is unreliable, and α is 0.0, the correlation value COR becomes 1.0. As a result, the correction result of G22 according to the aforementioned equation (2) becomes the average value of the nearest same-color pixels G00, G04, G40, and G44 in the periphery, making it possible to reduce the effect of noise. In other words, if the image output from the imaging unit B106 has more noise than that from the imaging unit A102, the correlation value calculation unit 401 can reduce the effect of noise by lowering the value of the calculated confidence level α.

[0064] Furthermore, if RAW image 2 has low noise relative to the signal value and the pixel ratio (RATIO) is reliable, it is possible to obtain correction results based on the correlation with surrounding pixels, similar to Example 1.

[0065] It should be noted that Examples 1 to 4 merely illustrate specific examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by these examples. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. For example, the optical path may be divided from the same optical system using a half-mirror or the like, so that the same subject is imaged on both the imaging unit A102 and the imaging unit B106. By imaged on both the imaging unit A102 and the imaging unit B106 with the same subject image, the alignment processing performed by the alignment processing unit 400 can be omitted.

[0066] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0067] 100 System Control Unit 102 Imaging Unit A 103 Defective Pixel Correction Unit A 106 Imaging Unit B 107 Defective Pixel Correction Unit B

Claims

1. A first imaging means and a second imaging means having different structures, A determination means that determines, based on the image output from the first imaging means, to be used to correct pixels that correspond to a group of adjacent pixels whose output does not satisfy predetermined conditions, and A first calculation means calculates a correlation value between the pixel to be corrected and the pixels surrounding the pixel to be corrected, based on the image output by the second imaging means and the difference in characteristics between the first imaging means and the second imaging means. The system includes a second calculation means for calculating a correction value for the pixel value of the pixel to be corrected based on the correlation value calculated above, The first calculation means calculates the correlation value based on the pixel values ​​of the reference pixels in the image output of the second imaging means, which are located at the same relative positions as the pixel to be corrected in the image output of the first imaging means and the surrounding pixels. An imaging device characterized by the following features.

2. The first calculation means determines the reference pixel based on the difference in the number of pixels between the first imaging means and the second imaging means. The imaging apparatus according to feature 1.

3. The first calculation means determines the reference pixel based on the difference in sampling of the color filters of the first imaging means and the second imaging means. The imaging apparatus according to claim 1 or 2.

4. The second calculation means calculates the correlation value based on the difference in noise characteristics between the first imaging means and the second imaging means. The imaging apparatus according to any one of claims 1 to 3.

5. The second calculation means calculates the correlation value based on the reliability of the pixel value correction, which is determined according to the brightness value of the image output by the second imaging means. The imaging apparatus according to feature 4.

6. The second calculation means lowers the reliability value if the output image has more noise when using the second imaging means than when using the first imaging means. The imaging apparatus according to feature 5.

7. The system includes alignment means for aligning the image output by the first imaging means with the image output by the second imaging means, The second calculation means calculates the correlation value based on the image output from the second imaging means that has been aligned. The imaging apparatus according to any one of claims 1 to 6.

8. The first imaging means and the second imaging means perform photoelectric conversion using different principles. The imaging apparatus according to any one of claims 1 to 7.

9. The first imaging means is a SPAD (Single Photon Avalanche Diode) sensor. The imaging apparatus according to any one of claims 1 to 8.

10. The optical system has an optical system that divides the optical path to form the same subject image on both the first imaging means and the second imaging means. The imaging apparatus according to any one of claims 1 to 9.

11. A control method for an imaging device having a first imaging means and a second imaging means having different structures, A determination step in which, based on the image output from the first imaging means, pixels that fall under a group of adjacent pixels whose output does not satisfy predetermined conditions are determined to be subject to correction, A first calculation step of calculating a correlation value between the pixel to be corrected and the pixels surrounding the pixel to be corrected, based on the image output by the second imaging means and the difference in characteristics between the first imaging means and the second imaging means. The process includes a second calculation step of calculating a correction value for the pixel value of the pixel to be corrected based on the correlation value calculated above, The first calculation step calculates the correlation value based on the pixel values ​​of the reference pixels in the image output of the second imaging means, which are located at the same relative positions as the pixels to be corrected in the image output of the first imaging means and the surrounding pixels. A control method characterized by the following:

12. A program characterized by causing a computer to execute each step of the control method described in claim 11.

Images