Image processing apparatus, image processing method, and image processing program

Abstract

An image processing device according to the present disclosure comprises: a defect candidate pixel detection unit that detects a defect candidate pixel for each captured image by performing defect candidate pixel detection processing on a plurality of captured images captured in a state in which a positional relationship between an imaging range and an image sensor having a plurality of pixels is different; and an interpolation target defect pixel determination unit that determines, as an interpolation target defect pixel, a pixel detected as the defect candidate pixel the number of times equal to or greater than a threshold value by the defect candidate pixel detection unit.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Techniques for interpolating data of pixel positions lacking in each of a plurality of frame images captured using a shift shooting mode, and techniques for correcting defective pixels are known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the prior art, before synthesizing a plurality of images, a process of interpolating defective pixels in each image is performed.

[0005] However, in the prior art, in detection-type defect correction in which defective pixels are detected and pixels determined to be defective pixels are corrected, the processing for defective pixels performed before synthesizing a plurality of images is not always appropriate. Specifically, it is possible that the determination differs in each of a plurality of images, such that a certain pixel is detected as a defective pixel or not detected. In such a case, the image quality of an image (synthesized image) generated by synthesis may be adversely affected. Therefore, it is desired to appropriately determine defective pixels.

[0006] Therefore, the present disclosure proposes an image processing apparatus, an image processing method, and an image processing program capable of preventing deterioration of the image quality of an image obtained in a pixel shift high image quality imaging mode by appropriately determining defective pixels.

Means for Solving the Problems

[0007] In order to solve the above problems, an image processing apparatus according to an aspect of the present disclosure performs a defect candidate pixel detection process on each of a plurality of captured images captured in a state where the positional relationship between the imaging range and an image sensor having a plurality of pixels is different from each other, thereby detecting defect candidate pixels for each of the plurality of captured images, and includes a defect candidate pixel determination unit that determines, by the defect candidate pixel detection unit, a pixel detected as the defect candidate pixel and an interpolation target defect pixel a predetermined number of times or more as an interpolation target defect pixel.

Brief Description of Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25A

Figure 25B

Figure 25C

Figure 26A

Figure 26B

Figure 26C

Figure 27

Figure 28

Figure 29

Figure 30A

Figure 30B

Figure 31

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the image processing apparatus, image processing method, and image processing program according to the present application are not limited by this embodiment. Also, in each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] The present disclosure will be described according to the following item order. 1. Embodiment 1-1. Overview 1-1-1. Pixel Shift High-Quality Imaging Mode 1-1-2. Defect Correction 1-1-3. Example of Procedure for Pixel Shift High-Quality Imaging Mode 1-1-4. Comparative Technology and Its Problems 1-2. Overview of Image Processing According to Embodiment of the Present Disclosure 1-2-1. Processing Overview 1-2-2. Specific Example of Processing 1-2-3. Other Variations 1-3. Configuration of Equipment Applicable as Image Processing Apparatus 1-3-1. Configuration of Imaging Apparatus 1-4. Configuration of Image Processing Apparatus 1-5. Other Configuration of Imaging Apparatus 1-6. Processing Procedure by Image Processing System 1-7. Processing Example by Image Processing System 1-7-1. First Processing Example 1-7-2. Second Processing Example 1-7-3. Third Processing Example 1-7-4. Fourth Processing Example 1-7-5. Fifth Processing Example 1-7-6. Sixth Processing Example 1-7-7. Seventh Processing Example 2. Other Embodiments 2-1. Others 3. Effects of the Present Disclosure 4. Hardware Configuration

[0011] [1. Embodiment] [1-1. Overview] Prior to the description of the embodiments of the present disclosure, first, the overview of the present disclosure will be described.

[0012] [1-1-1. Pixel Shift High-Definition Imaging Mode] There is known a pixel shift high-definition imaging method in which imaging is performed a plurality of times while physically shifting an image sensor, a lens, etc. by a minute distance, and a plurality of captured images are combined to generate a high-quality image.

[0013] [1-1-2. Defect Correction] For defect pixel correction, there are a method of storing in advance the existence position (defect pixel address) of a defect pixel and performing defect correction on the pixel at the stored defect pixel position (hereinafter also referred to as "address-type defect correction"), and a method of performing defect correction on a pixel when it is detected as a defect pixel (hereinafter also referred to as "detection-type defect correction"). Detection-type defect correction estimates whether a pixel (also referred to as a "target pixel") to be processed and pixels in its vicinity are defect pixels from the values of the pixel and its neighboring pixels, and corrects the pixel when it is determined (detected) to be a defect pixel. Address-type defect correction and detection-type defect correction are often used in combination.

[0014] In detection-type defect correction, pixels that receive light from shiny areas that are likely to occur on reflective surfaces such as metal or from stars in the night sky may be erroneously determined as defects. In the case of capturing a single normal image (frame), the side effects due to misdetection of defective pixels are a slight decrease in contrast or a slight tint. However, in the pixel shift high-quality imaging described above, if defect correction is performed or not performed on the same subject portion in a plurality of captured images (multiple frames), the image quality may deteriorate as a result of combining those images.

[0015] [1-1-3. Example procedure of pixel shift high-quality imaging mode] Here, an example procedure of the pixel shift high-quality imaging mode to which the technology of the present disclosure is applied will be described. In the imaging method of the example procedure, the image sensor is provided with a mechanism that moves vertically and horizontally within the light-receiving plane in units of, for example, 1 pixel or 0.5 pixel. As the pixel shift high-quality imaging as described above, for example, the following first to third example procedures can be considered.

[0016] The first example procedure is a method of moving in units of 1 pixel. It moves "0" and "1" pixels horizontally and "0" and "1" pixels vertically, and performs a total of 2×2 = 4 captures, and synthesizes the 4 captured images (RAW images). A RAW image is image data that records the output of the image sensor as it is. For example, it is an image in the form of an array of color filters output from the image sensor.

[0017] In the second example procedure, it moves in units of 0.5 pixel. It moves "0", "0.5", "1.0", and "1.5" pixels horizontally and "0", "0.5", "1.0", and "1.5" pixels vertically, that is, performs 4×4 = 16 captures, and synthesizes the 16 captured images (RAW images).

[0018] In the third procedure example, imaging is performed eight times. Hereinafter, two types will be described as the third procedure examples (a) and (b). In the third procedure example (a), imaging is performed with movements of "0", "0.5", "1.0", and "1.5" pixels in the horizontal direction and movements of "0" and "0.5" pixels in the vertical direction, a total of 4×2 = 8 times (alternatively, it may be two points in the horizontal direction and four points in the vertical direction), and the eight captured images (RAW images) are combined. In the third procedure example (b), first, similar to the first procedure example, imaging is performed with movements of "0" and "1" pixels in the horizontal direction and movements of "0" and "1" pixels in the vertical direction, a total of 2×2 = 4 times. Further, each of the four imaging operations is shifted diagonally by "0.5" pixels four times, for a total of eight imaging operations, and the eight captured images (RAW images) are combined.

[0019] Here, when the image sensor has a Bayer array as shown in FIG. 16, the final number of pixels of the synthesized image obtained in the first procedure example is the same as the image obtained by a single imaging operation without performing the synthesis process. However, improvement in false colors, moire, and jaggy, as well as improvement in the resolution of color signals and diagonal resolution, can be expected.

[0020] FIG. 16 is a diagram showing an example of a Bayer array. The Bayer array BA shown in FIG. 16 is an array of color filters (Bayer array) used in a so-called image sensor. Four pixels are grouped together, and color filters of one red pixel, two green pixels, and one blue pixel are assigned and arranged regularly. The Bayer array BA consists of a filter RF that is a color filter that transmits light in the red wavelength band, a filter GF that is a color filter that transmits light in the green wavelength band, and a filter BF that is a color filter that transmits light in the blue wavelength band. In FIG. 16, those with R described in the rectangular frame correspond to the filter RF, those with G described in the rectangular frame correspond to the filter GF, and those with B described in the rectangular frame correspond to the filter BF.

[0021] In the second procedure example, an image with a pixel count 2×2 = 4 times that of the image obtained by a single imaging operation is obtained, the resolution is improved by approximately two times, and moire and jaggy of color luminance are reduced.

[0022] In the third procedure example (a), the number of green pixels becomes four times the original, while the increase in the number of red and blue pixels only doubles. Therefore, although the color resolution of the third procedure example (a) is inferior to that of the second procedure example, image quality comparable to that of the second procedure example can be expected in other aspects. In the third procedure example (b), the resolution in the diagonal direction is inferior to that of the second procedure example for both luminance and color, but a resolution comparable to that of the second procedure example can be expected in the horizontal and vertical directions.

[0023] In this specification, the first procedure example is referred to as the pixel shift image quality improvement imaging mode, and the second and third procedure examples are referred to as the pixel shift super-resolution imaging modes. Collectively, both are called the pixel shift high image quality imaging mode. The synthesis process in the pixel shift image quality improvement imaging mode of the first procedure example is called the image quality improvement synthesis process, and the synthesis process in the pixel shift super-resolution imaging modes of the second and third procedure examples is called the super-resolution synthesis process. Also, when collectively referring to the image quality improvement synthesis process and the super-resolution synthesis process, it is called the high image quality synthesis process. Note that in addition to the examples shown below, other methods are conceivable for the pixel shift high image quality imaging mode, and the scope of application of the present invention is not limited to the procedure examples shown below.

[0024] The third procedure example will be described with reference to FIGS. 17 to 24. FIGS. 17 to 22 are diagrams for explaining the process of the third procedure example (a). FIGS. 23 and 24 are diagrams for explaining the process of the third procedure example (b).

[0025] First, the green pixels shown in FIGS. 17 to 19 will be described. The pixel group GD1 in FIG. 17 is obtained by extracting only the pixels (green pixels G1) corresponding to the filter GF in the Bayer array BA in FIG. 16. The green pixel G2, which is shown with a different hatching from the green pixel G1 in the pixel group GD2 in FIG. 17, represents the green pixel when the image sensor is shifted one pixel to the right from the imaging corresponding to the pixel group GD1. The green pixel G1 and the green pixel G2 complete the pixel group GD2 in which all the positions of the original pixels except the endpoints are aligned.

[0026] Hereinafter, for convenience of explanation, as shown in FIG. 18, the green pixels G1 and G2 in the pixel group GD2 shall be represented as small as the green pixels G1 and G2 in the pixel group GD3. The pixel group GD4 shown in FIG. 19 shows the case where the third and fourth imaging operations are shifted 0.5 pixels and 1.5 pixels to the right from the initial state. The green pixel G3 in the image group GD4 corresponds to the third imaging operation and is shifted 0.5 pixels to the right from the green pixel G1 of the first imaging operation, and the green pixel G4 corresponds to the fourth imaging operation and is shifted 1.5 pixels to the right from the green pixel G1 of the first imaging operation. The pixel group GD5 shows the case where each of the first to fourth imaging operations is shifted 0.5 pixels downward to perform the fifth to eighth imaging operations. As a result, except for the endpoints, the pixel group GD5 with double the density of the original pixels is completed.

[0027] Next, the red pixels shown in FIGS. 20 to 22 will be described. The pixel group RD1 in FIG. 20 extracts only the pixels (red pixel R1) corresponding to the filter RF in the Bayer array BA in FIG. 16. The red pixel R2 shown with a different hatching from the red pixel R1 in the pixel group RD2 in FIG. 20 indicates the red pixel when the image sensor is shifted 1 pixel to the right for imaging from the imaging corresponding to the pixel group RD.

[0028] Hereinafter, for convenience of explanation, as shown in FIG. 21, the red pixels R1 and R2 in the pixel group RD2 shall be represented as small as the red pixels R1 and R2 in the pixel group RD3. The image group RD4, similar to the green image group GD5, shows the case where the third and fourth imaging operations are shifted 0.5 pixels and 1.5 pixels to the right from the initial state for imaging, and each of the first to fourth imaging operations is shifted 0.5 pixels downward to perform the fifth to eighth imaging operations. The hatched rectangles (red pixels R1 to R4, etc.) in the image group RD4 in FIG. 22 correspond to the red pixels. The non-hatched rectangles in the image group RD4 indicate the positions where the red pixels are not imaged, and the red pixels at those positions are estimated by an algorithm similar to demosaicing for a normal Bayer image, for example.

[0029] Regarding the cyan pixels, through the same processing as in FIGS. 20 to 22, the positions of the rectangles without hatching in the image group RD4 are imaged. And the rectangles with hatching in the image group RD4 indicate the positions where cyan pixels are not imaged, and the cyan pixels at those positions are estimated by the same algorithm as the red pixels. Thus, in the third procedure example (a), it is necessary to perform a process of estimating missing information, that is, a process similar to demosaicing for a normal Bayer image, for red and cyan. In the third procedure example (a), the rearrangement of the images and the process similar to demosaicing for red and cyan become a synthesis process (super-resolution synthesis process).

[0030] Also, the processing of the third procedure example (b) is as shown in FIGS. 23 and 24. FIG. 23 shows a case where the pixel group GD11 is imaged four times by shifting vertically by 1 pixel, horizontally (for example, to the right) by 1 pixel, and diagonally (for example, diagonally down to the right) by 1 pixel from the initial state, and each of the four images is shifted diagonally (for example, diagonally down to the right) by 0.5 pixel four times. All the green pixels G in the pixel group GD11 are imaged twice.

[0031] The pixel group RD11 shown in FIG. 24 shows the arrangement of red pixels obtained as a result of the same imaging procedure as that performed to obtain the green pixel group GD11 shown in FIG. 23. The red pixels R of the pixel group RD11 are imaged at the same positions as the green pixels G of the pixel group GD11. The same applies to the cyan pixels as in the case of the red pixels shown in FIG. 24. In the case of the method shown in FIGS. 23 and 24, all RGB values can be obtained at the checkered positions with twice the number of pixels.

[0032] Also, in the case of the method shown in FIGS. 23 and 24, pixel interpolation processing is required to make the checkered RGB each have no missing pixels. This pixel interpolation processing is similar to the processing of interpolating green for demosaicing of a normal Bayer image.

[0033] [1-1-4. Comparative Technology and Its Problems] In this specification, for a plurality of images obtained in the pixel-shifting high-image-quality imaging mode described above, those to which detection-type defect correction is independently applied to each are used as a comparative technique, and the problems thereof will be described here. Not limited to the case of the pixel-shifting high-image-quality imaging mode, in detection-type defect correction, there is a possibility of erroneously determining as a defect a pixel that has received light from a subject with a small area and a large luminance difference from its vicinity, such as the sparkle that easily occurs on a reflective surface such as metal or the light of stars in the night sky. These subjects can give a value that stands out from the values of neighboring pixels, and thus have the same nature as a defect, which is why it is difficult to distinguish them.

[0034] Also, in the case of imaging a normal single image, the side effects due to false detection appear as a phenomenon that the brightness of the bright spot weakens and the contrast decreases in the case of a subject with a fine pattern, and in many cases, it is within the subjective tolerance range. However, in the pixel-shifting high-image-quality imaging mode, there are cases where even a subject that gives a value that stands out from the values of the corresponding pixels of each image before synthesis that constitutes them does not necessarily have a value that stands out as the value of the corresponding pixel in the synthesized image.

[0035] The above points will be described with reference to FIGS. 25A, 25B, and 25C. FIGS. 25A, 25B, and 25C are diagrams showing an example of the relationship between pixels and detection-type defect correction. Hereinafter, when FIGS. 25A, 25B, and 25C are collectively referred to, they may be described as "FIG. 25". Hereinafter, a conceptual explanation will be given using FIG. 25. For example, FIG. 25 is a diagram for explaining side effects caused by detection-type defect correction specific to the pixel-shifting high-image-quality imaging mode, taking the pixel-shifting super-resolution imaging mode as an example.

[0036] In FIG. 25, it is assumed that the image sensor is a black-and-white line sensor. After the first imaging is performed with the image sensor, as an example, a super-resolution image is obtained by synthesizing two images obtained by shifting the image sensor 0.5 pixels to the right and performing the second imaging. Lines LS1 in FIG. 25A and LS2 in FIG. 25B are graphs showing the lateral brightness of a subject whose center in the lateral direction is the brightest (high luminance). Note that the vertical axis is normalized by the value of the maximum luminance. Also, in the example of FIG. 25, for simplicity of explanation, the aperture effect of the image sensor is not considered.

[0037] In the imaging shown in FIG. 25A (first imaging), pixel values PT11, PT12, PT13, PT14, and PT15 indicated by filled circles “〇” are obtained as the outputs of five pixels of the image sensor. The imaging shown in FIG. 25B (second imaging) results in pixel values PT21, PT22, PT23, PT24, and PT25 indicated by squares “□” because the sampling points are shifted 0.5 pixels to the right from the time of imaging in FIG. 25A.

[0038] Then, detection-type defect correction is performed on the pixel values PT11, PT12, PT13, PT14, and PT15 in FIG. 25A and the pixel values PT21, PT22, PT23, PT24, and PT25 in FIG. 25B. Here, as an example of the algorithm of the detection-type defect correction, the procedures of the following detection step and interpolation step are assumed.

[0039] First, in the detection step, when the polarities (plus, minus) of the differences (two differences) between the pixel value of the pixel of interest and the pixel values of each of the two adjacent pixels are the same and the absolute values of the differences (two differences) both exceed a threshold value (here, 0.8), the pixel of interest is detected as a defective pixel.

[0040] Then, in the interpolation step, when the pixel of interest is detected as a defective pixel in the detection step, the value of the pixel of interest (defective pixel) is replaced (interpolated) with the average of the pixel values of the two adjacent pixels. Hereinafter, when referring to the interpolation step, it may be described as the “interpolation step of the comparative technique”.

[0041] By performing the above processing, the pixel value PT13 in FIG. 25A is interpolated by neighboring pixels. From the observation of FIG. 25A, the pixel at the position of the horizontal axis 0 is detected as a defective pixel and replaced (interpolated) with the average of the pixel values of its two adjacent pixels. As a result, the pixel value PT13 is corrected to the interpolation value CP13 indicated by the cross "×". Also, in FIG. 25B, since there is no pixel detected as a defective pixel, the pixel values in FIG. 25B are not corrected, and the pixel values in FIG. 25B are used for subsequent processing.

[0042] FIG. 25C shows the result of synthesizing the captured images shown in FIGS. 25A and 25B. FIG. 25C is obtained by synthesizing the two captured images of FIGS. 25A and 25B to obtain a high-resolution super-resolution image, but it has a waveform with a hole in the central part of the waveform of the original subject. Specifically, since the interpolation value CP13 corresponding to the pixel at the position of the horizontal axis 0 is smaller than the pixel value PT22 corresponding to the pixel at the position of the horizontal axis -0.5 and the pixel value PT23 corresponding to the pixel at the position of the horizontal axis 0.5, the position of the horizontal axis 0 has a concave waveform.

[0043] Therefore, when the super-resolution image in FIG. 25C is developed, an image with deteriorated image quality is obtained. For example, if the captured image in FIG. 25A is developed as it is, it only results in a simple contrast reduction, but in the super-resolution image obtained by super-resolution synthesis, the image quality deteriorates from the above-mentioned perspective. Thus, in a pixel shift high-image-quality imaging mode such as the pixel shift super-resolution imaging mode, side effects may occur due to detection-type defect correction.

[0044] [1-2. Overview of Image Processing According to Embodiment of the Present Disclosure] Therefore, in the present disclosure, a method for appropriately determining defective pixels is proposed by using the results of defect detection for each pixel of each image to determine defective pixels.

[0045] [1-2-1. Processing Overview] First, referring to FIG. 25, while explaining the general outline of the processing of the present disclosure, the principle of the present technology will be explained. In FIG. 25A, the brightest part of the subject coincides with the pixel at the position of the horizontal axis coordinate 0 (also referred to as "horizontal axis 0"). However, in FIG. 25B where this is shifted 0.5 pixels to the right and the second imaging is performed, the pixel that was at the position of the horizontal axis coordinate 0 moves to the position of 0.5 and deviates from the brightest part of the subject, so that it no longer has the value of the brightest part. In other words, the difference (absolute value) from the surrounding pixels becomes a value lower than the threshold for defect detection.

[0046] Here, the processing in the case where there is a defective pixel will be explained with reference to FIGS. 26A, 26B, and 26C. FIGS. 26A, 26B, and 26C are diagrams showing another example of the relationship between pixels and detection-type defect correction. Hereinafter, when FIGS. 26A, 26B, and 26C are collectively referred to, they may be described as "FIG. 26". Hereinafter, a conceptual explanation will be given using FIG. 26. Note that, in FIG. 26, explanations for the same points as in FIG. 25 will be omitted as appropriate.

[0047] In the imaging shown in FIG. 26A (first imaging), pixel values PT31, PT32, PT33, PT34, and PT35 indicated by round "〇" are obtained as the outputs of five pixels of the image sensor. Here, it is assumed that the pixel at the position of the horizontal axis coordinate 0 is defective. The imaging shown in FIG. 26B (second imaging) results in pixel values PT41, PT42, PT43, PT44, and PT45 indicated by squares "□" because the sampling point is shifted 0.5 pixels to the right from the time of imaging in FIG. 26A.

[0048] Then, similar to FIG. 25, detection-type defect correction is performed on the pixel values PT31, PT32, PT33, PT34, and PT35 in FIG. 26A and the pixel values PT41, PT42, PT43, PT44, and PT45 in FIG. 26B by the interpolation step of the comparative technique.

[0049] The pixel value PT33 in Fig. 26A will be interpolated by the neighboring pixels. From the observation of Fig. 26A, the pixel at the position of the horizontal axis 0 is detected as a defective pixel and replaced (interpolated) with the average of the pixel values of its two adjacent pixels. As a result, the pixel value PT33 is corrected to the interpolated value CP33 indicated by the cross "×". Also, the pixel value PT43 in Fig. 26B will be interpolated by the neighboring pixels. From the observation of Fig. 26B, the pixel at the position of the horizontal axis 0 is detected as a defective pixel and replaced (interpolated) with the average of the pixel values of its two adjacent pixels. As a result, the pixel value PT43 is corrected to the interpolated value CP43 indicated by the cross "×". Fig. 26C shows the result of synthesizing the captured images shown in Fig. 26A and Fig. 26B. Fig. 26C is obtained by synthesizing the two captured images of Fig. 26A and Fig. 26B to obtain a high-resolution super-resolution image, and the waveform is such that the pixel value of the defective pixel is corrected. As shown in Fig. 26, when the pixel at the position of the horizontal axis 0 is a defective pixel, it will also be determined as a defective pixel when the image is captured for the second time with the pixels shifted to the right by 0.5 pixels. The present disclosure eliminates the side effects of the above-described detection-type defect correction by using the difference between these two cases to determine defective pixels.

[0050] [1-2-2. Specific Example of Processing] Hereinafter, with reference to Fig. 1, the principle of the process for determining defective pixels will be described. In Fig. 1, for simplicity of explanation, only 16 pixels P1 to P16 are exemplified.

[0051] In Fig. 1, the pixel shift high-image-quality imaging mode conceptually shows that the position of the subject SB1 moves within the plane (light-receiving surface) of the image sensor 121. Specifically, in Fig. 1, the movement of the position of the subject SB1 within the plane (light-receiving surface) of the image sensor 121 is schematically shown by the positional relationship between the subject SB1 and each captured image IM1 to IM4, and the dotted lines between the subject SB1 and each captured image IM1 to IM4.

[0052] In FIG. 1, first, an image processing apparatus TD (see FIG. 2) performs a defective candidate pixel detection process on each pixel of a captured image IM1 obtained by capturing an object SB1 in an imaging range where the position of the object SB1 in the plane (light-receiving surface) of the image sensor 121 is at a first position (step S1). As a simple example of the algorithm for the defective candidate pixel detection process, the same process as the above-described detection step may be used. That is, when the polarities (plus, minus) of the differences between the pixel to be the target of the defective candidate pixel detection process (the pixel of interest) and each of the pixels on both sides adjacent to the pixel of interest are the same, and the absolute values of the differences both exceed a threshold value (for example, 0.6, 0.75, etc.), the pixel of interest is detected as a defective candidate pixel. Here, the term "both sides adjacent" refers to at least one of the two adjacent pixels in the vertical (up and down) direction, the two adjacent pixels in the horizontal (left and right) direction, and the two adjacent pixels in the diagonal (upper right, lower right, upper left, lower left) direction. For example, all of them may be used. Note that any algorithm may be used for the defective candidate pixel detection process as long as it detects defective candidate pixels based on the comparison between the value of the pixel of interest and the values of the pixels in the vicinity of the pixel of interest. The pixels in the vicinity here are, for example, the pixels around the pixel of interest, and they may or may not be adjacent to the pixel of interest.

[0053] In FIG. 1, for the captured image IM1, the pixels corresponding to the pixels P6, P8, and P15 of the image sensor 121 are detected as defective candidate pixels. As a result, as shown in the defective candidate pixel detection map MP1-1, "1" is added to the count value of the address corresponding to each of the pixels P6, P8, and P15 of the image sensor 121, and the count values of the pixels P6, P8, and P15 become "1". Note that although the defective candidate pixel detection maps MP1-1 to MP1-4 are described according to the change in the count value, when they are described without distinction, the defective candidate pixel detection map MP1 is used.

[0054] Next, the image processing device TD performs a defective candidate pixel detection process on each pixel of the captured image IM2 obtained by moving the position of the subject SB1 within the plane (light-receiving surface) of the image sensor 121 from the first position to the second position and imaging with different imaging ranges (step S2). In FIG. 1, pixels corresponding to the pixels P6, P8, and P11 of the image sensor 121 are detected as defective candidate pixels. As a result, as shown in the defective candidate pixel detection map MP1-2, "1" is added to the addresses corresponding to the pixels P6, P8, and P11 of the image sensor 121, the count values of the pixels P6 and P8 become "2", and the count values of the pixels P11 and P15 become "1".

[0055] Next, the image processing device TD performs a defective candidate pixel detection process on each pixel of the captured image IM3 obtained by moving the position of the subject SB1 within the plane (light-receiving surface) of the image sensor 121 from the second position to the third position and imaging with different imaging ranges (step S3). In FIG. 1, pixels corresponding to the pixels P1 and P6 of the image sensor 121 are detected as defective candidate pixels. As a result, as shown in the defective candidate pixel detection map MP1-3, "1" is added to the addresses corresponding to the pixels P1 and P6 of the image sensor 121, the count value becomes "3", the count value of the pixel P8 becomes "2", and the count values of the pixels P1, P11, and P15 become "1".

[0056] Next, the image processing device TD performs a defective candidate pixel detection process on each pixel of the captured image IM4 obtained by moving the position of the subject SB1 within the plane (light-receiving surface) of the image sensor 121 from the third position to the fourth position and imaging with different imaging ranges (step S4). In FIG. 1, pixels corresponding to the pixels P3, P6, and P11 of the image sensor 121 are detected as defective candidate pixels. As a result, as shown in the defective candidate pixel detection map MP1-4, "1" is added to the addresses corresponding to the pixels P3, P6, and P11 of the image sensor 121, the count value of the pixel P6 becomes "4", the count values of the pixels P8 and P11 become "2", and the count values of the pixels P1, P3, and P15 become "1".

[0057] Then, the image processing device TD determines defective pixels using the defective pixel candidate detection map MP1-4 (step S5). In the example of FIG. 1, each count value in the defective pixel candidate detection map MP1-4 is compared with the threshold value "4", and a pixel whose count value in the defective pixel candidate detection map MP1-4 is "4" is determined as an interpolation target defective pixel. In the example of FIG. 1, as shown in the interpolation target defective pixel information DPI, a pixel P6 whose count value is "4" is determined as an interpolation target defective pixel.

[0058] Then, the image processing device TD interpolates each pixel in the captured images IM1 to IM4 corresponding to the pixel P6 determined to be an interpolation target defective pixel with neighboring pixels, and performs defective pixel interpolation processing using neighboring pixels on the captured image including the interpolation target defective pixel for the interpolation target defective pixel. For example, when the image sensor 121 is a color image sensor using a Bayer array color filter and the pixel P6 is a green pixel, the image processing device TD replaces the pixel value of the pixel P6 with the average of the pixel values of the nearest green pixels P1, P3, P9, and P11. For example, the image processing device TD uses the pixels P1, P3, P9, and P11 included in the same captured image IM1 as the pixel P6, which is an interpolation target defective pixel, as neighboring pixels for interpolation to interpolate the pixel P6 of the captured image IM1. The image processing device TD performs the same interpolation process for the pixel P6 on the captured images IM2 to IM4. Then, the image processing device TD generates a composite image by synthesizing the captured images IM1 to IM4 after the defective pixel interpolation process.

[0059] As described above, the number of times each pixel is detected as a defective pixel candidate is counted, and a pixel whose count reaches the threshold value is determined as an interpolation target defective pixel. Thereby, the interpolation target defective pixel can be appropriately determined.

[0060] Also, defective pixel interpolation processing is enabled for the pixel determined to be an interpolation target defective pixel. Thereby, appropriate detection-type defect correction can be performed with the pixel that is a real defect as the interpolation target defective pixel.

[0061] If the target pixel is truly a defective pixel, since it is detected as a defective pixel to be interpolated in a number of images equal to or greater than the threshold among the images used for synthesis, the count value matches the number of captured images used for synthesis. On the other hand, even if there is an image in which a non-defective pixel is misdetected as a defective pixel candidate due to a pattern, it is unlikely to be misdetected as a defective pixel candidate in all images. Therefore, by determining the pixels detected as defective pixel candidates in all the images used for synthesis as defective pixels to be interpolated and enabling only the defective pixel interpolation process for the defective pixels to be interpolated, it is possible to obtain good image quality without side effects due to misdetection.

[0062] [1-2-3. Other Variations] In the above description, the determination criterion (threshold) for defective pixels to be interpolated is set to all of the number of captured images (N images, 4 images in FIG. 1) to be synthesized, but it may be changed. For example, a threshold considering that a true defective pixel may not be determined as a defective pixel to be interpolated due to noise or the like may be used. That is, a value less than the number of captured images may be used as the threshold, and the pixels detected as defective pixel candidates more than the threshold times may be determined as defective pixels to be interpolated. For example, when the number of captured images is 16, a value of "14", which is less than 16, may be used as the threshold, and the pixels detected as defective pixel candidates more than the threshold times may be determined as defective pixels to be interpolated. When the total number of images is, for example, 16, the image processing device TD may determine defective pixels using, for example, 80% of that number, that is, 12.8 or 13 after rounding it. For example, when there is a lot of noise in the input image and the reliability of defect determination is low, the threshold can be slightly lowered to make it easier to determine as a defect, thereby enhancing the overall image quality improvement effect. Also, when the threshold is less than the number of captured images, for the captured images determined to include defective pixels to be interpolated among the plurality of captured images, the captured images obtained by interpolating the defective pixels to be interpolated are used as target images for synthesis, and for the images determined not to include defective pixels to be interpolated, the captured images without interpolation are used as target images for synthesis, and the synthesis process may be performed to generate a synthesized image.

[0063] The image sensor 121 may be either a black-and-white sensor (monochrome sensor) or a color image sensor. Hereinafter, when a color captured image is the target, that is, when the image sensor 121 is a color image sensor, the side effects of the detection type defect correction will be specifically described. For example, in a color image sensor using a Bayer array color filter, green pixels are densely arranged, and red and blue pixels are arranged more sparsely than green pixels. Therefore, in the detection type defect correction, defects are detected while referring to pixels farther away for red and blue than for green.

[0064] For this reason, the rate at which a high-brightness subject with a small area and a large luminance difference from the vicinity is misrecognized as a defect is higher for red and blue pixels than for green pixels. Therefore, in a case like Figure 25, it is easy for a situation to occur where detection type defect correction is performed on red and blue pixels near the highest luminance part, and detection type defect correction is not performed on green pixels. Thus, in the case of color, since the sample interval for red and blue is longer than that for green, miscorrection occurs with a higher probability than for green. As a result, for red and blue, defect pixel interpolation processing is performed with the pixel values of neighboring pixels and the values decrease, but the green value does not decrease. In some cases, bright spots, which are high-brightness parts, may turn green. As a result, the composite image is likely to be an image with a green mark near the center of the high-brightness part. Figures 28 and 29 are schematic diagrams showing actual examples thereof. Figure 28 is a diagram showing an example of the side effects of the detection type defect correction. Figure 29 is an enlarged view showing an example of the side effects of the detection type defect correction. Figure 28 is an example of a composite image in which side effects have occurred, and Figure 29 is an enlarged view of a part of the example of the composite image in which side effects have occurred.

[0065] The composite image example IMS shown in FIG. 28 shows an example of an image in which there is a green scratch in a metal object. FIG. 29 is a schematic diagram enlarging a portion of the region TAR in the image example IMS of FIG. 28, and a portion such as a green scratch SER occurs near the center of the bright portion. In FIG. 29, for convenience of color representation, the metal object is shown in gray, the region of the shiny and bright reflection surface among the metal objects is shown in white, and the portion that has turned green within the region of the reflection surface is shown in black. Thus, in the example of FIG. 29, the composite image is such that a green scratch is attached near the center of the portion that reflects brightly and shiny.

[0066] Here, an example of side effect simulation will be described with reference to FIGS. 30A and 30B. FIG. 30A is a diagram showing an example of simulation when detection type defect correction is not performed. FIG. 30B is a diagram showing an example of simulation when detection type defect correction is performed. Hereinafter, when FIGS. 30A and 30B are collectively referred to, "FIG. 29" may be described. Hereinafter, a conceptual explanation will be given using FIG. 29. FIG. 29 is a simulation result when detection type defect correction is applied to each imaging image before synthesis using the above-described comparative technique when imaging a subject with a very small area and much brighter than its surroundings in the pixel shift super-resolution imaging mode.

[0067] FIG. 30A shows a simulation result CS1 in which the defect correction of the detection type is turned off and a defect pixel portion DF1 with a size of 4×4 pixels appears in the upper left. Thus, FIG. 30A shows that the green pixel in the upper left of the screen is the defect pixel portion, and when the detection type defect correction is not performed, the defect pixel portion DF1 in the upper left of the screen remains as a defect at the same position in the composite image.

[0068] FIG. 30B shows a simulation result CS2 in the case where the detection type defect correction is turned on. In this case, the influence of the defective pixel portion DF1 disappears, but a green flaw (point DF2) occurs in the subject center AR1 portion. For example, FIG. 30B shows a case where detection type defect correction is performed on an image including a defective pixel portion DF1 of size 4×4 in the upper left, similar to FIG. 30A. As described above, in FIG. 29, the defective pixel portion DF1 is corrected by the detection type defect correction, but on the other hand, a green point DF2 appears in the brightest portion AR1.

[0069] In the present technology, the number of times each pixel of each captured image is detected as a defect is counted. For example, using an image sensor 121 (see FIG. 6) using a color filter of a Bayer array BA as shown in FIG. 16, in a pixel shift image quality improvement imaging mode which is one of the pixel shift high image quality imaging modes as shown in FIG. 27, a plurality of color images are captured. FIG. 27 is an explanatory diagram for explaining an example of the pixel shift image quality improvement imaging mode. Note that the Bayer array BA shown in FIG. 16 is merely an example, and color images may be captured using color filters of various arrays, not limited to the Bayer array.

[0070] Using the image sensor 121 to which the Bayer array BA shown in FIG. 16 is applied, a plurality of color images are captured in the above-described pixel shift high image quality imaging mode. FIG. 27 is a diagram conceptually showing the process of the pixel shift high image quality imaging mode using the image sensor 121 to which the Bayer array BA shown in FIG. 16 is applied. In FIG. 27, the image sensor 121 is shifted one pixel at a time along the vertical direction to capture a plurality of captured images. In this way, by aligning the range of the subject (imaging range) corresponding to one pixel with the pixel positions of red, green, and blue and performing imaging a plurality of times, a composite image is generated by directly combining a plurality of captured images without performing a demosaicing (color separation) process. For example, by sequentially shifting the image sensor 121 one pixel at a time and performing continuous shooting, a plurality of captured images IM51 to IM54 (corresponding to Shot1 to Shot4 in FIG. 27) are acquired.

[0071] The image processing device TD performs defect detection processing on each of a plurality of captured images. In the example of FIG. 27, the image processing device TD performs defect candidate pixel detection processing on each of four captured images, namely, the captured image IM51 of Shot1, the captured image IM52 of Shot2, the captured image IM53 of Shot3, and the captured image IM54 of Shot4. In the case of a color image, it is desirable to perform defect candidate pixel detection processing with reference to pixels of the same color.

[0072] For example, the image processing device TD (see FIG. 2) detects a pixel (target pixel) to be subjected to defect candidate pixel detection processing as a defect candidate pixel when the polarity (plus or minus) of the difference between the target pixel and a pixel of the same color in the vicinity is the same and the absolute value of the difference exceeds a threshold value (for example, 0.6 or 0.75, etc.). For example, the image processing device TD may use the four pixels above, below, left, and right of the target pixel as the neighboring pixels, or may use the eight pixels above, below, left, right, and in the diagonal directions of the target pixel as the neighboring pixels. Also, the image processing device TD may use the pixels within a 5×5 region centered on the target pixel or the pixels within a 7×7 region centered on the target pixel as the neighboring pixels. For example, the image processing device TD may detect the target pixel as a defect candidate pixel using the statistical quantity of the pixels within a 5×5 or 7×7 region centered on the target pixel. For example, for the target pixel, when the distances between the upper-right pixel and the lower-left pixel among the pixels of the same color as the target pixel are shorter than those of other pixels, the image processing device TD may perform defect candidate pixel detection processing on the target pixel using the upper-right pixel and the lower-left pixel as the neighboring pixels. The image processing device TD counts the number of times each pixel of each captured image is detected as a defect candidate pixel by performing defect candidate pixel detection processing on each pixel.

[0073] Then, a pixel for which the number of times detected as a defect candidate pixel reaches the number of sheets of a plurality of captured images (for example, N) is determined as an interpolation target defect pixel. Thereby, it is possible to appropriately determine interpolation target defect pixels even for color images.

[0074] Further, the image processing apparatus TD performs defect pixel interpolation processing using neighboring pixels only for the pixels determined as defect pixels to be interpolated. As a result, the image processing apparatus TD can perform a detection-type defect correction process that appropriately targets the defect pixels to be interpolated, which are the true defect pixels, for color images as well as for black-and-white images. The image processing apparatus TD generates images of each color in which defect pixel interpolation processing using neighboring pixels is performed only for the pixels determined as defect pixels to be interpolated, and generates a composite image by a high-quality composite process that composites the images corresponding to each color in which defect pixel interpolation processing is performed only for the pixels determined as defect pixels to be interpolated. In the example of FIG. 27, the image processing apparatus TD generates four images, namely, one red image RI, two green images GI1 and GI2, and one blue image BI, by defect pixel interpolation processing, and generates a composite image by a high-quality composite process that composites the red image RI, the green images GI1 and GI2, and the blue image BI. For example, the image processing apparatus TD rearranges the four images to form four planes of the green images GI1 and GI2, the red image RI, and the blue image BI, averages the two green images GI1 and GI2 to generate a green image GI, and performs a high-quality composite process to form three planes of the red image RI, the green image GI, and the image BI. In the example of the second procedure of the pixel shift high-quality imaging mode that performs imaging 16 times, the green images GI1 and GI2, the red image RI, and the blue image BI, whose number of pixels has become four times as a result of the rearrangement, are obtained, and similarly, the two green images GI1 and GI2 are averaged to generate a green image GI, and a high-quality composite process is performed to form three planes of the red image RI, the green image GI, and the image BI.

[0075] [1-3. Configuration of Devices Applicable as Image Processing Apparatus] The image processing apparatus that performs the above-described processing can be realized in various devices. First, devices to which the technology of the present disclosure can be applied will be described with reference to FIG. 2. FIG. 2 is an explanatory diagram of the devices used in the embodiment of the present disclosure.

[0076] FIG. 2A shows an example of an image source VS and an image processing device TD that acquires an image file MF from the image source VS. The image processing device TD is a device that performs image processing on the image data acquired from the image source VS. Here, the image processing includes at least one of a defective candidate pixel detection process for detecting defective candidate pixels for each captured image, an interpolation target defective pixel determination process for determining interpolation target defective pixels from the defective candidate pixels, and a defective pixel interpolation process for interpolating the interpolation target defective pixels. Note that the “interpolation target defective pixel determination process” may be described as a “false defect detection determination process”.

[0077] As the image source VS, an imaging device 1, a server 4, a recording medium 5, etc. are assumed. As the image processing device TD, a mobile terminal 2 such as a smartphone, a personal computer 3, etc. are assumed. In addition, as the image processing device TD, various devices such as a dedicated image editing device, a cloud server, a television device, and a video recording / playback device are assumed as the image processing device TD.

[0078] The imaging device 1 as the image source VS is a digital camera or the like, and transfers the image file MF obtained by imaging to a mobile terminal 2, a personal computer 3, etc. via wired communication or wireless communication. The server 4 may be any of a local server, a network server, a cloud server, etc., and refers to a device that can provide the image file MF imaged by the imaging device 1. The server 4 transfers the image file MF to a mobile terminal 2, a personal computer 3, etc. via some transmission path.

[0079] The recording medium 5 may be any of a solid memory such as a memory card, a disk-shaped recording medium such as an optical disk, a tape-shaped recording medium such as a magnetic tape, etc., but is a removable recording medium on which the image file MF imaged by the imaging device 1 is recorded. The image file MF read from the recording medium 5 is read by a mobile terminal 2, a personal computer 3, etc.

[0080] Portable terminals 2, personal computers 3, etc. as the image processing device TD can perform image processing on the image file MF acquired from the above image source VS.

[0081] Note that the portable terminal 2 or the personal computer 3 may also serve as an image source VS for other portable terminals 2 or personal computers 3 that function as the image processing device TD.

[0082] FIG. 2B shows the imaging device 1 or the portable terminal 2 as a device that can function as both the image source VS and the image processing device TD. For example, a microcomputer or the like inside the imaging device 1 performs image processing. For example, the imaging device 1 can output an image in which defective pixels are interpolated by performing defective pixel candidate detection processing, defective pixel determination processing for interpolation target, and defective pixel interpolation processing on the image file MF generated by imaging.

[0083] The same applies to the portable terminal 2. Since it can be an image source VS by having an imaging function, it can output an image in which defective pixels are interpolated by performing defective pixel candidate detection processing, defective pixel determination processing for interpolation target, and defective pixel interpolation processing on the image file MF generated by imaging. Note that not limited to the imaging device 1 and the portable terminal 2, various other devices can be considered as devices that can be both an image source and an image processing device.

[0084] As described above, the devices functioning as the image processing device TD and the image source VS in the embodiment are diverse. In FIG. 3 below, the imaging device 1, which is the image source VS and image processing device TD in FIG. 2B, will be described as an example.

[0085] [1-3-1. Configuration of Imaging Device] The configuration of the imaging device 1, which is the image source VS and image processing device TD in FIG. 2B, will be described with reference to FIG. 3. FIG. 3 is a block diagram of the imaging device according to the embodiment of the present disclosure.

[0086] The imaging device 1 in FIG. 3 includes a lens system 11, an imaging element unit 12, a detection type defect correction processing unit 13, a synthesis processing unit 14, a development processing unit 15, a recording control unit 16, a display unit 17, an output unit 18, an operation unit 19, a memory unit 20, a control unit 21, a driver unit 22, and a sensor unit 23.

[0087] The lens system 11 includes lenses such as a cover lens, a zoom lens, and a focus lens (for example, the lens 111 in FIG. 6) and a diaphragm mechanism. The light (incident light) from the subject is guided by this lens system 11 and condensed on the imaging element unit 12.

[0088] The imaging element unit 12 is configured to include an image sensor 121 (imaging element) such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type. In this imaging element unit 12, for the electrical signal obtained by photoelectrically converting the light received by the image sensor 121, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. are executed, and further A / D (Analog / Digital) conversion processing is performed. Then, the imaging signal as digital data is output to the subsequent detection type defect correction processing unit 13 and the control unit 21.

[0089] The image sensor 121 may be configured such that each of a plurality of pixels detects the intensity of light and captures a monochrome image. Also, as shown in FIG. 16, the pixels of the image sensor 121 may be configured to detect any one of red, blue, and green. The Bayer array BA in FIG. 16 may be applied to the pixels of the image sensor 121, and the number of pixels corresponding to the filter GF and detecting green light may be larger than the number of pixels corresponding to the filter RF and detecting red light and the number of pixels corresponding to the filter BF and detecting blue light.

[0090] The imaging element unit 12 has a driving unit (for example, the image sensor driving unit 123 in FIG. 6) that moves the position of the image sensor 121, and the image sensor 121 can be shifted in the horizontal and vertical directions in pixel units or sub-pixel units. For example, the imaging element unit 12 has an actuator (for example, the image sensor driving mechanism 122 in FIG. 6), and according to an instruction from the control unit 21, the shift operation (shift direction and shift amount) is controlled. For example, the imaging element unit 12 can move the image sensor 121 by a predetermined unit (for example, 0.5 pixel pitch, 1 pixel pitch, etc.) in the horizontal and vertical directions within at least a light receiving surface (a predetermined surface) perpendicular to the optical axis so that the image sensor 121 can capture a plurality of images.

[0091] In the above example, the case where the imaging device 1 realizes the pixel shift high-quality imaging mode by changing the position of the image sensor 121 has been described as an example. However, imaging may be performed by changing the position and orientation of components other than the image sensor 121. For example, the imaging device 1 may change the position and orientation of the lens system 11 or the device itself (that is, the entire imaging device 1) and perform imaging. For example, the imaging device 1 may change the position or orientation of the lens of the lens system 11. Also, the position or orientation of the imaging device 1 may be changed by a driving device (such as a pan-tilt head, a tripod, a stabilizer, etc.) or a user other than the imaging device 1.

[0092] The detection type defect correction processing unit 13 performs defect candidate pixel detection processing, interpolation target defect pixel determination processing, defect pixel interpolation processing, etc. on the captured image input from the imaging element unit 12, and passes the captured image after the interpolation processing to the composition processing unit 14. The detection type defect correction processing unit 13 is, for example, a detection type defect correction circuit 130 as shown in FIG. 4, but the details of FIG. 4 will be described later. Note that the detection type defect correction processing unit 13 may perform processing by software.

[0093] The composition processing unit 14 performs high-quality composition processing on the defect-corrected image in which defect pixels are interpolated by the detection type defect correction processing unit 13. The composition processing unit 14 performs high-quality composition processing according to the number of captured images as described above.

[0094] The developing processing unit 15 performs developing processing on the composite image generated by the composite processing unit 14. The developing processing performed by the developing processing unit 15 includes processing for converting RGB data into luminance data Y and color difference data C in the YC format, adjustment of white balance, γ correction, and the like. Note that the developing processing performed by the developing processing unit 15 does not include demosaicing processing.

[0095] The recording control unit 16 performs recording and playback on a recording medium, for example, a non-volatile memory. The recording control unit 16 performs processing for recording, for example, image files MF such as moving image data and still image data, and images after developing processing, on the recording medium. The actual form of the recording control unit 16 can be considered in various ways. For example, the recording control unit 16 may be configured as a flash memory built into the imaging device 1 and its writing / reading circuit, or may be in the form of a card recording / playback unit that performs recording / playback access on a recording medium that can be attached to and detached from the imaging device 1, such as a memory card (portable flash memory, etc.). Further, the recording control unit 16 may be an HDD (Hard Disk Drive) or the like in a form built into the imaging device 1.

[0096] The display unit 17 is a display unit that performs various displays for the imaging person, and is, for example, a display panel or viewfinder using a display device such as a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL (Electro-Luminescence) display arranged on the housing of the imaging device 1. The display unit 17 executes various displays on the display screen based on an instruction from the control unit 21. For example, the display unit 17 displays a reproduced image of the image data read from the recording medium in the recording control unit 16. Further, the display unit 17 executes displays such as various operation menus, icons, messages, etc., that is, displays as a GUI (Graphical User Interface) on the screen based on an instruction from the control unit 21.

[0097] The output unit 18 performs wired or wireless data communication and network communication with external devices. The output unit 18 transmits and outputs captured image data (still image files or video files) to, for example, an external display device, recording device, playback device, etc. Further, assuming that the output unit 18 is a network communication unit, it may perform communication via various networks such as the Internet, home network, LAN (Local Area Network), etc., and perform various data transmissions and receptions with servers, terminals, etc. on the network.

[0098] The operation unit 19 generally indicates input devices for the user to perform various operation inputs. Specifically, the operation unit 19 indicates various operation elements (keys, dials, touch panels, touch pads, etc.) provided on the housing of the imaging device 1. An operation of the user is detected by the operation unit 19, and a signal corresponding to the input operation is sent to the control unit 21.

[0099] The control unit 21 is composed of a microcomputer (arithmetic processing unit) equipped with a CPU (Central Processing Unit). The memory unit 20 stores information and the like used by the control unit 21 for processing. The memory unit 20 is shown comprehensively as, for example, ROM (Read Only Memory), RAM (Random Access Memory), flash memory, etc. The memory unit 20 may be a memory area built into the microcomputer chip as the control unit 21, or may be composed of a separate memory chip. The control unit 21 controls the entire imaging device 1 by executing programs stored in the ROM, flash memory, etc. of the memory unit 20. For example, the control unit 21 controls the shutter speed of the imaging element unit 12, gives instructions for various image processes in the detection type defect correction processing unit 13, imaging operations and recording operations according to the user's operations, playback operations of the recorded image files, operations of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, user interface operations, etc., and controls the operations of the necessary parts.

[0100] The RAM in the memory unit 20 is used for temporarily storing data, programs, etc. as a working area during various data processes of the CPU in the control unit 21. The ROM and flash memory (non-volatile memory) in the memory unit 20 are used for storing the OS (Operating System) for the CPU to control each part, content files such as image files, application programs for various operations, firmware, etc.

[0101] The driver unit 22 is provided with, for example, a motor driver for the zoom lens drive motor, a motor driver for the focus lens drive motor, a motor driver for the motor of the aperture mechanism, etc. These motor drivers apply a drive current to the corresponding driver according to an instruction from the control unit 21, and execute the movement of the focus lens and zoom lens, the opening and closing of the aperture blades of the aperture mechanism, etc.

[0102] The sensor unit 23 comprehensively shows various sensors mounted on the imaging device. As the sensor unit 23, for example, an acceleration sensor, a position information sensor, an illuminance sensor, etc. may be mounted. Note that the imaging device 1 in FIG. 3 may have the configuration shown in FIG. 6 other than the address type defect correction processing unit 24. The imaging device 1 in FIG. 3 may have a memory unit 20, a lens 111, a lens drive mechanism 221, a lens drive unit 222, an image sensor drive mechanism 122, and an image sensor drive unit 123.

[0103] Here, FIG. 4 will be described. FIG. 4 is a detailed block diagram of the detection type defect correction circuit. As shown in FIG. 4, the detection type defect correction circuit 130 has a defect candidate pixel detection unit 131, an interpolation target defect pixel determination unit 132, and an interpolation target defect pixel interpolation unit 133.

[0104] The defect candidate pixel detection unit 131 performs defect candidate pixel detection processing on the input RAW image. The input RAW image is stored as the captured image 203 temporarily stored in the memory 202. The memory 202 may be the memory unit 20. Note that the memory 202 may be inside the detection type defect correction circuit 130.

[0105] The detection result 204 of the defective pixel candidate detection process by the defective pixel candidate detection unit 131 is stored in the memory 202. The detection result 204 is, for example, a defective pixel candidate detection map indicating the addresses and count values of defective pixel candidates. Note that the detection result 204 is not limited to the defective pixel candidate detection map, and may be a defective pixel candidate address list or the like indicating the addresses of the pixels detected as defective pixel candidates. When the defective pixel candidate detection unit 131 performs a count-up, the defective pixel candidate detection unit 131 holds the count value of each pixel and stores the detection result 204 of the defective pixel candidate detection process in the memory 202. When the defective pixel candidate detection unit 131 grasps only the address of the defective pixel candidate, the defective pixel candidate detection unit 131 performs a count-up when overwriting the corresponding address in the memory 202 with the memory 202.

[0106] The imaging device 1 associates the input RAW image with the defect candidate pixel detection map which is the detection result 204 of the defect candidate pixel detection unit. Here, the term "associate" means, for example, enabling the use (linking) of one piece of information (data, command, program, etc.) when processing the other piece of information. That is, the information associated with each other may be grouped as one file or the like, or may be individual information. For example, the information B associated with the information A may be transmitted on a transmission path different from that of the information A. Also, for example, the information B associated with the information A may be recorded on a different recording medium (or a different recording area of the same recording medium). Note that this "association" may be a part of the information instead of the entire information. For example, an image and the information corresponding to the image may be associated with each other in any unit such as a plurality of frames, one frame, or a part within a frame. More specifically, for example, actions such as assigning the same ID (identification information) to a plurality of pieces of information, recording a plurality of pieces of information on the same recording medium, storing a plurality of pieces of information in the same folder, storing a plurality of pieces of information in the same file (assigning one as metadata to the other), embedding a plurality of pieces of information in the same stream, embedding metadata in an image like an electronic watermark, etc. are included in "associate". The recording control unit 16, the output unit 18, or the control unit 21 in FIG. 3 functions as an association unit that performs the above-described "associate" process.

[0107] The interpolation target defect pixel determination unit 132 performs an interpolation target defect pixel determination process. The interpolation target defect pixel determination unit 132 determines interpolation target defect pixels using the detection result 204 stored in the memory 202. For example, the interpolation target defect pixel determination unit 132 determines, among the defect candidate pixel detection map which is the detection result 204, pixels whose count value is equal to or greater than a threshold value as interpolation target defect pixels. The interpolation target defect pixel determination unit 132 passes an interpolation target defect pixel address list indicating the pixels determined as interpolation target defect pixels to the interpolation target defect pixel interpolation unit 133.

[0108] The interpolation target defective pixel interpolation unit 133 performs defective pixel interpolation processing based on the determination result by the interpolation target defective pixel determination unit 132. The interpolation target defective pixel interpolation unit 133 interpolates each pixel corresponding to the interpolation target defective pixel indicated by the interpolation target defective pixel address list among the pixels in the captured image 203 temporarily stored in the memory 202 with neighboring pixels. The interpolation target defective pixel interpolation unit 133 passes the defect-corrected image CIM generated by the defective pixel interpolation processing to the composition processing unit 14.

[0109] [1-4. Configuration of Image Processing Apparatus] Also, the image file MF can be transferred to an image processing apparatus TD such as the mobile terminal 2 and subjected to image processing. The mobile terminal 2 and the personal computer 3 that serve as the image processing apparatus TD can be realized as an image processing apparatus having, for example, the configuration shown in FIG. 5. Note that the server 4 can also be realized by an image processing apparatus having the configuration of FIG. 5. FIG. 5 is a block diagram of the image processing apparatus.

[0110] In FIG. 5, the CPU 71 of the image processing apparatus 70 executes various processes according to a program stored in the ROM 72 or a program loaded from the storage unit 79 to the RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes. The CPU 71, the ROM 72, and the RAM 73 are interconnected via a bus 74. An input / output interface 75 is also connected to this bus 74.

[0111] Connected to the input / output interface 75 is an input unit 76 composed of an operator and an operation device. For example, as the input unit 76, various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, and a remote controller are assumed. An operation of the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0112] In addition, to the input / output interface 75, a display unit 77 composed of an LCD or an organic EL panel, etc., and an audio output unit 78 composed of a speaker, etc. are connected integrally or separately. The display unit 77 is a display unit that performs various displays, and is configured by, for example, a display device provided on the housing of the image processing apparatus 70, or a separate display device connected to the image processing apparatus 70. The display unit 77 executes displays of images for various image processes, etc. on the display screen based on instructions from the CPU 71. Further, the display unit 77 performs displays such as various operation menus, icons, messages, etc., that is, displays as a GUI (Graphical User Interface) based on instructions from the CPU 71.

[0113] The input / output interface 75 may also be connected to a storage unit 79 composed of a hard disk, a solid state memory, etc., and a communication unit 80 composed of a modem, etc. The communication unit 80 performs communication processing via a transmission path such as the Internet, or performs communication by wired / wireless communication, bus communication, etc. with various devices.

[0114] In addition, a drive 82 is connected to the input / output interface 75 as necessary, and a removable recording medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted. With the drive 82, data files such as image files MF, etc., and various computer programs can be read from the removable recording medium 81. The read data files are stored in the storage unit 79, or images and sounds included in the data files are output by the display unit 77 and the audio output unit 78. Further, computer programs, etc. read from the removable recording medium 81 are installed in the storage unit 79 as necessary.

[0115] In this image processing apparatus 70, for example, software for image processing as the image processing apparatus of the present disclosure can be installed via network communication by the communication unit 80 or via the removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0116] [1-5. Other configurations of the imaging device] Here, with reference to FIG. 6, another example of the configuration of the imaging device 1, which is the image source VS and the image processing device TD of FIG. 2B, will be described. FIG. 6 is a block diagram of the imaging device. The imaging device 1 shown in FIG. 6 is different from the imaging device 1 shown in FIG. 3 in that it has an address-type defect correction processing unit 24 and a defect address storage unit 201. Note that descriptions of the same points as those of the imaging device 1 shown in FIG. 3 will be omitted as appropriate.

[0117] The imaging device 1 in FIG. 6 includes a lens 111, a lens driving mechanism 221, a lens driving unit 222, an image sensor 121, an image sensor driving mechanism 122, an image sensor driving unit 123, an address-type defect correction processing unit 24, a defect address storage unit 201, a detection-type defect correction processing unit 13, a synthesis processing unit 14, a development processing unit 15, an output unit 18, a memory unit 20, and a control unit 21. For example, the lens 111 corresponds to the lens system 11 in FIG. 3, and the lens driving mechanism 221 and the lens driving unit 222 correspond to the driver unit 22 in FIG. 3. Also, the image sensor 121, the image sensor driving mechanism 122, and the image sensor driving unit 123 correspond to the imaging element unit 12 in FIG. 3. Note that the imaging device 1 in FIG. 6 may include a recording control unit 16, a display unit 17, an operation unit 19, and a sensor unit 23.

[0118] The image sensor driving unit 123 shifts the image sensor 121 in pixel units or sub-pixel units in the horizontal and vertical directions. For example, the image sensor driving unit 123 shifts the image sensor 121 in accordance with an instruction from the control unit 21 by the image sensor driving mechanism 122, which is an actuator.

[0119] When changing the position or orientation of the lens 111, the lens driving unit 222 may shift the lens 111 in accordance with an instruction from the control unit 21 by the lens driving mechanism 221, which is an actuator.

[0120] The address type defect correction processing unit 24 performs address type defect correction processing on the captured image captured by the image sensor 121. The address type defect correction processing unit 24 performs address type defect correction processing on the defective pixels indicated by the addresses (positions) of the defective pixels stored in the defective address storage unit 201. The imaging device 1 may not have the address type defect correction processing unit 24 and the defective address storage unit 201. The output unit 18, the memory unit 20, or the control unit 21 in FIG. 6 functions as an association unit that performs the above-described "association" processing. Since other points are the same as those of the imaging device 1 shown in FIG. 3, the description thereof is omitted.

[0121] Note that some functions of the imaging device 1 described with reference to FIGS. 3 and 6 may be performed by an image processing device 70 such as a PC separate from the imaging device 1. For example, the imaging device 1 may perform processing up to the detection type defect correction processing unit 13, and the image processing device 70 may perform processing after the synthesis processing unit 14. Also, for example, the imaging device 1 may perform processing up to the interpolation target defective pixel determination unit 132, and the image processing device 70 may perform processing after the interpolation target defective pixel interpolation unit 133. Thus, image processing may be performed by the image processing device 70 as the image processing device TD and the imaging device 1. Note that the above is merely an example, and the functional sharing between the imaging device 1 and the image processing device 70 is not limited to the above. Hereinafter, a case where an image processing system 50 including the imaging device 1 and the image processing device 70 performs processing will be described.

[0122] [1-6. Processing Procedure by Image Processing System] First, with reference to FIGS. 7 and 8, the processing procedure by the image processing system 50 will be described. For the processing described below with the image processing system 50 as the main body of the processing, any of the imaging device 1 and the image processing device 70 included in the image processing system 50 may perform the processing.

[0123] First, with reference to FIG. 7, the flow of processing related to the image processing system will be described. FIG. 7 is a flowchart showing the processing procedure of the image processing system according to the embodiment of the present disclosure.

[0124] As shown in FIG. 7, the image processing system 50 detects defective candidate pixels for each captured image by performing defective candidate pixel detection processing on each of a plurality of captured images (step S101). The image processing system 50 determines pixels detected as defective candidate pixels a threshold number of times or more as defective pixels to be interpolated (step S102). When each of the plurality of captured images includes defective pixels to be interpolated, the image processing system 50 interpolates the defective pixels to be interpolated using pixels in the vicinity of the defective pixels to be interpolated (step S103).

[0125] Next, with reference to FIG. 8, an example of processing when functions are shared between the imaging device 1 and the image processing device 70 included in the image processing system 50 will be described. FIG. 8 is a sequence diagram showing the processing procedure of the image processing system.

[0126] As shown in FIG. 8, the imaging device 1 acquires a plurality of captured images by performing imaging processing in the pixel shift high-definition imaging mode (step S201).

[0127] Then, the imaging device 1 performs defective candidate pixel detection processing (step S202). Note that the steps shown in FIG. 8 are for convenience of explaining the processing, and the start of step S202 is not limited to the case where it starts after the completion of step S201. The imaging device 1 may perform imaging processing for acquiring the next captured image while performing defective candidate pixel detection processing on the captured image. In this way, the imaging device 1 may execute step S201 and step S202 in parallel.

[0128] Then, the imaging device 1 performs defective pixel to be interpolated determination processing (step S203). The imaging device 1 compares the number of times (count value) each pixel is detected as a defective candidate pixel with a threshold value, and determines a pixel detected as a defective candidate pixel a threshold number of times or more as a defective pixel to be interpolated.

[0129] Then, the imaging device 1 transmits a plurality of captured images and interpolation target defective pixel information DPI indicating interpolation target defective pixels to the image processing device 70 (step S204). Note that the imaging device 1 may transmit a plurality of captured images that have not undergone the detection type defect correction process in the present embodiment, but rather a plurality of captured images that have undergone the detection type defect correction process and interpolation target defective pixel information DPI at its own device to the image processing device 70. This point will be described with reference to FIG. 15.

[0130] Then, the image processing device 70 that has received information from the imaging device 1 performs defective pixel interpolation processing on each interpolation target defective pixel of the captured images including interpolation target defective pixels among the plurality of captured images (step S205).

[0131] Then, the image processing device 70 performs high-quality composite processing (step S206). For the image on which the defective pixel interpolation processing has been performed, the image processing device 70 uses the image after the defective pixel interpolation processing, and for the image on which the defective pixel interpolation processing has not been performed, the image processing device 70 uses the unprocessed image to perform high-quality composite processing.

[0132] Then, the image processing device 70 performs development processing on the composite image synthesized by the high-quality composite processing (step S207).

[0133] [1-7. Processing Examples by Image Processing System] Hereinafter, with reference to FIGS. 9 to 15, processing examples by the image processing system 50 will be described. Note that the interface LN1 indicated by the dotted line in FIGS. 9 to 15 is an example of the interface of the processing when, among the processing methods (variations) in each of FIGS. 9 to 15, the first half of the processing is performed by the imaging device 1 and the second half of the processing is performed by the image processing device 70 (for example, development software). As will be described later, the processing interface is not limited to the interface LN1 and may be in various forms.

[0134] [1-7-1. First Processing Example] First, with reference to FIG. 9, the first processing example will be described. FIG. 9 is a diagram showing an example of the outline of processing by the image processing system. FIG. 9 is a diagram showing the first processing example.

[0135] First, the premise of the diagram shown in FIG. 9 will be briefly described. The interface LN1 in FIG. 9 indicates the interface of processing in the image processing system 50 between the imaging device 1 side and the image processing device 70 side. The side (upper side) indicated by "(CM)" of the interface LN1 in FIG. 9 corresponds to the imaging device 1, and the side (lower side) indicated by "(PC)" of the interface LN1 in FIG. 9 corresponds to the image processing device 70. Note that the interface LN1 is an example, and any device in the image processing system 50 may perform each process. For example, the imaging device 1 may perform all of the processes shown in FIG. 9, or the image processing device 70 may perform all of the processes shown in FIG. 9. The same applies to FIGS. 10 to 15 in this regard.

[0136] The upper side of the interface LN1 in FIG. 9 indicates the processing and information generated by the imaging device 1. Also, the lower side of the interface LN1 in FIG. 9 indicates the processing and information generated by the image processing device 70. Further, the arrows crossing the interface LN1 in FIG. 9 indicate the transmission and reception of information between devices. For example, the arrow extending from the defect candidate pixel detection map MP11 to the interpolation target defect pixel determination unit 132 in FIG. 9 indicates that the defect candidate pixel detection map MP11 is transmitted from the imaging device 1 to the image processing device 70.

[0137] First, the outline of the processing in FIG. 9 will be described. FIG. 9 is an outline of the detection type defect correction processing in the first processing example. A defect candidate pixel detection map MP11 is generated by the defect candidate pixel detection processing of the imaging image group IMG by the defect candidate pixel detection unit 131. The defect candidate pixel detection map MP11 in FIG. 9 is a map having the same size as the RAW image (each imaging image IM1 to IM3, etc.) for storing whether each pixel was detected as an interpolation target defect pixel or not. For example, the defect candidate pixel detection map MP11 is developed on the memory 202 in FIG. 4. In the defect candidate pixel detection map, "1" is added (incremented) to the memory address corresponding to the pixel detected as a defect candidate pixel.

[0138] Here, for example, in the case of a true defective pixel, after processing the captured images (N images in FIG. 9), the value (count value) at the corresponding address (position) on the defective pixel candidate detection map MP11 becomes equal to or greater than the threshold value. On the other hand, in the case of a false detection depending on, for example, a subject other than a true defective pixel, since the image sensor moves slightly, it may be determined as a defect or not determined as a defect, so the possibility that the numerical value becomes equal to or greater than the threshold value is low. That is, in the case of a false detection, the value (count value) of the pixel corresponding to the false detection becomes less than the threshold value. Then, the defective pixel interpolation process by the interpolation target defective pixel interpolation unit 133 is applied only to the pixels whose count values are equal to or greater than the threshold value. Note that the specific processing contents of the "defective pixel candidate detection process" and the "interpolation target defective pixel interpolation process" may be the same as the above-described processes, but are not limited thereto.

[0139] Next, each process will be specifically described with reference to FIG. 9. In the example of FIG. 9, the imaging device 1 shows a case where processing is performed on an imaging image group IMG including N imaging images having a size (number of pixels) of horizontal W × vertical H. The imaging image group IMG includes imaging images such as IM1, IM2, and IM3, and the number of imaging images in the pixel shift high-quality imaging mode is the number of shift times (imaging times).

[0140] The defective pixel candidate detection unit 131 of the imaging device 1 performs defective pixel candidate detection processing on each pixel of each captured image in the captured image group IMG, and generates a defective pixel candidate detection map MP11 indicating the number of times (count value) each pixel is detected as a defective pixel. The defective pixel candidate detection map MP11 is a map with a size of horizontal W × vertical H corresponding to the size of each captured image in the captured image group IMG (W × H elements). For example, in the defective pixel candidate detection map MP11, the count values of all pixels are initialized to 0 before the defective pixel candidate detection processing, and 1 is added (incremented) to the address (count value) of the pixel detected as a defective pixel candidate. The imaging device 1 associates the captured image group IMG with the defective pixel candidate detection map MP11. The imaging device 1 transmits the defective pixel candidate detection map MP11 and the captured image group IMG to the image processing device 70. For example, in FIG. 9, the captured image group IMG and the defective pixel candidate detection map MP11 are associated with each other by any one of the recording control unit 16, the output unit 18, or the control unit 21 (see FIG. 3) or their cooperation and passed to the image processing device 70 (PC).

[0141] Then, the image processing device 70 uses the captured image group IMG and the defective pixel candidate detection map MP11 received from the imaging device 1 to perform interpolation target defective pixel determination processing by the interpolation target defective pixel determination unit 132 and defective pixel interpolation processing by the interpolation target defective pixel interpolation unit 133. The image processing device 70 determines that a pixel with a value in the defective pixel candidate detection map MP11 greater than or equal to the threshold is an interpolation target defective pixel, and applies defective pixel interpolation processing using neighboring pixels only for the interpolation target defective pixels.

[0142] Then, the image processing device 70 generates a defect-corrected image group CIG including N images such as defect-corrected images CI1, CI2, CI3, etc. in which defective pixel interpolation processing is performed on the interpolation target defective pixels. Then, the image processing device 70 performs high-quality synthesis processing and development processing using the defect-corrected image group CIG to generate a final image (output image) which is the image after the high-quality synthesis processing and the development processing. Note that when there is no need to perform display or the like, the development processing is unnecessary, and the synthesized image may be recorded, transmitted, or output.

[0143] [1-7-2. Second Processing Example] Next, with reference to FIG. 10, the second processing example will be described. FIG. 10 is a diagram showing an example of the outline of processing by the image processing system. FIG. 10 is a diagram showing the second processing example. FIGS. 10 to 15 described below are variations (modification examples) of FIG. 9. All of these have the same effect as FIG. 9. Note that descriptions of the same points as those in FIG. 9 will be omitted as appropriate.

[0144] First, the outline of the processing in FIG. 10 will be described. FIG. 10 shows a method of sending a defective pixel address list LT11 indicating the detected defective addresses to the latter half of the processing (below the interface LN1 in FIG. 10, that is, on the side of the image processing apparatus 70) instead of the defective candidate pixel detection map MP11 having the same size as the image size. In this case, reduction in the size of the data to be sent to the latter half of the processing (image processing apparatus 70) can be expected. Note that the length of the defective pixel address list LT11 to be transmitted is variable. Also, the defective pixel address list LT11 corresponds to the interpolation target defective pixel addresses described in FIG. 4.

[0145] Hereinafter, mainly the differences from FIG. 9 will be described with reference to FIG. 10. In the example of FIG. 10, the imaging device 1 generates a defective pixel address list LT11 from the defective candidate pixel detection map MP11. The configuration of the imaging device 1 that generates the defective pixel address list LT11 in FIG. 10 corresponds to the interpolation target defective pixel determination unit 132 (see FIG. 4). The imaging device 1 generates a defective pixel address list LT11 indicating the addresses of pixels whose values in the defective candidate pixel detection map MP11 are equal to or greater than the threshold value. The imaging device 1 associates the captured image group IMG with the defective pixel address list LT11. The imaging device 1 transmits the defective pixel address list LT11 and the captured image group IMG to the image processing apparatus 70.

[0146] Then, the image processing apparatus 70 performs defective pixel interpolation processing by the interpolation target defective pixel interpolation unit 133 using the captured image group IMG received from the imaging apparatus 1 and the defective address list LT11. The image processing apparatus 70 applies the defective pixel interpolation processing only to the pixels corresponding to the addresses included in the defective pixel address list LT11.

[0147] Then, the image processing apparatus 70 generates a corrected defective image group CIG including N images such as the corrected defective images CI1, CI2, CI3, etc. in which the defective pixels are corrected. Then, the image processing apparatus 70 generates a final image (output image) by performing high-quality composite processing and development processing using the corrected defective image group CIG.

[0148] [1-7-3. Third Processing Example] Next, the third processing example will be described with reference to FIG. 11. FIG. 11 is a diagram showing an example of the outline of processing by the image processing system. FIG. 11 is a diagram showing the third processing example. Note that descriptions of the same points as in FIGS. 9 and 10 will be omitted as appropriate.

[0149] First, the outline of the processing in FIG. 11 will be described. FIG. 11 assumes a case where the detection result of the defective candidate pixel detection processing by the detection type defective correction circuit CC can be output from the detection type defective correction circuit CC and the image processing apparatus 70 can receive the detection result. Other points are the same as those of the image processing system 50 corresponding to FIG. 9.

[0150] Hereinafter, while referring to FIG. 11, differences from FIGS. 9 and 10 etc. will be mainly described. In the example of FIG. 11, the imaging device 1 performs detection-type defect correction on the imaging image group IMG using the detection-type defect correction circuit CC. The detection-type defect correction circuit CC is a circuit (module) having a defect candidate pixel detection unit 131 that performs defect candidate pixel detection processing and a defect pixel interpolation unit 133a that performs defect pixel interpolation processing. Note that the detection-type defect correction circuit CC may be software that executes defect candidate pixel detection processing by the defect candidate pixel detection unit 131 and defect pixel interpolation processing by the defect pixel interpolation unit 133a. Also, the defect pixel interpolation processing by the defect pixel interpolation unit 133a of the detection-type defect correction circuit CC is processing for correcting pixels detected as defects for each image. The defect pixel interpolation processing by the defect pixel interpolation unit 133a shown in FIG. 11 is the same as the interpolation step of the above-described comparative technique.

[0151] The detection-type defect correction circuit CC performs defect candidate pixel detection processing on each image of the imaging image group IMG by defect candidate pixel detection processing, and generates a defect candidate pixel detection map MP11 based on the result. The detection-type defect correction circuit CC generates a defect candidate pixel detection map MP11 indicating the number of times (count value) each pixel is detected as a defect. For example, the detection-type defect correction circuit CC generates a defect candidate pixel detection map MP11 which is a map of size horizontal W×vertical H (W×H elements) corresponding to the size of each image of the imaging image group IMG by defect candidate pixel detection processing.

[0152] The detection-type defect correction circuit CC performs defect pixel interpolation processing using the result of the defect candidate pixel detection processing, and performs defect pixel interpolation processing on the defect pixels. Thereby, the detection-type defect correction circuit CC generates a defect-corrected image group CIG1 including N images such as defect-corrected images CI11, CI12, CI13 etc. in which the defect pixels are corrected. The detection-type defect correction circuit CC outputs the defect-corrected image group CIG1. The detection-type defect correction circuit CC can generate a defect-corrected image group CIG1 that is not synthesized and has been defect-corrected, and the imaging device 1 can acquire the defect-corrected image group CIG1.

[0153] The imaging device 1 acquires a defective candidate pixel detection map MP11 from the detection type defect correction circuit CC, and transmits the acquired defective candidate pixel detection map MP11 and the imaging image group IMG to the image processing device 70.

[0154] Then, the image processing device 70 performs an interpolation target defective pixel determination process by the interpolation target defective pixel determination unit 132 and a defective pixel interpolation process by the interpolation target defective pixel interpolation unit 133 using the imaging image group IMG and the defective candidate pixel detection map MP11 received from the imaging device 1. The image processing device 70 applies the defective pixel interpolation process only to the pixels whose values in the defective candidate pixel detection map MP11 are equal to or greater than the threshold value. In this case, the image processing device 70 applies the defective pixel interpolation process only to the pixels detected as defects in all the images of the imaging image group IMG.

[0155] Then, the image processing device 70 generates a defect-corrected image group CIG2 including N images such as defect-corrected images CI1, CI2, CI3, etc. in which the defective pixels are corrected. Then, the image processing device 70 generates a final image (output image) by performing high-image-quality synthesis processing and development processing using the defect-corrected image group CIG2.

[0156] [1-7-4. Fourth processing example] Next, the fourth processing example will be described with reference to FIG. 12. FIG. 12 is a diagram showing an example of the outline of the processing by the image processing system. FIG. 12 is a diagram showing the fourth processing example. Note that descriptions of the same points as in FIGS. 9 to 11 will be omitted as appropriate.

[0157] First, the outline of the process in FIG. 12 will be described. FIG. 12 shows a case where only the input and output of the detection type defect correction circuit CC can be observed (information can be acquired). In other words, it is a case where the processing inside the detection type defect correction circuit CC is a black box. For example, it is assumed that the detection type defect correction circuit CC is composed of a third-party manufactured signal processing IC or third-party manufactured software (IP, etc.). Other points are the same as those of the image processing system 50 corresponding to FIG. 9. The image processing system 50 corresponding to FIG. 12 may be changed in the same way as the change from the image processing system 50 corresponding to FIG. 9 to the image processing system 50 corresponding to FIG. 10.

[0158] Hereafter, while referring to FIG. 12, the differences from FIGS. 9 to 11 and the like will be mainly described. In the example of FIG. 12, the imaging device 1 performs detection type defect correction on the captured image group IMG using the detection type defect correction circuit CC. The detection type defect correction circuit CC in FIG. 12 executes the same processing as the detection type defect correction circuit CC in FIG. 11, but is different from the detection type defect correction circuit CC in FIG. 11 in that the result of the defect candidate pixel detection process by the defect candidate pixel detection unit 131 cannot be obtained. Since the detection type defect correction circuit CC in FIG. 12 is a black box, as long as the defect-corrected image group CIG1 can be output when the captured image group IMG is input, the internal configuration can be any configuration.

[0159] The imaging device 1 uses the detection type defect correction circuit CC to generate a defect-corrected image group CIG1 including N images such as defect-corrected images CI11, CI12, CI13, etc. in which defective pixels are corrected.

[0160] The difference processing unit 134 of the imaging device 1 calculates the difference between each pixel of each image in the captured image group IMG and each pixel of the image in the defect-corrected image group CIG1 corresponding to each image in the captured image group IMG. When a defective pixel is detected and corrected, the difference will not be zero. Therefore, a pixel with a non-zero difference can be detected as a defective pixel and it can be determined that defect correction has been performed. The difference processing unit 134 calculates the difference between each pixel of the captured image IM1 in the captured image group IMG and each pixel of the defect-corrected image CI11 in the defect-corrected image group CIG1 corresponding to that pixel. Then, for pixels with a non-zero difference, the difference processing unit 134 adds "1" to the defective pixel candidate detection map MP11. Note that not only when the difference is zero, but for example, considering noise, etc., "1" may be added when the difference is a predetermined value. Also, the difference processing unit 134 calculates the difference between each pixel of the captured image IM2 and each pixel of the defect-corrected image CI12 corresponding to that pixel. Then, for pixels with a non-zero difference, the difference processing unit 134 adds "1" to the defective pixel candidate detection map MP11. The difference processing unit 134 repeats the same process for N images. In this way, the imaging device 1 generates the defective pixel candidate detection map MP11 based on the comparison result between the captured image group IMG and the defect-corrected image group CIG1. The difference processing unit 134 associates the captured image group IMG with the defective pixel candidate detection map MP11. The imaging device 1 transmits the generated defective pixel candidate detection map MP11 and the captured image group IMG to the image processing device 70.

[0161] Then, the image processing device 70 uses the captured image group IMG and the defective pixel candidate detection map MP11 received from the imaging device 1 to perform the defective pixel interpolation target determination process by the defective pixel interpolation target determination unit 132 and the defective pixel interpolation process by the defective pixel interpolation unit 133. The subsequent processing is the same as that in FIG. 11, so the description is omitted.

[0162] [1-7-5. Fifth Processing Example] Next, with reference to FIG. 13, the fifth processing example will be described. FIG. 13 is a diagram showing an example of the outline of the processing by the image processing system. FIG. 13 is a diagram showing the fifth processing example. Note that the description of the same points as in FIGS. 9 to 12 will be omitted as appropriate.

[0163] First, the outline of the process in FIG. 13 will be described. FIG. 13 assumes a case where the detection result of the defective pixel detection process by the detection type defect correction circuit CC can be output from the detection type defect correction circuit CC, and the image processing apparatus 70 can receive the detection result. Thus, the detection type defect correction circuit CC in FIG. 13 is the same as the detection type defect correction circuit CC in FIG. 11.

[0164] Hereafter, while referring to FIG. 13, differences from FIGS. 9 to 12 and the like will mainly be described. In the example of FIG. 13, the imaging device 1 performs detection type defect correction on the captured image group IMG using the detection type defect correction circuit CC.

[0165] The detection type defect correction circuit CC performs defective pixel candidate detection processing on each image of the captured image group IMG by the defective pixel candidate detection processing by the defective pixel candidate detection unit 131, and generates a defective pixel candidate detection map MP11 based on the result. Further, the detection type defect correction circuit CC performs defective pixel interpolation processing on defective pixels by the defective pixel interpolation unit 133a using the result of the defective pixel candidate detection processing by the defective pixel candidate detection unit 131. Thereby, the detection type defect correction circuit CC generates a defect corrected image group CIG1 including N images such as defect corrected images CI11, CI12, CI13, etc. in which defective pixels are corrected. The detection type defect correction circuit CC outputs the defect corrected image group CIG1.

[0166] Also, as described above, the imaging device 1 can transmit the result of the defective pixel candidate detection processing by the defective pixel candidate detection unit 131 of the detection type defect correction circuit CC to the image processing apparatus 70. The imaging device 1 associates the captured image group IMG with the defective pixel candidate detection map MP11. The imaging device 1 acquires the defective pixel candidate detection map MP11 from the detection type defect correction circuit CC, and associates the captured image group IMG, the defective pixel candidate detection map MP11, and the defect corrected image group CIG1. The imaging device 1 transmits the defective pixel candidate detection map MP11, the captured image group IMG, and the defect corrected image group CIG1 to the image processing apparatus 70.

[0167] Then, for the pixels that use the captured image group IMG, the image processing apparatus 70 selects switch SW1, and for the pixels that use the defect-corrected image group CIG1, the image processing apparatus 70 selects switch SW2 to generate a defect-corrected image group CIG2. The image processing apparatus 70 selects switch SW2 for the pixels whose values in the defect candidate pixel detection map MP11 are equal to or greater than the threshold value. That is, for the pixels whose values in the defect candidate pixel detection map MP11 are equal to or greater than the threshold value, i.e., the defect pixels, the image processing apparatus 70 uses the pixels of the corrected defect-corrected image group CIG1, and for the pixels that are not defect pixels, the image processing apparatus 70 uses the pixels of the uncorrected captured image group IMG. The image processing apparatus 70 may have a selection unit that performs the above selection by switches SW1 and SW2.

[0168] Thereby, the image processing apparatus 70 generates a defect-corrected image group CIG2 including N images such as defect-corrected images CI1, CI2, CI3, etc. in which defect pixels are corrected. Then, the image processing apparatus 70 performs high-quality synthesis processing and development processing using the defect-corrected image group CIG to generate a final image (output image). Note that the image processing apparatus 70 may receive only the information necessary for generating the defect-corrected image group CIG2 from the imaging apparatus 1. In this case, the image processing apparatus 70 may request the imaging apparatus 1 for the selected image for each pixel and receive the pixel information of the selected image from the imaging apparatus 1.

[0169] [1-7-6. Sixth Processing Example] Next, with reference to FIG. 14, the sixth processing example will be described. FIG. 14 is a diagram showing an example of the outline of processing by the image processing system. FIG. 14 is a diagram showing the sixth processing example. Note that descriptions of the same points as in FIGS. 9 to 13 will be omitted as appropriate.

[0170] First, the outline of the processing in FIG. 14 will be described. FIG. 14 assumes a case where the imaging apparatus 1 cannot access information other than the image before correction and the image after correction. As such, in FIG. 14, the detection-type defect correction circuit CC is black-boxed, and only the defect-corrected image group CIG1 is output.

[0171] Hereinafter, with reference to FIG. 14, differences from FIGS. 9 to 13 and the like will be mainly described. In the example of FIG. 14, the imaging device 1 performs detection-type defect correction on the captured image group IMG using the detection-type defect correction circuit CC.

[0172] The imaging device 1 generates a defect-corrected image group CIG1 including N images such as defect-corrected images CI11, CI12, CI13, etc. in which defective pixels are corrected, using the detection-type defect correction circuit CC.

[0173] The difference processing unit 134 of the imaging device 1 generates a defect candidate pixel detection map MP11 by the same processing as in FIG. 12, such as taking the difference between each pixel of each image in the captured image group IMG and each pixel of each image in the defect-corrected image group CIG1 corresponding to that image. The difference processing unit 134 associates the captured image group IMG with the defect candidate pixel detection map MP11. The imaging device 1 transmits the generated defect candidate pixel detection map MP11, the captured image group IMG, and the defect-corrected image group CIG1 to the image processing device 70.

[0174] Then, the image processing device 70 selects the switch SW1 for the pixels using the captured image group IMG and selects the switch SW2 for the pixels using the defect-corrected image group CIG1 to generate the defect-corrected image group CIG2. Since the subsequent processing is the same as in FIG. 13, the description is omitted.

[0175] Also, in FIGS. 13 and 14, in the form where the processing below the interface LN1 (the latter half processing) is executed by the development software on the PC (such as the image processing device 70), the processing on the PC side can be lightened.

[0176] [1-7-7. The Seventh Processing Example] Next, with reference to FIG. 15, the seventh processing example will be described. FIG. 15 is a diagram showing an example of the outline of the processing by the image processing system. FIG. 15 is a diagram showing the seventh processing example. Note that the description of the same points as in FIGS. 9 to 14 will be omitted as appropriate.

[0177] First, the outline of the process in FIG. 15 will be described. FIG. 15 is an example of a case where data sent from the camera to the PC can be processed without including the image before defect correction. In FIG. 15, a table is generated that includes the addresses of the pixels determined to be defective and the absolute value of the difference between the interpolated pixel value and the pre-interpolated pixel value of each such pixel, and this table is sent to the subsequent stage (image processing device 70).

[0178] Hereafter, while referring to FIG. 15, the differences from FIGS. 9 to 14 and the like will mainly be described. In the example of FIG. 15, the imaging device 1 performs detection-type defect correction on the captured image group IMG using the detection-type defect correction circuit CC. The detection-type defect correction circuit CC in FIG. 15 is the same as the detection-type defect correction circuit CC in FIG. 12.

[0179] The imaging device 1 uses the detection-type defect correction circuit CC to generate a defect-corrected image group CIG1 that includes N images such as defect-corrected images CI11, CI12, CI13, etc. in which defective pixels have been corrected.

[0180] The difference processing unit 134 of the imaging device 1 performs difference processing to take the difference between corresponding pixels at corresponding positions between each image of the captured image group IMG and each image of the defect-corrected image group CIG1 corresponding to that image, and generates a difference map that includes all the differences.

[0181] In FIG. 15, the imaging device 1 generates a difference map group DMG that includes a plurality of difference maps each having a size of vertical H × horizontal W (H × W elements) corresponding to the size of each image of the captured image group IMG from the difference processing. For example, the imaging device 1 generates a difference map DM1 that shows the difference between each pixel of the captured image IM1 of the captured image group IMG and the defect-corrected image CI11 of the defect-corrected image group CIG1. The imaging device 1 generates the difference map DM1 by subtracting the pixel value of the pixel of the captured image IM1 corresponding to each pixel from the pixel value of the pixel of the defect-corrected image CI11. The imaging device 1 repeats the same process for N images. Thereby, the imaging device 1 generates a difference map group DMG that includes a plurality of difference maps such as difference maps DM1 to DM3.

[0182] Then, the imaging device 1 generates a table (also referred to as the "address-difference table TB11") for pixels whose values are not zero in an image with a number of images equal to or greater than a threshold value. Note that, not limited to the case where the value is zero, for example, taking noise or the like into account, the address-difference table TB11 may be generated for pixels whose values are a predetermined value in an image with a number of images equal to or greater than a threshold value. The address-difference table TB11 is information in which information (address) specifying each pixel is associated with information (difference) indicating the difference in each image. For example, in the address-difference table TB11, to the address #1 specifying one pixel, there are associated N differences such as the difference #1 indicating the difference between the first image combination (captured image IM1 and defect-corrected image CI11) and the difference #2 indicating the difference between the second image combination (captured image IM2 and defect-corrected image CI12).

[0183] In this way, the required data size of the address-difference table TB11 becomes variable length. On the other hand, by sorting the pixels included in the address-difference table TB11 in descending order of the absolute value of the difference and sending only the addresses of the upper pixels, the table size can be made fixed length and at the same time the data size can be reduced. In this case, pixels with a small absolute value of the difference, that is, defects with a small level, will not be corrected.

[0184] The imaging device 1 associates the defect-corrected image group CIG1 with the address-difference table TB11. The imaging device 1 transmits the address-difference table TB11 and the defect-corrected image group CIG1 to the image processing device 70. Note that, when the image processing device 70 does not have information such as the positional relationship of each image in the defect-corrected image group CIG1, the imaging device 1 may transmit metadata indicating the positional relationship and the like of each image in the defect-corrected image group CIG1 to the image processing device 70.

[0185] Then, the pixel value restoration unit 135 of the image processing apparatus 70 restores the pixel values before correction using the defect-corrected image group CIG1 received from the imaging apparatus 1 and the address-difference table TB11. The pixel value restoration unit 135 generates (restores) the pixel values before correction from the difference information for the pixels at the addresses recorded in the address-difference table TB11. For example, the pixel value restoration unit 135 restores the pixel value of that pixel in the captured image group IMG by subtracting the difference from the pixel value of that pixel in the defect-corrected image group CIG1 using the address of the pixel indicated by the address-difference table TB11 and the difference of that pixel. As a result, the image processing apparatus 70 generates a defect-corrected image group CIG2 including N images such as defect-corrected images CI1, CI2, CI3, etc. in which the pixel values of the pixels indicated by the address-difference table TB11 among the pixels in the defect-corrected image group CIG1 are restored to the values before correction. Then, the image processing apparatus 70 generates a final image (output image) by performing high-quality composite processing and development processing using the defect-corrected image group CIG2.

[0186] Note that the numerical values in the address-difference table TB11 may be the original pixel values instead of the absolute values of the differences. There is also a use case where, even in a form where N RAW images (data) are recorded for pixel shift high-quality improvement and sent to a PC, only 1 out of the N images is used for development at normal resolution. Therefore, according to the method of FIG. 15, since all N RAW images sent to the PC are defect-corrected, when only 1 out of them is used for development at normal resolution, it can be handled exactly the same as a normal single-shot RAW image (data) without any special processing. Also, not limited to the example of FIG. 15, when only 1 out of the N RAW images captured for high-quality improvement is used, if defect detection is performed on the N RAW images by the method described so far, false detection of defective pixels can be reduced, and by applying the defect detection result for correction, it is also possible to obtain better image quality with reduced contrast degradation and coloration.

[0187] As described above, for each of FIGS. 9 to 15, all of them may be performed on the camera (imaging device 1). Also, for each of FIGS. 9 to 15, all of them may be performed on the development processing software of the PC (image processing device 70). Further, the first half of the processing may be performed on the camera (imaging device 1), and the second half of the processing may be performed on the development processing software of the PC (image processing device 70). That is, the interface LN1 shown in FIGS. 9 to 15 is an example of the interface, and the interface of the system is not limited to being set to the interface LN1.

[0188] [2. Other Embodiments] The processing according to each of the above-described embodiments may be implemented in various different forms (modification examples) other than the above-described embodiments and modification examples.

[0189] [2-1. Others] Also, among the respective processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, regarding the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings, they can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0190] Also, each component of each illustrated device is a functional concept, and it is not necessarily physically configured as illustrated. That is, the specific form of the distribution and integration of each device is not limited to that shown, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage situations.

[0191] Also, the above-described embodiments and modification examples can be appropriately combined as long as the processing contents do not conflict.

[0192] Also, the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0193] [3. Effects of the Present Disclosure] As described above, the image processing apparatus (imaging apparatus 1 in the embodiment) according to the present disclosure includes a defective candidate pixel detection unit (defective candidate pixel detection unit 131 in the embodiment) and an interpolation target defective pixel determination unit (interpolation target defective pixel determination unit 132 in the embodiment). The defective candidate pixel detection unit performs defective candidate pixel detection processing on each of a plurality of captured images captured in a state where the positional relationship between the imaging range and the image sensor having a plurality of pixels is different from each other, thereby detecting defective candidate pixels for each captured image. The interpolation target defective pixel determination unit determines, as interpolation target defective pixels, the pixels detected as defective candidate pixels by the defective candidate pixel detection unit a threshold number of times or more.

[0194] In this way, the image processing apparatus according to the present disclosure performs defective candidate pixel detection processing on each of a plurality of captured images, detects defective candidate pixels for each captured image, and determines, as interpolation target defective pixels, the pixels detected as defective candidate pixels a threshold number of times or more, thereby appropriately determining defective pixels and preventing deterioration of the image quality of the image obtained in the pixel shift high image quality imaging mode.

[0195] The threshold value is a value less than or equal to the number of a plurality of captured images. In this way, the image processing apparatus can appropriately determine interpolation target defective pixels according to the number of a plurality of captured images by using a threshold value that is less than or equal to the number of a plurality of captured images.

[0196] The threshold value is a value less than the number of a plurality of captured images. In this way, the image processing apparatus can appropriately determine interpolation target defective pixels according to the number of a plurality of captured images by using a threshold value that is less than the number of a plurality of captured images.

[0197] The defect candidate pixel detection unit detects whether each target pixel is a defect candidate pixel based on the comparison result between the pixel value of each target pixel in each captured image of a plurality of captured images and the pixel values of detection neighboring pixels that are pixels in the neighborhood of each target pixel. In this way, the image processing apparatus can appropriately detect defect candidate pixels by detecting whether each target pixel is a defect candidate pixel based on the comparison result between the pixel value of each target pixel and the pixel values of neighboring pixels.

[0198] When the polarities of the differences between the target pixel and each of a plurality of detection neighboring pixels are the same and the absolute value of the difference exceeds a threshold value, the interpolation target defective pixel determination unit determines the target pixel as an interpolation target defective pixel when the target pixel has been determined to be a defect candidate pixel a threshold number of times or more by the defect candidate pixel detection process that detects the target pixel as a defect candidate pixel. In this way, the image processing apparatus can appropriately detect defect candidate pixels by using the polarities of the differences between the target pixel and each of a plurality of detection neighboring pixels to detect defect candidate pixels.

[0199] The image processing apparatus includes an interpolation target defective pixel interpolation unit (in the embodiment, the interpolation target defective pixel interpolation unit 133). When each of a plurality of captured images includes an interpolation target defective pixel, the interpolation target defective pixel interpolation unit interpolates the interpolation target defective pixel by using interpolation neighboring pixels that are pixels in the neighborhood of the interpolation target defective pixel, and outputs a plurality of defect-corrected images corresponding to the plurality of captured images. In this way, the image processing apparatus can appropriately interpolate the interpolation target defective pixel by interpolating the interpolation target defective pixel by using pixels in the neighborhood of the interpolation target defective pixel.

[0200] The interpolation target defective pixel interpolation unit interpolates the interpolation target defective pixel by using pixels included in the same captured image as the interpolation target defective pixel as the interpolation neighboring pixels. In this way, the image processing apparatus can appropriately interpolate the interpolation target defective pixel by interpolating the interpolation target defective pixel by using pixels included in the same captured image as the interpolation target defective pixel.

[0201] The image processing apparatus includes a synthesizing unit (in the embodiment, the synthesizing processing unit 14). For an imaging image in which the interpolation target defective pixel determination unit determines that an interpolation target defective pixel is included among a plurality of imaging images, the synthesizing unit uses, as a synthesis target image, an imaging image in which the interpolation target defective pixel interpolation unit has interpolated the interpolation target defective pixel. For an image in which the interpolation target defective pixel determination unit determines that no interpolation target defective pixel is included, the synthesizing unit uses the image as a synthesis target image and performs a synthesis process (high image quality synthesis process) using a plurality of synthesis target images to generate a synthesized image that is of higher image quality than each of the plurality of synthesis target images. In this way, the image processing apparatus can generate a synthesized image that is of higher image quality than the image before synthesis.

[0202] The synthesized image has a larger number of pixels than each of the plurality of imaging images. For example, for a shift amount with a unit of less than 1 pixel, an image obtained by imaging, for example, 8, 16, or more images and synthesizing them has a larger number of pixels than an image developed by a normal method using one of the plurality of imaging images. In this way, the image processing apparatus can generate a synthesized image that has a larger number of pixels than the image before synthesis.

[0203] The synthesized image has a higher color resolution than each of the plurality of imaging images. For example, an image obtained by synthesizing images taken 4 times with a 1-pixel shift has a higher oblique resolution and color resolution than an image developed by a normal method using one of the 4 images. In this way, the image processing apparatus can generate a synthesized image that has a higher color resolution than the image before synthesis.

[0204] The image processing apparatus calculates the difference between pixels at corresponding positions for each of the plurality of imaging images and the defect-corrected images among the plurality of defect-corrected images corresponding to each of the plurality of imaging images, and generates interpolation target defective pixel information (interpolation target defective pixel information DPI) indicating that a pixel with a difference equal to or greater than a predetermined value is a defective pixel. In this way, the image processing apparatus can generate interpolation target defective pixel information indicating an interpolation target pixel by using the difference between pixels of the plurality of imaging images and the defect-corrected image after interpolation processing.

[0205] The image processing apparatus includes an association unit (in the embodiment, the difference processing unit 134). The association unit associates a plurality of captured images with interpolation target defective pixel information. In this way, the image processing apparatus can identify interpolation target pixels among the pixels of the plurality of captured images based on the association by associating the plurality of captured images with the interpolation target defective pixel information.

[0206] The plurality of captured images are images captured in a state where the positional relationship between the imaging range and the image sensor is different from each other in pixel units or sub-pixel units. The image processing apparatus can appropriately determine interpolation target defective pixels for the plurality of captured images captured with different positional relationships in pixel units or sub-pixel units.

[0207] The defective pixel candidate detection unit performs defective pixel candidate detection processing on each of the plurality of captured images captured in a state where the positional relationship between the imaging range and the image sensor having a plurality of pixels is different from each other, thereby detecting defective pixel candidates for each captured image. The association unit (in the embodiment, the recording control unit 16, the output unit 18, the memory unit 20, or the control unit 21, etc.) associates the captured image with the detection result of the defective pixel candidate detection unit. In this way, the image processing apparatus can identify defective pixel candidates among the pixels of the plurality of captured images based on the association by associating the plurality of captured images with the detection result of the defective pixel candidates.

[0208] The detection result is defective pixel candidate detection information (defective pixel candidate detection map) indicating the number of times each pixel of the captured image is detected as a defective pixel candidate by the defective pixel candidate detection unit. In this way, the image processing apparatus can identify the number of times each pixel of the plurality of captured images is detected as a defective pixel candidate based on the association by associating the plurality of captured images with the defective pixel candidate detection information indicating the number of times detected as a defective pixel candidate.

[0209] The interpolation target defective pixel determination unit determines, based on the number of times equal to or greater than a threshold value, the defective candidate pixels and the detected pixels detected by the defective candidate pixel detection unit as interpolation target defective pixels. In this way, the image processing apparatus can prevent the degradation of the image quality of the image obtained in the pixel shift high image quality imaging mode by appropriately determining defective pixels by determining the defective candidate pixels and the detected pixels detected by the defective candidate pixel detection unit as interpolation target defective pixels based on the number of times equal to or greater than the threshold value.

[0210] The detection result is interpolation target defective pixel address information (interpolation target defective pixel address list) indicating the addresses of the interpolation target defective pixels. In this way, the image processing apparatus can make it possible to specify the addresses of the interpolation target defective pixels based on the association by associating a plurality of captured images with the interpolation target defective pixel address information indicating the addresses of the interpolation target defective pixels.

[0211] The association unit associates, among the absolute values of a plurality of differences calculated for pixels at corresponding positions in a plurality of captured images and a plurality of defect-corrected images, the address difference information indicating the addresses and differences of the pixels from the largest absolute value of the differences to a predetermined rank with the plurality of defect-corrected images. In this way, the image processing apparatus can make it possible to specify the image to which the address difference information is to be applied by associating the address difference information indicating the addresses and differences of the pixels with the larger absolute value of the differences with the plurality of defect-corrected images. As a result, the receiving-side apparatus that has received the address difference information from the image processing apparatus can specify the image to which the address difference information is to be applied and apply the address difference information to that image. Further, the image processing apparatus can suppress an increase in the amount of data to be transmitted to an external apparatus by using the address difference information.

[0212] The image processing apparatus includes a pixel value restoration unit (in the embodiment, the pixel value restoration unit 135). The pixel value restoration unit restores the pixel value of the pixel in the plurality of captured images by using a plurality of defect-corrected images corresponding to each of the plurality of captured images, the address of the pixel for which the absolute value of the difference calculated for the pixels at the corresponding positions in the plurality of captured images is greater than or equal to a predetermined value, the address difference information indicating the difference of the pixel, and the plurality of defect-corrected images. In this way, the image processing apparatus can restore the pixel value of the pixel in the plurality of captured images from the plurality of defect-corrected images by using the address difference information.

[0213] [4. Hardware Configuration] An information device such as the image processing apparatus TD according to each of the above-described embodiments is realized by a computer 1000 having a configuration as shown in FIG. 31, for example. FIG. 31 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of an image processing apparatus such as the image processing apparatus TD. Hereinafter, the imaging apparatus 1 according to the embodiment will be described as an example. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface 1500, and an input / output interface 1600. Each part of the computer 1000 is connected by a bus 1050.

[0214] The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each part. For example, the CPU 1100 expands a program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0215] The ROM 1300 stores a boot program such as a BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started up, and programs depending on the hardware of the computer 1000.

[0216] The HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100, data used by such programs, and the like. Specifically, the HDD 1400 is a recording medium that records an image processing program according to the present disclosure, which is an example of program data 1450.

[0217] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices or transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0218] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. Also, the CPU 1100 transmits data to output devices such as a display, a speaker, and a printer via the input / output interface 1600. Further, the input / output interface 1600 may function as a media interface for reading programs and the like recorded on a recording medium. The recording medium is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0219] For example, when the computer 1000 functions as the imaging device 1 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 21 and the like by executing the image processing program loaded on the RAM 1200. Further, the HDD 1400 stores the image processing program according to the present disclosure and the data in the memory unit 20. Although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, these programs may be acquired from other devices via the external network 1550.

[0220] Note that the present technology can also adopt the following configuration. (1) A defect candidate pixel detection unit that detects defect candidate pixels for each of the plurality of captured images by performing defect candidate pixel detection processing on each of the plurality of captured images captured in a state where the positional relationship between the imaging range and the image sensor having a plurality of pixels is different from each other; An interpolation target defect pixel determination unit that determines, by the defect candidate pixel detection unit, the defect candidate pixels and the detected pixels as interpolation target defect pixels a threshold number of times or more; An image processing apparatus comprising: (2) The threshold value is a value equal to or less than the number of the plurality of captured images The image processing apparatus according to (1). (3) The threshold value is a value less than the number of the plurality of captured images The image processing apparatus according to (1). (4) The defect candidate pixel detection unit detects whether each of the target pixels is a defect candidate pixel based on a comparison result between the pixel value of each target pixel in each of the plurality of captured images and the pixel value of a detection target neighboring pixel that is a neighboring pixel of each target pixel The image processing apparatus according to any one of (1) to (3). (5) The interpolation target defect pixel determination unit When the polarities of the differences between the target pixel and each of the plurality of neighboring pixels for detection are the same, and the absolute value of the difference exceeds a threshold value, the target pixel is detected as a defective candidate pixel by the defective candidate pixel detection process. When the target pixel is determined to be the defective candidate pixel the number of times that the value is equal to or greater than the threshold value, the target pixel is determined to be an interpolation target defective pixel. The image processing apparatus according to (4). (6) When each of the plurality of captured images includes the interpolation target defective pixel, an interpolation target defective pixel interpolation unit that outputs a plurality of defect-corrected images corresponding to the plurality of captured images by interpolating the interpolation target defective pixel using an interpolation neighboring pixel that is a pixel in the neighborhood of the interpolation target defective pixel. The image processing apparatus according to any one of (1) to (5), further comprising this. (7) The interpolation target defective pixel interpolation unit Interpolates the interpolation target defective pixel using a pixel included in the same captured image as the interpolation target defective pixel as the interpolation neighboring pixel. The image processing apparatus according to (6). (8) Among the plurality of captured images For the captured image determined by the interpolation target defective pixel determination unit to include the interpolation target defective pixel, the captured image obtained by interpolating the interpolation target defective pixel by the interpolation target defective pixel interpolation unit is used as a synthesis target image. For the image determined by the interpolation target defective pixel determination unit not to include the interpolation target defective pixel, after using this image as a synthesis target image A synthesis unit that generates a synthesized image with higher image quality than each of the plurality of synthesis target images by performing a synthesis process using the plurality of synthesis target images. The image processing apparatus according to (6) or (7), further comprising this. (9) The synthesized image Has more pixels than each of the plurality of captured images The image processing apparatus according to (8). (10) The synthesized image Higher color resolution than each of the plurality of captured images The image processing apparatus according to (8) or (9). (11) For each of the plurality of captured images and each of the plurality of defect-corrected images corresponding to each of the plurality of captured images among the plurality of defect-corrected images, calculate the difference between the pixels at corresponding positions, and generate interpolation target defect pixel information indicating that the pixel with a difference of a predetermined value or more is a defective pixel The image processing apparatus according to any one of (6) to (10). (12) An association unit that associates the plurality of captured images with the interpolation target defect pixel information The image processing apparatus according to (11), further comprising the above. (13) The plurality of captured images are Images captured in a state where the positional relationship between the imaging range and the image sensor is made different from each other in pixel units or sub-pixel units The image processing apparatus according to any one of (1) to (12). (14) A defect candidate pixel detection unit that performs a defect candidate pixel detection process on each of a plurality of captured images captured in a state where the positional relationship between the imaging range and an image sensor having a plurality of pixels is made different from each other, and detects defect candidate pixels for each of the captured images, An association unit that associates the captured image with the detection result of the defect candidate pixel detection unit An image processing apparatus comprising the above. (15) The detection result is Defect candidate pixel detection information indicating the number of times each pixel of the captured image is detected as the defect candidate pixel by the defect candidate pixel detection unit The image processing apparatus according to (14). (16) An interpolation target defect pixel determination unit that determines, by the defect candidate pixel detection unit, a pixel detected as the defect candidate pixel a threshold number of times or more as an interpolation target defect pixel The image processing apparatus according to (14) or (15), further comprising the above. (17) The detection result is interpolation target defective pixel address information indicating the address of the interpolation target defective pixel the image processing apparatus according to (16). (18) An association unit that associates the addresses of pixels up to a predetermined rank from the larger of the absolute values of a plurality of differences calculated for pixels at corresponding positions in the plurality of captured images and the plurality of defect-corrected images, and the address difference information indicating the differences, with the plurality of defect-corrected images, The image processing apparatus according to any one of (6) to (12), further comprising. (19) A pixel value restoration unit that restores the pixel value of the pixel in the plurality of captured images using the plurality of defect-corrected images corresponding to each of the plurality of captured images, the address of the pixel whose absolute value of the difference calculated for the pixels at corresponding positions in the plurality of captured images and the plurality of defect-corrected images is equal to or greater than a predetermined value, and the address difference information indicating the difference of the pixel, An image processing apparatus comprising. (20) By performing a defect candidate pixel detection process on each of a plurality of captured images captured in a state where the positional relationship between the imaging range and an image sensor having a plurality of pixels is different from each other, defect candidate pixels are detected for each of the captured images, The defect candidate pixel and the detected pixel are determined as interpolation target defective pixels for a threshold number of times or more, An image processing method for executing control. (21) By performing a defect candidate pixel detection process on each of a plurality of captured images captured in a state where the positional relationship between the imaging range and an image sensor having a plurality of pixels is different from each other, defect candidate pixels are detected for each of the captured images, An image processing program for causing the defect candidate pixel and the detected pixel to be determined as interpolation target defective pixels for a threshold number of times or more and executing control.

Description of Signs

[0221] 1 Imaging device 11 Lens system 12 Image sensor unit 121 Image sensor 13 Detection-type defect correction processing unit 131 Defect candidate pixel detection unit 132 Interpolation target defect pixel determination unit 133 Interpolation target defect pixel interpolation unit 14 Composite processing unit 15 Development processing unit 16 Recording control unit 17 Display unit 18 Output unit 19 Operation unit 20 Memory unit 21 Control unit 22 Driver unit 23 Sensor unit 50 Image processing system 70 Image processing apparatus

Claims

1. A defect candidate pixel detection unit that detects defect candidate pixels for each of a plurality of captured images captured in a state where the positional relationship between the imaging range and an image sensor having a plurality of pixels is made different from each other, by performing defect candidate pixel detection processing on each of the plurality of captured images; An interpolation target defect pixel determination unit that determines, by the defect candidate pixel detection unit, a pixel detected as the defect candidate pixel and the interpolation target defect pixel a threshold number of times or more; Comprising; The plurality of captured images are Images captured in a state where the positional relationship between the imaging range and the image sensor is made different from each other in pixel units or sub-pixel units Image processing apparatus.

2. The threshold value is A value less than or equal to the number of the plurality of captured images The image processing apparatus according to claim 1.

3. The threshold value is A value less than the number of the plurality of captured images The image processing apparatus according to claim 1.

4. The defect candidate pixel detection unit Detects whether each of the target pixels is a defect candidate pixel based on a comparison result between the pixel value of each target pixel in each of the plurality of captured images and the pixel value of a detection target neighboring pixel that is a neighboring pixel of each target pixel The image processing apparatus according to claim 1.

5. The interpolation target defect pixel determination unit When the polarities of the differences between the target pixel and each of the plurality of detection target neighboring pixels are the same and the absolute value of the difference exceeds a threshold value, and when the target pixel is determined to be the defect candidate pixel a threshold number of times or more by the defect candidate pixel detection process that detects the target pixel as the defect candidate pixel, determines the target pixel as the interpolation target defect pixel The image processing apparatus according to claim 4.

6. An interpolation target defect pixel interpolation unit that outputs a plurality of defect-corrected images corresponding to the plurality of captured images by interpolating the interpolation target defect pixels using interpolation neighboring pixels that are neighboring pixels of the interpolation target defect pixels when each of the plurality of captured images includes the interpolation target defect pixels; The image processing apparatus according to claim 1, further comprising.

7. The interpolation target defect pixel interpolation unit Interpolates the interpolation target defect pixel using a pixel included in the same captured image as the interpolation target defect pixel as the interpolation neighboring pixel The image processing apparatus according to claim 6.

8. Among the plurality of captured images, For the captured image determined by the interpolation target defective pixel determination unit to include the interpolation target defective pixel, the captured image obtained by interpolating the interpolation target defective pixel by the interpolation target defective pixel interpolation unit is used as the composite target image, For the captured image determined by the interpolation target defective pixel determination unit not to include the interpolation target defective pixel, after using the captured image as the composite target image, A composite unit that generates a composite image with higher image quality than each of the plurality of composite target images by performing a composite process using the plurality of composite target images, The image processing apparatus according to claim 6, further comprising.

9. The composite image, Has more pixels than each of the plurality of captured images The image processing apparatus according to claim 8.

10. The composite image, Has higher color resolution than each of the plurality of captured images The image processing apparatus according to claim 8.

11. For each of the plurality of captured images and the plurality of defect-corrected images, among the plurality of defect-corrected images corresponding to each of the plurality of captured images, calculate the difference between the pixels at corresponding positions, and generate interpolation target defective pixel information indicating that pixels with a difference of a predetermined value or more are defective pixels The image processing apparatus according to claim 6.

12. An association unit that associates the plurality of captured images with the interpolation target defective pixel information, The image processing apparatus according to claim 11, further comprising.

13. A defect candidate pixel detection unit that detects defect candidate pixels for each of the plurality of captured images by performing a defect candidate pixel detection process on each of the plurality of captured images captured with different positional relationships between the imaging range and the image sensor having a plurality of pixels, An association unit that associates the captured image with the detection result of the defect candidate pixel detection unit, An interpolation target defective pixel determination unit that determines, by the defect candidate pixel detection unit, the defect candidate pixel and the detected pixel as interpolation target defective pixels a threshold number of times, Comprising, The plurality of captured images, Are images captured with different positional relationships between the imaging range and the image sensor in pixel units or sub-pixel units Image processing apparatus.

14. The detection result, Is interpolation target defective pixel address information indicating the address of the interpolation target defective pixel The image processing apparatus according to claim 13.

15. An association unit that associates the addresses of pixels up to a predetermined rank from the ones with larger absolute values of a plurality of differences calculated for pixels at corresponding positions in the plurality of captured images and the plurality of defect-corrected images, and the address difference information indicating the differences, with the plurality of defect-corrected images. The image processing apparatus according to claim 6, further comprising the above.

16. For each of a plurality of captured images captured in a state where the positional relationship between the imaging range and an image sensor having a plurality of pixels is made different from each other, defect candidate pixel detection processing is performed to detect defect candidate pixels for each of the captured images. Determine, as interpolation target defect pixels, the defect candidate pixels and the detected pixels a threshold number of times or more. Execute control. The plurality of captured images are Images captured in a state where the positional relationship between the imaging range and the image sensor is made different from each other in pixel units or sub-pixel units. Image processing method.

17. For each of a plurality of captured images captured in a state where the positional relationship between the imaging range and an image sensor having a plurality of pixels is made different from each other, defect candidate pixel detection processing is performed to detect defect candidate pixels for each of the captured images. Cause a computer to execute control to determine, as interpolation target defect pixels, the defect candidate pixels and the detected pixels a threshold number of times or more. The plurality of captured images are Images captured in a state where the positional relationship between the imaging range and the image sensor is made different from each other in pixel units or sub-pixel units. Image processing program.

Images