Image processing apparatus

Abstract

To provide a technology that can suppress deterioration of stereoscopic vision due to a foreign object.SOLUTION: An image processing apparatus according to the present invention comprises: acquisition means that acquires a first image imaged through a first optical system and a second image imaged through a second optical system; detection means that detects an area satisfying a predetermined condition as a foreign object area from one image of the first image and the second image; and correction means that corrects a pixel value of the foreign object area in the one image using a pixel value of the other image of the first image and the second image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] There is a known technology that displays two images with a parallax between them to enable stereoscopic viewing. If a foreign object (for example, a foreign object attached to the optical system used for capturing images) appears in only one of the two images, stereoscopic viewing becomes difficult and the immersive feeling in stereoscopic viewing is lost (deterioration of stereoscopic viewing).

[0003] Patent Document 1 discloses a technique for detecting a foreign object adhering to an optical system based on the parallax of stereo images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-130549 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if a foreign object can be detected using the technology disclosed in Patent Document 1, the stereoscopic vision will not improve unless the foreign object is removed from the optical system. Some users may not be able to remove the foreign object well, and may end up adhering more foreign objects to the optical system or damaging the optical system, further worsening the stereoscopic vision.

[0006] An object of the present invention is to provide a technique capable of suppressing deterioration of stereoscopic vision caused by foreign matter. [Means for solving the problem]

[0007] A first aspect of the present invention is an image processing device characterized by having an acquisition means for acquiring a first image captured via a first optical system and a second image captured via a second optical system, a detection means for detecting an area satisfying a predetermined condition from one of the first image and the second image as a foreign object area, and a correction means for correcting pixel values ​​of the foreign object area in the one image using pixel values ​​of the other of the first image and the second image.

[0008] A second aspect of the present invention is an image processing method comprising the steps of: acquiring a first image captured via a first optical system and a second image captured via a second optical system; detecting an area in one of the first image and the second image that satisfies predetermined conditions as a foreign object area; and correcting pixel values ​​of the foreign object area in the one image using pixel values ​​of the other of the first image and the second image.

[0009] A third aspect of the present invention is a program for causing a computer to function as each of the means of the image processing device described above.A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device described above. Effect of the Invention

[0010] According to the present invention, it is possible to suppress deterioration of stereoscopic vision caused by foreign matter. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of a configuration of an imaging device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a configuration of an image processing unit according to the first embodiment. [Diagram 3] FIG. 2 is a schematic diagram showing an example of a distribution of pixel values ​​according to the first embodiment. [Figure 4]5 is a flowchart showing an example of image correction processing according to the first embodiment. [Diagram 5] FIG. 2 is a schematic diagram showing an example of various data according to the first embodiment. [Figure 6] FIG. 2 is a schematic diagram showing an example of an image circle according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of the configuration of an image processing unit according to a second embodiment. [Figure 8] FIG. 11 is a schematic diagram showing an example of a method for determining an effective angle of view according to the second embodiment. [Figure 9] 10 is a flowchart showing an example of image correction processing according to the second embodiment. [Figure 10] FIG. 11 is a schematic diagram showing an example of various data according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Embodiment 1> Hereinafter, a first embodiment of the present invention will be described. Note that, although an example in which the present invention is applied to an imaging device will be described, the device to which the present invention can be applied is not limited to the imaging device. The present invention is applicable to various electronic devices (image processing devices) capable of performing image processing on captured images. For example, the present invention is also applicable to a personal computer connected to an imaging device, a head-mounted display connected to an imaging device, or a video see-through type head-mounted display.

[0013] (Imaging device) FIG. 1 is a block diagram showing an example of the configuration of an imaging device 100 according to the first embodiment.

[0014] The control unit 101 is, for example, a CPU, and controls each unit of the imaging device 100 by reading a program from the ROM 102, expanding it into the RAM 103, and executing it. The ROM 102 is a rewritable non-volatile memory, and stores various data (information) such as the program executed by the control unit 101 and parameters necessary for controlling each unit. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data output by each unit of the imaging device 100.

[0015] Each of the optical systems 104A and 104B is an optical system used for capturing an image, and includes a lens and an aperture. The control unit 101 controls the optical system 104A to perform focus adjustment and exposure adjustment of an image captured via the optical system 104A. The control unit 101 can also control the optical system 104B to perform focus adjustment and exposure adjustment of an image captured via the optical system 104B.

[0016] The imaging unit 105 is an imaging element such as a CCD or CMOS sensor. The optical system 104A and the optical system 104B are arranged side by side. Therefore, as shown in FIG. 6, in the imaging unit 105, the image circle IC1 of the optical system 104A and the image circle IC2 of the optical system 104B are also arranged side by side. The light that has passed through the optical system 104A forms an image in the image circle IC1, and the light that has passed through the optical system 104B forms an image in the image circle IC2. The imaging unit 105 performs photoelectric conversion of the two optical images formed in the two image circles IC1 and IC2, respectively, and outputs the obtained analog image signal to the A / D conversion unit 106. The A / D conversion unit 106 performs A / D conversion processing on the input analog image signal, and outputs (stores) the obtained digital image data to the RAM 103.

[0017] In the first embodiment, one image including two image areas captured through two optical systems is obtained using one image sensor. The method of acquiring the two images is as follows: For example, two images may be captured by using two image pickup elements and through two optical systems, respectively, to obtain two separate images.

[0018] The image processing unit 107 performs various image processing such as white balance adjustment, color interpolation, reduction / enlargement, filtering, encoding, and decoding on the image data stored in the RAM 103. The image processing unit 107 can also perform various image processing on the image data (image data including two image areas corresponding to the two image circles IC1 and IC2 in FIG. 6) while taking into consideration the optical center shift, and generate image data for stereoscopic viewing. The optical center shift corresponds to the shift between the center of the image circle IC1 and the center of the image circle IC2, and also corresponds to the shift between the optical axis of the optical system 104A and the optical axis of the optical system 104B. The image processing unit 107 can also perform image processing on the image data to remove foreign matter (for example, foreign matter attached to the optical system or scratches on the optical system) that appears in one of the two image areas corresponding to the two image circles IC1 and IC2.

[0019] The storage medium 108 stores various data. For example, the storage medium 108 stores image data stored in the RAM 103 (for example, image data after image processing by the image processing unit 107, or image data output from the A / D conversion unit 106) in a predetermined file format. The storage medium 108 may be a storage device that is detachable from the imaging device 100, for example, a memory card. The communication unit 109 transmits and receives various data to and from an external device by wired communication or wireless communication. For example, the communication unit 109 transmits an image file stored in the storage medium 108 to the external device.

[0020] The display unit 110 displays various images. For example, the display unit 110 displays an image that has been captured and stored in the storage medium 108, a live view image, or a menu screen. By displaying a live view image, the display unit 110 can be used as an electronic viewfinder.

[0021] The measurement unit 112 includes various sensors such as a gyro sensor and a distance measuring sensor, and acquires various information such as information on the attitude of the image capturing device 100 and information on the distance from the image capturing device 100 to a subject.

[0022] The operation unit 111 is a group of input devices for the user to input various instructions to the imaging device 100, and includes various input devices such as a shutter button, a menu button, direction keys, and an enter key. If the display unit 110 is a touch display, the display unit 110 also serves as the operation unit 111. Note that the operation unit 111 may include an input device that does not require physical operation, such as a combination of a microphone and a voice command recognition unit, or a combination of an eye gaze detection sensor and an eye gaze input recognition unit.

[0023] (Image processing unit) 2 is a block diagram showing an example of the configuration of the image processing unit 107. In the first embodiment, an image including two image regions corresponding to the image circles IC1 and IC2, respectively, is acquired, but in the following, each of the two image regions will be described as an image.

[0024] The image selection unit 201 selects one of two images (two image areas corresponding to image circles IC1 and IC2, respectively) captured via the optical systems 104A and 104B as a reference image, and selects the other of the two images as an image to be corrected. For example, the image selection unit 201 aligns the captured current image (current frame) with a past image (e.g., the frame immediately preceding the current frame) read from the RAM 103, and calculates a value related to the correlation between those images (correlation value). The image selection unit 201 selects the current image corresponding to the optical system 104A (image circle IC1) and the past image corresponding to the optical system 104B (image circle IC2) as a reference image, and selects the other of the two images as an image to be corrected. For example, the image selection unit 201 aligns the captured current image (current frame) with a past image read from the RAM 103 (e.g., the frame immediately preceding the current frame), and calculates a value related to the correlation between those images (correlation value). The image selection unit 201 then calculates a correlation value for each of the two current images corresponding to the circle IC2) and selects, from among the two current images, the image having a stronger correlation with the past image as the reference image, and selects, from among the two current images, the image having a weaker correlation with the past image as the image to be corrected.

[0025] The image selection unit 201 calculates the sum of absolute differences as the correlation value, for example, using the following formula 1.

number

[0026] In formula 1, f(i,j) represents a pixel value at coordinates (horizontal position, vertical position)=(i,j) of the current image, and g(i,j) represents a pixel value at coordinates (i,j) of the past image. The absolute value (absolute difference value) of the value obtained by subtracting pixel value g(i,j) from pixel value f(i,j) is calculated, and the sum of the multiple absolute difference values ​​corresponding to multiple coordinates is calculated as correlation value SAD1 (sum of absolute difference values). The smaller the correlation value SAD1, the stronger the correlation between the current image and the past image, and the larger the correlation value SAD1, the weaker the correlation between the current image and the past image. The correlation value SAD1 is also a value related to the similarity between the current image and the past image. The smaller the correlation value SAD1, the higher the similarity between the current image and the past image, and the larger the correlation value SAD1, the lower the similarity between the current image and the past image.

[0027] The image selection unit 201 calculates the correlation value SAD1 for each of the multiple regions in the current image. When selecting the reference image and the correction target image, the image selection unit 201 uses a representative value such as an average value or a total value of the multiple correlation values ​​SAD1 corresponding to the multiple regions as a correlation value. The image selection unit 201 may calculate the sum of absolute differences of the entire image and use it when selecting the reference image and the correction target image. The region for calculating the correlation value SAD1 may be a region of one pixel, or may be a region consisting of multiple pixels. The correlation value SAD1 of one pixel is, for example, the sum of absolute differences of a region of a predetermined size (for example, a region of 5 pixels horizontally by 5 pixels vertically) centered on the pixel to which the correlation value SAD1 is to be calculated. The pattern of the multiple regions may be a fixed pattern that is determined in advance, or may not be the same. A plurality of templates corresponding to the multiple patterns may be prepared in advance in the ROM 102. The image selection unit 201 may select and use a template designated by the user or a template according to the shooting conditions from the multiple templates.

[0028] The correlation value is not limited to the sum of absolute differences, and may be other correlation values ​​such as the sum of squared differences (SSD) or normalized cross correlation (NCC). In addition, both the correlation value for each pixel and the correlation value for each region consisting of multiple pixels may be used. The reference image and the correction target image may be specified by the user. The correlation value may be any value related to the correlation between the reference image and the correction target image (for example, a value indicating the strength (weakness) of the correlation).

[0029] Furthermore, the image selection unit 201 detects an area in the correction target image where the correlation value is greater than a threshold (where the correlation with the past image is weaker than the threshold) as a tentative foreign object area (foreign object area), and outputs information on the tentative foreign object area. The threshold may be a fixed value determined in advance, or may be a value designated by the user. A plurality of thresholds may be prepared in advance in the ROM 102. Then, the image selection unit 201 may select a threshold according to the shooting conditions from the plurality of thresholds and use it.

[0030] After the above-mentioned series of processes, the image selection unit 201 selects two current images corresponding to the optical systems 104A and 104B (image circles IC1 and IC2), respectively, and a tentative image of the foreign object area. The information is output (stored) in the RAM 103 .

[0031] The phase difference acquisition unit 202 reads out the reference image and the correction target image selected by the image selection unit 201 from the RAM 103, and acquires an evaluation value (phase difference) related to the parallax with respect to the reference image for each of a plurality of regions in the correction target image. Then, the phase difference acquisition unit 202 outputs (stores) the acquired evaluation value to the RAM 103.

[0032] 3(A) and 3(B) are schematic diagrams showing an example of the distribution of pixel values ​​of a reference image and the distribution of pixel values ​​of an image to be corrected. The horizontal axis of FIG. 3(A) and 3(B) indicates the horizontal position, and the vertical axis of FIG. 3(A) and 3(B) indicates the pixel value. FIG. 3(A) shows a situation where the distance from the imaging device 100 to the subject is long (the subject in the image is out of focus). In FIG. 3(A), the distribution of pixel values ​​of the reference image and the distribution of pixel values ​​of the image to be corrected are largely offset from each other, and the parallax between the reference image and the image to be corrected is large. When the subject approaches the imaging device 100 from the situation in FIG. 3(A) (the subject is closer to the in-focus state), the offset between the distribution of pixel values ​​of the reference image and the distribution of pixel values ​​of the image to be corrected becomes smaller, as shown in FIG. 3(B), and the parallax between the reference image and the image to be corrected becomes smaller. From FIG. 3(A) and 3(B), it can be seen that the correlation between the reference image and the image to be corrected is related to the parallax between the reference image and the image to be corrected.

[0033] The phase difference acquisition unit 202 calculates the sum of absolute differences as the evaluation value (phase difference) using, for example, the following Equation 2.

number

[0034] In Equation 2, h(i,j) represents a pixel value at coordinates (horizontal position, vertical position)=(i,j) of the reference image, and k(i,j) represents a pixel value at coordinates (i,j) of the correction target image. The absolute value (absolute difference value) of the value obtained by subtracting pixel value k(i,j) from pixel value h(i,j) is calculated, and the sum of multiple absolute difference values ​​corresponding to multiple coordinates is calculated as phase difference SAD2 (sum of absolute difference values). The smaller the phase difference SAD2, the stronger the correlation between the reference image and the correction target image, and the smaller the parallax between the reference image and the correction target image. And the larger the phase difference SAD2, the weaker the correlation between the reference image and the correction target image, and the larger the parallax between the reference image and the correction target image. The phase difference SAD2 is also a value related to the similarity between the reference image and the correction target image. The smaller the phase difference SAD2, the higher the similarity between the reference image and the correction target image, and the larger the phase difference SAD2, the lower the similarity between the reference image and the correction target image.

[0035] The phase difference acquisition unit 202 calculates the phase difference SAD2 for each of a plurality of regions in the current image. The region for calculating the phase difference SAD2 may be a region of one pixel, or may be a region consisting of a plurality of pixels. The phase difference SAD2 of one pixel is, for example, the sum of absolute differences in a region of a predetermined size (for example, a region of 5 pixels in the horizontal direction by 5 pixels in the vertical direction) centered on the pixel for which the phase difference SAD2 is to be calculated. The pattern of the plurality of regions may be a fixed pattern that is determined in advance, or may not be the same. A plurality of templates corresponding to the plurality of patterns may be prepared in advance in the ROM 102. Then, the phase difference acquisition unit 202 may select and use a template designated by the user or a template according to the shooting conditions from the plurality of templates.

[0036] The evaluation value (phase difference) is not limited to the sum of absolute differences, and may be other correlation values ​​such as the sum of squared differences (SSD) or normalized cross-correlation (NCC). In addition, both an evaluation value for each pixel and an evaluation value for each region consisting of multiple pixels may be used. The calculation formula for the evaluation value used by the phase difference acquisition unit 202 may be the same as the calculation formula for the correlation value used by the image selection unit 201. The evaluation value may be any value related to the disparity of the image to be corrected with respect to the reference image (for example, a value indicating the magnitude of the disparity).

[0037] The phase difference correction unit 203 reads information on the tentative foreign substance region detected by the image selection unit 201 and the phase difference of each region acquired by the phase difference acquisition unit 202 from the RAM 103, and corrects the phase difference of each region based on the tentative foreign substance region. Then, the phase difference correction unit 203 outputs (stores) the phase difference of each region after correction to the RAM 103. For example, the phase difference correction unit 203 detects a region whose distance from the tentative foreign substance region is shorter than a threshold value and whose phase difference is larger than a threshold value (correlation between the reference image and the correction target image is weaker than the threshold value) as the final foreign substance region. Then, the phase difference correction unit 203 corrects the phase differences of the multiple regions corresponding to the foreign substance region to an evaluation value obtained by spline interpolation using the phase differences of the multiple regions corresponding to the periphery of the final foreign substance region. The distance threshold value and the phase difference threshold value may be fixed values ​​determined in advance, or may be values ​​designated by the user. A plurality of threshold values ​​may be prepared in advance in the ROM 102. The phase difference correction unit 203 may select a threshold value according to the shooting conditions from among a plurality of threshold values ​​and use the selected threshold value.

[0038] The phase difference correction method is not limited to the above method, and various known interpolation processes can be used, for example. A high-pass filter process may be applied to the reference image to detect the boundary (contour) of the subject, and the phase difference may be corrected taking into account the boundary. Information on the distance from the imaging device 100 to the subject may be obtained from the measurement unit 112, and the phase difference may be corrected taking into account the distance.

[0039] Furthermore, in the first embodiment, an area from the correction target image that satisfies all of the following three conditions 1 to 3 is detected as a foreign substance area, but the predetermined conditions for detecting a foreign substance area are not limited to conditions 1 to 3. For example, an area (an area detected by the image selection unit 201) that satisfies only condition 1 may be detected as the final foreign substance area without considering conditions 2 and 3. An area that satisfies only condition 3 may be detected as the final foreign substance area without considering conditions 1 and 2. In this case, the image selection unit 201 does not need to detect a tentative foreign substance area. Condition 1: The correlation with past images is weaker than the threshold. Condition 2: The distance from the area where the correlation with the past image is weaker than the threshold is shorter than the threshold. Condition 3: The correlation with the reference image is weaker than the threshold.

[0040] The image estimation unit 204 reads out from the RAM 103 the phase difference of each region after correction by the phase difference correction unit 203 and the reference image selected by the image selection unit 201, and generates an estimated image from the reference image based on the phase difference of each region read out. Then, the image estimation unit 204 outputs (stores) the estimated image to the RAM 103. For example, the image estimation unit 204 generates an estimated image by determining the amount of pixel shift from the phase difference and performing a process of moving the pixel (region) of the reference image by the amount of pixel shift for each pixel (each region) of the reference image. Note that the method of generating an estimated image is not limited to this, and an estimated image may be generated using, for example, the result of machine learning (trained model).

[0041] The combination ratio determination unit 205 reads out information on the tentative foreign object region detected by the image selection unit 201 and the phase difference of each region acquired by the phase difference acquisition unit 202 from the RAM 103. The combination ratio determination unit 205 detects, as the final foreign object region, a region whose distance from the tentative foreign object region is shorter than a threshold value and whose phase difference is larger than a threshold value (the correlation between the reference image and the correction target image is weaker than the threshold value) in the same manner as the phase difference correction unit 203. The combination ratio determination unit 205 may also acquire information on the final foreign object region detected by the phase difference correction unit 203. Then, the combination ratio determination unit 205 determines, for each of a plurality of regions in the correction target image, a combination ratio for combining the estimated image with the correction target image, based on the final foreign object region. The combination ratio determination unit 205 outputs (stores) the determined combination ratio for each region to the RAM 103. For example, in a foreign object region, the combination ratio (ratio of the estimated image to the correction target image) is set to 100. %, and outside the foreign object region, the blending ratio is determined so that the blending ratio changes continuously from 100% to 0% as it moves away from the foreign object region. As described above, the provisional foreign object region may be used as the final foreign object region, or a region in which the phase difference output from the phase difference acquisition unit 202 is greater than a threshold value (a region in which the correlation with the reference image is weaker than a threshold value) may be used as the final foreign object region.

[0042] The image synthesis unit 206 reads out from the RAM 103 the correction target image selected by the image selection unit 201, the estimated image generated by the image estimation unit 204, and the synthesis ratio of each region determined by the synthesis ratio determination unit 205. Then, for each of the multiple regions in the correction target image, the image synthesis unit 206 generates a synthesized image by synthesizing pixel values ​​of the correction target image and pixel values ​​of the estimated image at the synthesis ratio determined by the synthesis ratio determination unit 205. In this way, the correction target image is corrected to a synthesized image.

[0043] (Image correction processing) Fig. 4 is a flowchart showing an example of image correction processing according to embodiment 1. Figs. 5(A) to 5(E) are schematic diagrams showing an example of various data used in the image correction processing according to embodiment 1.

[0044] In step S401, the control unit 101 controls the image selection unit 201 to select one of the two images corresponding to the optical systems 104A and 104B (image circles IC1 and IC2) as a reference image, and select the other of the two images as an image to be corrected. Then, the control unit 101 advances the process to step S402. FIG. 5(A) shows an example of two images corresponding to the image circles IC1 and IC2. In FIG. 5(A), the image corresponding to the image circle IC1 does not include a foreign object, but the image corresponding to the image circle IC2 includes a foreign object X. Therefore, the image corresponding to the image circle IC1 is selected as the reference image, and the image corresponding to the image circle IC2 is selected as the image to be corrected.

[0045] In step S402, the control unit 101 controls the phase difference acquisition unit 202 to acquire an evaluation value (phase difference) related to the parallax with respect to the reference image for each of a plurality of regions in the correction target image. Then, the control unit 101 advances the process to step S403. FIG. 5B shows an example of the distribution of phase differences in shading. In FIG. 5B, the phase difference is shown in a lighter (pale) color as the phase difference increases (lighter color as the correlation between the reference image and the correction target image decreases, and lighter color as the parallax between the reference image and the correction target image increases). In FIG. 5B, the phase difference is shown in a darker color as the phase difference decreases (darker color as the correlation between the reference image and the correction target image increases, and darker color as the parallax between the reference image and the correction target image decreases). In FIG. 5B, in the region of the foreign object X, the correlation between the reference image and the correction target image is weak, so the phase difference is large (almost equal to the phase difference of the background).

[0046] In step S403, the control unit 101 controls the phase difference correction unit 203 to correct the phase difference of each region acquired in step S402. Then, the control unit 101 advances the process to step S404. Fig. 5C shows an example of the phase difference of each region after correction. In Fig. 5C, the phase difference of the region of the foreign object X is corrected to a value close to the phase difference when the foreign object X is not present.

[0047] In step S404, the control unit 101 controls the image estimation unit 204 to generate an estimated image from the reference image based on the phase difference of each region after the correction in step S403. Then, the control unit 101 advances the process to step S405. Fig. 5(D) shows an example of the estimated image. No foreign object is captured in the estimated image in Fig. 5(D).

[0048] In step S405, the control unit 101 controls the blending ratio determination unit 205 to determine a plurality of blending ratios corresponding to a plurality of regions in the correction target image, respectively, based on the foreign substance region. Then, the control unit 101 advances the process to step S406.

[0049] In step S406, the control unit 101 controls the image synthesis unit 206 to generate a synthetic image by synthesizing pixel values ​​of the correction target image and pixel values ​​of the estimated image at the synthesis ratio determined in step S405 for each of the multiple regions in the correction target image. This corrects the correction target image into a synthetic image. FIG. 5(E) shows an example of two images after correction (two images corresponding to the optical systems 104A and 104B (image circles IC1 and IC2), respectively). In FIG. 5(E), no foreign object is captured in either of the two images.

[0050] As described above, according to the first embodiment, pixel values ​​of a foreign object region in one of two images corresponding to two optical systems are corrected using pixel values ​​of the other of the two images. This makes it possible to appropriately correct an image in which a foreign object is present into an image in which no foreign object is present, thereby preventing deterioration of stereoscopic vision due to foreign objects.

[0051] Although an example in which only one foreign object exists has been described, the number of foreign objects is not particularly limited, and multiple foreign objects may exist. Also, an example in which a reference image and a correction target image are determined and the correction target image is corrected has been described, but this is not limiting. For example, when a foreign object appears in both of two images corresponding to two optical systems, one image may be used to remove the foreign object in the other image, and the other image may be used to remove the foreign object in one image.

[0052] <Embodiment 2> Hereinafter, a description will be given of the second embodiment of the present invention. Note that in the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described.

[0053] (Imaging device) The configuration of the imaging device according to the second embodiment is similar to that of the first embodiment (FIG. 1).

[0054] (Image processing unit) 7 is a block diagram showing an example of the configuration of the image processing unit 107 according to the embodiment 2. The image processing unit 107 according to the embodiment 2 has a plurality of components shown in the embodiment 1 (FIG. 2) and a view angle control unit 707.

[0055] The angle-of-view control unit 707 reads out information on the tentative foreign object region detected by the image selection unit 201 and the phase difference of each region acquired by the phase difference acquisition unit 202 from the RAM 103. The angle-of-view control unit 707 detects, as the final foreign object region, a region whose distance from the tentative foreign object region is shorter than a threshold value and whose phase difference is larger than a threshold value (correlation between the reference image and the correction target image is weaker than the threshold value) in the same manner as the phase difference correction unit 203. The angle-of-view control unit 707 may also acquire information on the final foreign object region detected by the phase difference correction unit 203. Furthermore, the angle-of-view control unit 707 reads out information on the angle of view indicating a predetermined angle of view (predetermined region) from the RAM 103, and determines an effective angle of view based on the intermediate foreign object region and the predetermined angle of view. Then, the angle-of-view control unit 707 outputs (stores) information on the determined effective angle of view to the RAM 103. The predetermined angle of view is the minimum allowable effective angle of view. The predetermined angle of view may be a fixed angle of view determined in advance, or may be an angle of view specified by a user. Information on a plurality of angles of view may be prepared in advance in the ROM 102. The angle-of-view control unit 707 may select an angle of view according to the shooting conditions from the plurality of angles of view and use the selected angle.

[0056] 8(A) to 8(C) are schematic diagrams showing an example of a method for determining an effective angle of view. In FIG. 8(A) to 8(C), for an image 801 corresponding to the optical system 104A (image circle IC1) and an image 802 corresponding to the optical system 104B (image circle IC2), , a field angle 803 (the minimum allowable effective field angle) is set in advance.

[0057] FIG. 8(A) shows a case where no foreign object is captured in either image 801 or image 802. In this case, the entire image 801 and the entire image 802 are each determined as the effective angle of view. FIG. 8(B) shows a case where foreign object X is captured inside angle of view 803. In this case, the entire image 801 and the entire image 802 are each determined as the effective angle of view. FIG. 8(C) shows a case where foreign object X is captured outside angle of view 803. In this case, angle of view 804 is determined as the effective angle of view for each of images 801 and 802. Angle of view 804 is an angle of view that includes angle of view 803 and circumscribes the area of ​​foreign object X.

[0058] 8C, the effective angle of view is not limited to angle of view 804, and may be angle of view 803, for example. The effective angle of view does not necessarily have to have the same shape as the image circle. For example, when the optical system includes a fisheye lens and a fisheye image is captured, the effective angle of view may have a shape based on an image after image processing such as equirectangular transformation or perspective projection transformation.

[0059] The image synthesis unit 206 according to the second embodiment performs the same process as in the first embodiment to generate a synthetic image. However, when generating a synthetic image, the image synthesis unit 206 according to the second embodiment reads information on the effective angle of view determined by the angle of view control unit 707 from the RAM 103, and invalidates the area outside the effective angle of view. For example, the image synthesis unit 206 invalidates the area outside the effective angle of view by replacing pixel values ​​of the area outside the effective angle of view with a predetermined pixel value (e.g., a black pixel value) or by trimming the area outside the effective angle of view.

[0060] (Image correction processing) Fig. 9 is a flowchart showing an example of image correction processing according to embodiment 2. Figs. 10(A) to 10(E) are schematic diagrams showing an example of various data used in the image correction processing according to embodiment 2.

[0061] Steps S901 to S905 are similar to steps S401 to S405 in the first embodiment (FIG. 4).

[0062] FIG. 10A shows an example of two images corresponding to optical systems 104A and 104B (image circles IC1 and IC2), respectively. In FIG. 10A, the image corresponding to image circle IC1 does not show any foreign matter, but the image corresponding to image circle IC2 shows three foreign matters X1 to X3. Therefore, in step S901, the image corresponding to image circle IC1 is selected as the reference image, and the image corresponding to image circle IC2 is selected as the image to be corrected. In FIG. 10A, a field angle 1003 (the minimum allowable effective field angle) is preset for each of the reference image and the image to be corrected.

[0063] In step S902, for example, the phase difference shown in FIG. 10B is acquired. FIG. 10B shows an example of the distribution of phase differences in shading. In FIG. 10B, the phase difference is shown in a lighter (pale) color as the phase difference increases (lighter color as the correlation between the reference image and the image to be corrected decreases, and lighter color as the parallax between the reference image and the image to be corrected increases). The phase difference is shown in a darker color as the phase difference decreases (darker color as the correlation between the reference image and the image to be corrected increases, and darker color as the parallax between the reference image and the image to be corrected decreases). In FIG. 10B, in the regions of the foreign objects X1 to X3, the correlation between the reference image and the image to be corrected is weak, and therefore the phase difference is large (almost equal to the phase difference of the background).

[0064] In step S903, for example, the phase difference shown in FIG. 10B is corrected to the phase difference shown in FIG. 10C. FIG. 10C shows an example of the phase difference of each region after correction. In the example, the phase difference in the region of the foreign objects X1 to X3 is corrected to a value close to the phase difference when the foreign objects X1 to X3 are not present.

[0065] In step S904, for example, an estimated image shown in Fig. 10(D) is generated. Fig. 10(D) shows an example of the estimated image. No foreign object is captured in the estimated image in Fig. 10(D).

[0066] After step S905, in step S906, the control unit 101 controls the angle-of-view control unit 707 to determine the effective angle of view. In Fig. 10A, the foreign object X1 is captured inside the angle of view 1003 (the minimum allowable effective angle of view), and the foreign objects X2 and X3 are captured within the angle of view 1003. In this case, as shown in Fig. 10B, a circular angle of view 1004 that is centered at the center of the correction target image and that is tangent to the coordinates closest to the angle of view 1003 among the multiple coordinates in the areas of the foreign objects X2 and X3 is determined as the effective angle of view.

[0067] In step S907, the control unit 101 controls the image synthesis unit 206 to generate a synthetic image by performing the same process as in the first embodiment and invalidating the area outside the effective angle of view. FIG. 10(E) shows an example of two images after correction (two images corresponding to the optical systems 104A and 104B (image circles IC1 and IC2), respectively). In FIG. 10(E), no foreign object is captured in either of the two images. Also, in FIG. 10(E), the area outside the effective angle of view, 1004, is invalidated.

[0068] The process of step S906 may be performed before any of the processes of steps S902 to S904. In steps S902 to S904, the processes may be performed only on the inside of the effective angle of view. This can reduce the processing load. The process of step S906 may be performed in parallel with the processes of steps S902 to S904.

[0069] As described above, according to the second embodiment, when a foreign object area is detected outside the predetermined area, the correction target image is corrected to an image in which the foreign object area is invalid. This also makes it possible to correct an image in which a foreign object is present to an image in which no foreign object is present. Inside the predetermined area, the foreign object is removed by the method of the first embodiment, so that deterioration of stereoscopic vision can be suppressed.

[0070] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0071] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0072] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) An acquisition means for acquiring a first image captured via a first optical system and a second image captured via a second optical system; a detection means for detecting an area satisfying a predetermined condition as a foreign object area from one of the first image and the second image; The pixel values ​​of the foreign object region in the one image are compared with the pixel values ​​of the first image and the second image. A correction means for correcting the image by using pixel values ​​of the other image. 13. An image processing device comprising: (Configuration 2) a selection means for selecting, as the one image, one of the first image and the second image, which has a weaker correlation with a past image; Further having 2. The image processing device according to claim 1, (Configuration 3) The predetermined condition includes a condition that the correlation with a past image is weaker than a threshold value. 3. The image processing device according to configuration 1 or 2. (Configuration 4) The predetermined condition includes a condition that the distance from a region in which the correlation with the past image is weaker than a threshold value is shorter than a threshold value. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 5) The predetermined condition includes a condition that the correlation with the other image is weaker than a threshold value. 5. The image processing device according to any one of configurations 1 to 4. (Configuration 6) The correction means is a generating means for generating a third image from the other image; a synthesis means for correcting the one image to an image obtained by synthesizing pixel values ​​of the one image and pixel values ​​of the third image at a synthesis ratio based on the foreign substance region for each of a plurality of regions in the one image; have 6. The image processing device according to any one of configurations 1 to 5. (Configuration 7) a second acquiring means for acquiring an evaluation value associated with a parallax with respect to the other image for each of a plurality of regions in the one image; and The generating means generates a third image from the other image based on a plurality of evaluation values ​​respectively corresponding to the plurality of regions. 7. The image processing device according to configuration 6, (Configuration 8) The generating means corrects the evaluation value corresponding to the foreign substance region using evaluation values ​​corresponding to the surroundings of the foreign substance region, and generates a third image from the other image based on the plurality of evaluation values ​​after the correction. 8. The image processing device according to configuration 7. (Configuration 9) The evaluation value indicates a weak correlation between the first image and the second image. 9. The image processing device according to configuration 7 or 8. (Configuration 10) When the foreign object region is detected outside a predetermined region, the correction means corrects the one image to an image in which the foreign object region is invalid. 10. The image processing device according to any one of configurations 1 to 9. (Configuration 11) The predetermined area is an area designated by a user. 11. The image processing device according to configuration 10. (Configuration 12) When the foreign object region is detected outside the predetermined region, the correction means corrects the one image to an image in which the outside of the predetermined region is invalid. 12. The image processing device according to claim 10 or 11, (Configuration 13) When the foreign substance region is detected outside the predetermined region, the correction means corrects the one image to an image including the predetermined region and having an invalid area outside a region circumscribing the foreign substance region. 12. The image processing device according to claim 10 or 11, (Configuration 14) When the foreign substance region is detected outside the predetermined region, the correction means invalidates the foreign substance region by replacing pixel values ​​of the foreign substance region with predetermined pixel values. 14. The image processing device according to any one of configurations 10 to 13. (Configuration 15) When the foreign object region is detected outside the predetermined region, the correction means invalidates the foreign object region by trimming the foreign object region. 14. The image processing device according to any one of configurations 10 to 13. (method) Obtaining a first image captured via a first optical system and a second image captured via a second optical system; detecting a region that satisfies a predetermined condition as a foreign object region from one of the first image and the second image; correcting a pixel value of the foreign substance region in the one image using a pixel value of the other image of the first image and the second image; 13. An image processing method comprising: (program) A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 15. (medium) 16. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0073] 100: Imaging device 105: Imaging section 107: Image processing section

Claims

1. Acquisition means for acquiring a first image captured through a first optical system and a second image captured through a second optical system, A detection means for detecting a region satisfying predetermined conditions as a foreign object region from one of the first image and the second image, Correction means for correcting the pixel value of the foreign object region in one of the images using the pixel value of the other image between the first image and the second image, A selection means for selecting the image with the weaker correlation to past images from the first image and the second image as one of the images. An image processing apparatus characterized by having

2. The aforementioned predetermined conditions include the condition that the correlation with past images is weaker than a threshold. The image processing apparatus according to feature 1.

3. The aforementioned predetermined conditions include the condition that the distance from a region where the correlation with past images is weaker than a threshold is shorter than a threshold. The image processing apparatus according to feature 1.

4. The aforementioned predetermined condition includes the condition that the correlation with the other image is weaker than a threshold. The image processing apparatus according to feature 1.

5. The correction means is A generation means for generating a third image from the other image, A synthesis means for correcting the first image is provided to obtain an image obtained by combining the first image, its pixel values, and the pixel values ​​of the third image at a synthesis ratio based on the foreign matter region for each of the multiple regions in the first image. has The image processing apparatus according to feature 1.

6. For each of the multiple regions in the one image, the parallax relative to the other image. Second acquisition means for obtaining evaluation values ​​related to It further possesses, The generation means generates a third image from the other image based on a plurality of evaluation values ​​corresponding to each of the plurality of regions. The image processing apparatus according to feature 5.

7. The generation means corrects the evaluation value corresponding to the foreign object region using the evaluation value corresponding to the area around the foreign object region, and generates a third image from the other image based on the plurality of evaluation values ​​after the correction. The image processing apparatus according to claim 6.

8. The aforementioned evaluation value indicates the weakness of the correlation between the first image and the second image. The image processing apparatus according to claim 6.

9. If the foreign matter region is detected outside the predetermined region, the correction means corrects one of the images so that the foreign matter region is invalid. The image processing apparatus according to feature 1.

10. The aforementioned predetermined area is an area specified by the user. The image processing apparatus according to feature 9.

11. If the foreign matter region is detected outside the predetermined region, the correction means corrects one of the images so that the area outside the predetermined region is invalid. The image processing apparatus according to feature 9.

12. If the foreign matter region is detected outside the predetermined region, the correction means corrects one of the images so that the area outside the region that includes the predetermined region and is tangent to the foreign matter region is invalid. The image processing apparatus according to feature 9.

13. If the foreign matter region is detected outside the predetermined region, the correction means invalidates the foreign matter region by replacing the pixel value of the foreign matter region with a predetermined pixel value. The image processing apparatus according to feature 9.

14. If the foreign matter region is detected outside the predetermined region, the correction means invalidates the foreign matter region by trimming it. The image processing apparatus according to feature 9.

15. The steps include acquiring a first image captured through a first optical system and a second image captured through a second optical system, A step of detecting a region that satisfies predetermined conditions as a foreign object region from one of the first image and the second image, The steps include correcting the pixel values ​​of the foreign object region in one of the images using the pixel values ​​of the other image between the first and second images, A selection step in which one of the first and second images is selected as the image with the weaker correlation to past images. An image processing method characterized by having the following features.

16. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 14.

Images