Information processing apparatus, image capturing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To provide an information processing apparatus, an imaging apparatus, a control method of the information processing apparatus, and a program, capable of improving detection accuracy of reflection against a photographer's intention.SOLUTION: A similarity acquiring unit 103 that acquires an index value SAD indicating a degree of similarity between a left image and a right image for each pair of reflection regions formed by two reflection regions located at corresponding positions in the left image and the right image of a captured image having parallax, a distance information calculating unit 101 that acquires, for each reflection region, distance information indicating a distance from an image capturing unit 100 that has acquired the captured image to a subject in the reflection region, and a threshold value calculating unit 104 that acquires a threshold value Th for each pair of reflection regions in accordance with the positions of the pair of reflection regions in the left image and the right image and the distance information; An image processing apparatus includes a determination unit 105 for performing reflection determination for each reflection area pair by using an index value SAD and a threshold Th, and a reflection processing unit 106 for performing processing related to reflection based on a result of the reflection determination.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an imaging device, a control method for an information processing device, and a program. [Background technology]

[0002] When taking pictures with a compound eye camera having two lenses arranged side by side, there have been known techniques for preventing parts of the photographer's body, such as hands or head, or parts of the photographer's clothing, such as a hat, from appearing in the picture against the photographer's intention. For example, Patent Document 1 discloses a technique for determining and notifying the photographer of the occurrence of an unintended object being captured based on the difference in brightness distribution between two captured images, one on the left and one on the right. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4957850 Summary of the Invention [Problem to be solved by the invention]

[0004] There are two types of unwanted reflections. The first type occurs when an object is close enough to the camera to touch it (hereinafter referred to as "Type A reflection"). The second type occurs when an object is at a certain distance from the camera (hereinafter referred to as "Type B reflection"). Type A reflection occurs, for example, when the photographer's fingers touch either the left or right lens surface while holding the camera. Type B reflection occurs, for example, when the photographer leans forward and part of the photographer's head or part of the hat they are wearing overlaps the lens from above. Type B reflection also occurs when the photographer's hand, which is away from the camera, is captured somewhere in front of the lens. When taking ultra-wide-angle shots using a fisheye lens or the like, Type B reflections are more likely to occur due to the wide angle of view.

[0005] Generally, as described above, Type A reflection occurs when an object comes into contact with either the left or right lens surface. For this reason, Type A reflection results in a large difference between the two captured images obtained through the left and right lenses (hereinafter referred to as "between the left and right captured images"). In this regard, the technology disclosed in Patent Document 1, as described above, determines the occurrence of reflection that is contrary to the photographer's intention based on the difference in luminance distribution between the two captured images (left and right), and therefore is able to detect Type A reflection with relatively high accuracy.

[0006] On the other hand, when the object is at a certain distance from the camera, the object appears in both the left and right captured images, so the difference between the left and right captured images is often smaller than when the object appears in only one of the left and right captured images. This tendency is more pronounced as the distance from the camera to the object increases. Furthermore, when the object is at a certain distance from the camera, the difference between the left and right captured images is less likely to appear when the object appears near the vertical edge of the captured image than when the object appears near the horizontal edge of the captured image due to the lenses being lined up on the left and right.

[0007] However, the technology disclosed in Patent Document 1 does not fully consider the relationship between the position of such a reflected object in the captured image, the distance to the camera, and the difference that appears between the left and right captured images. As a result, there was a problem that for Type B reflections, sufficient detection accuracy could not be achieved depending on the position of the reflected object in the captured image and the distance to the camera.

[0008] The present invention has been made in view of the above-mentioned problems, and aims to provide an information processing device, an imaging device, a control method for an information processing device, and a program that can improve the accuracy of detecting unintended objects captured by a photographer. [Means for solving the problem]

[0009] In order to achieve the above object, the information processing device of the present invention is characterized by comprising: a first acquisition means for acquiring, for each reflection area pair made up of two reflection areas located at corresponding positions in the left and right images of a captured image having parallax, a first similarity indicating the degree of similarity between the left and right images of the captured image; a second acquisition means for acquiring, for each of the plurality of reflection areas, distance information indicating the distance from the imaging unit that acquired the captured image to a subject in the reflection area; a third acquisition means for acquiring a threshold for each reflection area pair in accordance with the position of the reflection area pair in the left and right images of the captured image and the distance information; a determination means for performing a reflection determination for each reflection area pair to determine whether reflection has occurred in the captured image using the first similarity and the threshold; and a reflection processing means for performing processing related to the reflection based on the result of the reflection determination. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the accuracy of detecting unwanted objects that are not intended by the photographer. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the configuration of an imaging device 1 including an information processing device according to the present invention. [Figure 2] 1 is a diagram showing an example of a digital image signal output from an imaging unit 100 divided into regions. [Figure 3] 4 is a flowchart showing the overall processing of the imaging device 1 of the first embodiment. [Figure 4] 1 is a schematic diagram showing an example of the configuration of an imaging unit 100. FIG. [Figure 5] 10 is a flowchart showing the processing of a distance information calculation unit 101. [Figure 6] 10 is a flowchart showing the processing of a threshold calculation unit 104. [Figure 7] 1A and 1B are diagrams showing an example of the positional relationship between the imaging device 1 and a reflected object, and an example of an image captured in that positional relationship. [Figure 8] 1A and 1B are diagrams showing an example of the positional relationship between the imaging device 1 and a reflected object, and an example of an image captured in that positional relationship. [Figure 9] 1A and 1B are diagrams showing an example of the positional relationship between the imaging device 1 and a reflected object, and an example of an image captured in that positional relationship. [Figure 10] 5 is a flowchart showing the process of a determination unit 105 in the first embodiment. [Figure 11] 10A and 10B are diagrams showing examples of processing of a captured image instructed by a reflection processing unit 106. FIG. [Figure 12] 10 is a flowchart showing the overall processing of the imaging device 1 of the second embodiment. [Figure 13A] 10 is a flowchart showing the process of a determination unit 105 in the second embodiment. [Figure 13B] 10 is a flowchart showing a first reflection determination process. [Figure 13C] 10 is a flowchart showing a second reflection determination process. DETAILED DESCRIPTION OF THE INVENTION

[0012] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in each embodiment. For example, each component constituting the present invention can be replaced with any configuration that can perform the same function. Also, any component may be added. Furthermore, any two or more configurations (features) of each embodiment can be combined.

[0013] First Embodiment The first embodiment will be described below with reference to Figs. 1 to 11. Fig. 1 is a block diagram showing the configuration of an imaging device 1 including an information processing device according to the present invention. The imaging device 1 has an imaging section 100, a distance information calculation section 101, a development processing section 102, a similarity acquisition section 103, a threshold calculation section 104, a determination section 105, a reflection processing section 106, a display image processing section 107, and a display device 108. The imaging device 1 further has a recording image processing section 109, a recording medium 110, and an output image processing section 111.

[0014] In the first embodiment, the imaging device 1 is implemented as a digital camera, but it can also be implemented as, for example, a digital video camera. That is, the captured image that is the target of the functions of each block of the imaging device 1 may be a still image or a video. Furthermore, the information processing device according to the present invention is not limited to those included in the imaging device 1. Therefore, examples of the information processing device include a personal computer, a mobile phone, a smartphone, a PDA, a tablet terminal, and the like. Furthermore, the personal computer, mobile phone, smartphone, PDA, tablet terminal, and the like may or may not have a camera.

[0015] The functions of each block of the imaging device 1 are realized by software or hardware, except for physical devices such as the imaging unit 100, display device 108, and recording medium 110. For example, the functions of each block may be realized by hardware such as a dedicated device, a logic circuit, and a memory. The functions of each block may also be realized by computer control such as a memory, a processing program stored in the memory, and a CPU that executes the processing program. Note that the following description of each flowchart will be given on the assumption that the function of each block is realized by the CPU of the imaging device 1 executing the processing program stored in the memory.

[0016] The imaging unit 100 has an imaging element. The imaging unit 100 photoelectrically converts an optical image formed on the imaging element via an optical system, and outputs a digital image signal (i.e., RAW data) by A / D converting the analog signal obtained by the optical conversion. The imaging unit 100 also calculates and outputs distance information for each pixel in the imaging element (i.e., information on the distance from the imaging unit 100 to the subject). The digital image signal is supplied from the imaging unit 100 to a development processing unit 102. The distance information for each pixel is supplied from the imaging unit 100 to a distance information calculation unit 101. The distance information calculation unit 101 (second acquisition means) calculates and outputs distance information for each of a plurality of reflection areas (described later) based on the distance information for each pixel supplied from the imaging unit 100. The distance information for each reflection area is supplied from the distance information calculation unit 101 to a threshold calculation unit 104 and a determination unit 105.

[0017] The development processing unit 102 generates developed image data by performing development processing on the digital image signal (RAW data) supplied from the imaging unit 100. Specifically, the development processing unit 102 generates developed image data expressed in luminance and color difference, such as YCbCr4:2:2, by performing color interpolation processing called DeBayer (demosaic) processing, color space conversion processing, noise suppression processing, optical distortion correction, enlargement / reduction processing, etc. The developed image data is supplied from the development processing unit 102 to the similarity acquisition unit 103, display image processing unit 107, recording image processing unit 109, and output image processing unit 111.

[0018] Based on the developed image data supplied from the development processing unit 102, the similarity acquisition unit 103 (first acquisition means) acquires and outputs a similarity indicating the degree of similarity between the captured images for each reflection area pair made up of two reflection areas at corresponding positions in the left and right images. The similarity for each reflection area pair is supplied from the similarity acquisition unit 103 to the determination unit 105. The threshold calculation unit 104 (third acquisition means) calculates and outputs a threshold for each reflection area pair to be used in comparison with the similarity acquired by the similarity acquisition unit 103, depending on the distance information of each reflection area etc. supplied from the distance information calculation unit 101. The threshold for each reflection area pair is supplied from the threshold calculation unit 104 to the determination unit 105.

[0019] The determination unit 105 (determination means) determines whether or not reflection occurs for each reflection area pair using the distance information supplied from the distance information calculation unit 101, the similarity supplied from the similarity acquisition unit 103, and the threshold supplied from the threshold calculation unit 104. The result of the determination by the determination unit 105 for each reflection area pair (hereinafter referred to as the "determination result") is output from the determination unit 105 and supplied from the determination unit 105 to the reflection processing unit 106. The reflection processing unit 106 (reflection processing means) generates instruction information for processing the captured image in accordance with the determination result supplied from the determination unit 105, and outputs the generated instruction information for processing the captured image. The instruction information for processing the captured image is supplied from the reflection processing unit 106 to the display image processing unit 107, the recording image processing unit 109, and the output image processing unit 111.

[0020] The display image processing unit 107 processes the developed image data supplied from the development processing unit 102 based on instruction information related to processing of the captured image supplied from the reflection processing unit 106. Thereafter, the display image processing unit 107 generates display image data by executing image processing such as gamma conversion processing, color space conversion processing, enlargement / reduction processing, and processing to fit into the I / F of the display device 108, and outputs the generated display image data. The display image data is supplied from the display image processing unit 107 to the display device 108. The display device 108 is used to check live view images and captured images, and is composed of an electronic viewfinder, a liquid crystal panel, etc. The display device 108 displays the display image data supplied from the display image processing unit 107.

[0021] The recording image processing unit 109 processes the developed image data supplied from the development processing unit 102 based on instruction information related to processing of the captured image supplied from the reflection processing unit 106. Thereafter, the recording image processing unit 109 performs compression processing and records the recording image data generated by the compression processing on the recording medium 110. At this time, if the recording image data is a still image, the recording image processing unit 109 compresses the recording image data by, for example, JPEG compression or HEIF compression. Furthermore, if the recording image data is a moving image, the recording image processing unit 109 compresses the recording image data by, for example, MPEG-2, H.264, H.265, etc. The recording medium 110 is a medium for recording captured images, and is composed of a magnetic tape, an optical disc, a semiconductor memory, etc.

[0022] The output image processing unit 111 processes the developed image data supplied from the development processing unit 102 based on instruction information related to processing of the captured image supplied from the reflection processing unit 106. Thereafter, the output image processing unit 111 generates image output data by executing image processing such as gamma conversion processing, color space conversion processing, enlargement / reduction processing, and processing to match with an external I / F such as SDI or HDMI (registered trademark), and outputs the generated image output data to the external I / F.

[0023] Next, the region division and reflection region of a captured image will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of region division of a digital image signal output from the imaging unit 100. When two lenses arranged side by side are used, the subject image that passes through the right optical system is formed in the region on the left side of the image sensor when viewed from the lens side, and the subject image that passes through the left optical system is formed in the region on the right side of the image sensor when viewed from the lens side. Furthermore, the formed subject image is inverted vertically and horizontally. In this embodiment, for ease of explanation, the image generated by the image sensor is shown rotated 180 degrees.

[0024] The invalid pixel area 200 is the shaded area in FIG. 2, and is an area that is not exposed to light, and therefore is essentially composed of invalid pixels. When a twin lens is attached to a typical interchangeable lens camera, the invalid pixel area 200 is composed of normal imaging pixels. The left-eye effective pixel area 201L is an area of ​​pixels that constitute a left image obtained through the left optical system, and corresponds to the left circle of the open circles in FIG. 2. Similarly, the right-eye effective pixel area 201R is an area of ​​pixels that constitute a right image obtained through the right optical system, and corresponds to the right circle of the open circles in FIG. 2. Since the first embodiment is an example using a fisheye lens, both images are fisheye images. Therefore, the left-eye effective pixel area 201L and the right-eye effective pixel area 201R form circular images.

[0025] In the left eye effective pixel region 201L and the right eye effective pixel region 201R, the areas where parts of the photographer's body, such as their hands or head, or parts of the photographer's clothing, such as their hat, are likely to be captured are generally near the edges of the regions, i.e., along the circular peripheries of the left and right images. Therefore, in the first embodiment, these peripheries are designated as the capture area, and are further divided into four regions surrounded by dotted lines. That is, in the left eye effective pixel region 201L, the area surrounded by the dotted line at the left edge is designated as the left edge capture area 202L, and the area surrounded by the dotted line at the right edge is designated as the right edge capture area 203L.

[0026] In the left-eye effective pixel region 201L, the region surrounded by a dotted line at the top is referred to as an upper-edge reflection region 204L, and the region surrounded by a dotted line at the bottom is referred to as a lower-edge reflection region 205L. Similarly, in the right-eye effective pixel region 201R, the regions surrounded by dotted lines at the left, right, top, and bottom ends are referred to as a left-edge reflection region 202R, a right-edge reflection region 203R, a top-edge reflection region 204R, and a bottom-edge reflection region 205R, respectively. Note that, hereinafter, the left and right ends are collectively referred to as the "horizontal ends." The top and bottom ends are collectively referred to as the "vertical ends."

[0027] Next, the overall processing flow of the imaging device 1 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the overall processing of the imaging device 1. The overall processing of the imaging device 1 shown in Fig. 3 (a control method for an information processing device) is realized by the CPU of the imaging device 1 expanding a program stored in the ROM into the RAM, executing it, and controlling each unit of the imaging device 1. This also applies to the processing shown in each flowchart described later. When the overall processing of the imaging device 1 shown in Fig. 3 starts, in step S300, the CPU of the imaging device 1 performs imaging processing using the imaging unit 100. This will be described in detail in the description of Fig. 4 described later. In step S301, the CPU of the imaging device 1 causes the development processing unit 102 to perform development processing on the digital image signal that is the output of step S300.

[0028] In step S302, the CPU of the imaging device 1 acquires a similarity for each reflected area pair using the similarity acquisition unit 103 based on the developed image data output in step S301 (first acquisition step). This will be described in detail later. In step S303, the CPU of the imaging device 1 calculates distance information for each reflected area using the distance information calculation unit 101 based on the distance information for each pixel output in step S300 (second acquisition step). This will be described in detail later in the description of FIG. 5. In step S304, the CPU of the imaging device 1 calculates a threshold for each reflected area pair using the threshold calculation unit 104, based on the distance information for each reflected area output in step S303, etc. (third acquisition step). This will be described in detail later in the description of FIG. 6.

[0029] In step S305, the CPU of the imaging device 1 performs a reflection determination for each reflection area pair using the determination unit 105 (determination step). At this time, the determination unit 105 uses the similarity output from step S302, the distance information output from step S303, and the threshold value output from step S304. A detailed description of this will be given in the description of FIG. 10, which will be described later. In step S306, the CPU of the imaging device 1 performs processing related to reflection using the reflection processing unit 106 (reflection processing step). In the reflection processing, instruction information for processing the captured image is generated according to the determination result for each reflection area pair, which is the output of step S305.

[0030] In step S307, the CPU of the imaging device 1 performs display image processing using the display image processing unit 107. In the display image processing, the developed image data output in step S301 is processed based on the instruction information related to processing of the captured image output in step S306. In the display image processing, predetermined image processing is further performed to generate display image data. In step S308, the CPU of the imaging device 1 performs record image processing using the record image processing unit 109. In the record image processing, the developed image data output in step S301 is processed based on the instruction information related to processing of the captured image output in step S306. In the record image processing, compression processing is further performed to generate record image data, and the generated record image data is recorded on the recording medium 110.

[0031] In step S309, the CPU of the imaging device 1 performs output image processing using the output image processing unit 111. In the output image processing, the developed image data output in step S301 is processed based on the instruction information related to processing of the captured image output in step S306. In the output image processing, predetermined image processing is further performed to generate image output data, which is output to the outside via the external I / F. In step S310, the CPU of the imaging device 1 displays the display image data output in step S307 on the display device 108. When the CPU of the imaging device 1 has completed the processing of all steps, it ends the overall processing of the imaging device 1 shown in FIG. 3.

[0032] Next, the configuration and processing contents of the imaging unit 100 will be described in detail using FIG. 4. FIG. 4 is a schematic diagram showing an example of the configuration of the imaging unit 100. The lens unit 400 has two optical systems, one on the left and one on the right, with different optical axis positions. These two optical systems are capable of forming images with parallax corresponding to left and right images. Each of these two optical systems is composed of multiple lenses and multiple reflecting mirrors or multiple prisms. Furthermore, each of these two optical systems is composed of a fisheye lens, and is capable of 180-degree wide-angle imaging in the horizontal, vertical, and diagonal directions.

[0033] The left and right images formed via the lens unit 400 are irradiated onto the image sensor 401. The image sensor 401 is composed of multiple elements that photoelectrically convert incident light. Each element photoelectrically converts the incident light to output an analog electrical signal for a unit pixel. Each element also has a microlens and two photoelectric conversion units, and has a pupil division structure (not shown), in order to calculate distance information for each pixel, which will be described later. The A / D conversion unit 402 converts the analog electrical signal supplied from the image sensor 401 into a digital image signal and outputs the converted digital image signal.

[0034] The distance measurement calculation unit 403 calculates distance information for each pixel using the digital image signal supplied from the A / D conversion unit 402, and outputs the calculated distance information for each pixel. For example, pupil division phase difference detection is used to calculate the distance information for each pixel. In pupil division phase difference detection, an image sensor 401 having a microlens and two photoelectric conversion units is used for each pixel. The image sensor 401 performs pupil division using the two photoelectric conversion units, detects the phase difference between the peaks of the signal levels of the same object, and can calculate the distance to the object using the detected phase difference based on the principle of triangulation.

[0035] Next, the processing flow of distance information calculation unit 101 will be described in detail with reference to Fig. 5. Fig. 5 is a flowchart showing the processing of distance information calculation unit 101. When the processing of distance information calculation unit 101 shown in Fig. 5 starts, in step S500, the CPU of imaging device 1 causes distance information calculation unit 101 to distribute the distance information for each pixel supplied from imaging unit 100 to each captured area. At this time, the distance information for each pixel is supplied from imaging unit 100 in a predetermined order. Therefore, distance information calculation unit 101 can determine, from the order in which the distance information for each pixel supplied from imaging unit 100 is supplied from imaging unit 100, to which pixel in the captured image the distance information corresponds.

[0036] As a result, in step S500, the distance information calculation unit 101 determines, based on the order in which the distance information for each pixel supplied from the image capture unit 100 was supplied from the image capture unit 100, to which of the reflected areas the distance information belongs. Furthermore, the distance information calculation unit 101 classifies the distance information for each pixel supplied from the image capture unit 100 as distance information for the corresponding reflected area based on the result of the above-mentioned determination. At this time, the distance information calculation unit 101 discards distance information that does not belong to any of the reflected areas. In step S501, the CPU of the image capture device 1 selects, by the distance information calculation unit 101, a reflected area to be subjected to the next process. In the first embodiment, the reflected area is selected one by one in order from the eight reflected areas, namely, the left edge reflected area 202L to the bottom edge reflected area 205L of the left image and the left edge reflected area 202R to the bottom edge reflected area 205R of the right image.

[0037] In step S502, the CPU of the imaging device 1 uses the distance information assigned to the captured image area selected in S501 to generate a representative distance information value for the captured image area using the distance information assigned to the captured image area by the distance information calculation unit 101. Specifically, the distance information calculation unit 101 sets the distance information that indicates the smallest distance among all the distance information assigned to the selected captured image area as the representative distance information value for the captured image area. The reason why the representative distance information value for the captured image area is generated in this manner is that, in most cases, the capture of a part of the photographer's body, such as the photographer's hands or head, or a part of the photographer's clothing, such as a hat, occurs at an extremely close distance to the lens.

[0038] In step S503, the CPU of imaging device 1 determines whether distance information calculation unit 101 has completed generating distance information representative values ​​for all reflected areas. If the CPU of imaging device 1 determines that distance information calculation unit 101 has not completed generating distance information representative values ​​for all reflected areas, the process returns to step S501. On the other hand, if the CPU of imaging device 1 determines that distance information calculation unit 101 has completed generating distance information representative values ​​for all reflected areas, the process proceeds to step S504.

[0039] In step S504, the CPU of imaging device 1 outputs the distance information representative value for each reflected area generated by distance information calculation unit 101 to threshold calculation unit 104 and determination unit 105 as distance information for each reflected area. As described above, the processes of steps S501 to S503 are repeated until generation of distance information representative values ​​for all reflected areas is completed, and the distance information representative value for each reflected area is output as distance information for each reflected area. Thereafter, the process of distance information calculation unit 101 shown in FIG. 5 ends.

[0040] Next, the processing content of the similarity acquisition unit 103 will be described in detail. In the first embodiment, the similarity acquisition unit 103 acquires only the luminance component of the developed image data supplied from the development processing unit 102, and uses the acquired luminance component for processing. Furthermore, the similarity acquisition unit 103 uses an index value of SAD (Sum of Absolute Difference) (hereinafter referred to as "index value SAD") as the similarity. The index value SAD (first similarity) of two images A and B can be calculated using the calculation formula shown in Equation (1).

[0041] SAD = Σ|A(i)-B(i)| (1) In equation (1), i represents the pixel position. A(i) and B(i) represent pixel values. Σ represents the sum with i as the subscript. | represents the absolute value of the elements enclosed within it. The larger the index value SAD calculated by equation (1), the greater the difference between the two images A and B. On the other hand, the smaller the index value SAD calculated by equation (1), the smaller the difference between the two images A and B, and the higher the similarity.

[0042] The similarity acquisition unit 103 first pairs reflection areas at corresponding positions in the left and right images. Specifically, the similarity acquisition unit 103 pairs the left edge reflection area 202L of the left image with the left edge reflection area 202R of the right image, and pairs the right edge reflection area 203L of the left image with the right edge reflection area 203R of the right image. Similarly, the similarity acquisition unit 103 pairs the top edge reflection area 204L of the left image with the top edge reflection area 204R of the right image, and pairs the bottom edge reflection area 205L of the left image with the bottom edge reflection area 205R of the right image. These four pairs correspond to the reflection area pairs described above.

[0043] Next, the similarity obtaining unit 103 calculates the index value SAD for each pair of reflection areas using the above-mentioned formula (1). Note that in the first embodiment, the index value SAD is used as the similarity as described above, but this is not limiting. For example, index values ​​such as SSD (Sum of Squared Difference) or NCC (Normalized Cross-Correlation) may be used as the similarity.

[0044] Next, the processing flow of the threshold calculation unit 104 will be described in detail with reference to FIG. 6. FIG. 6 is a flowchart showing the processing of the threshold calculation unit 104. In the processing of the threshold calculation unit 104 shown in FIG. 6, a threshold is calculated for each reflected area pair, and the processing is repeated until the calculation of thresholds for all reflected area pairs is completed. When the processing of the threshold calculation unit 104 shown in FIG. 6 starts, in step S600, the CPU of the imaging device 1 initializes the threshold value Th using the threshold calculation unit 104. The initial value used to initialize the threshold value Th may be any value suitable as a threshold when assuming that the distance between the reflected object and the lens is a predetermined reference distance D. For example, the smallest index value SAD obtained when generating reflections at the reference distance D multiple times through experiments or the like is predetermined as the initial value. In the first embodiment, the initial value is set to a value suitable as a threshold when assuming that the distance between the reflected object and the lens is the reference distance D and that the reflected object is located at the horizontal edge of the captured image.

[0045] In step S601, the CPU of the imaging device 1 selects a capture area pair to be processed next using the threshold calculation unit 104. In the first embodiment, it is assumed that each of the four capture area pairs described above is selected in turn. In step S602, the CPU of the imaging device 1 generates a distance information representative value d for the capture area pair currently being processed using the threshold calculation unit 104. The threshold calculation unit 104 generates the shorter distance from the distance information of the two capture area pair areas that make up the capture area pair currently being processed as the distance information representative value d for the capture area pair currently being processed.

[0046] Here, using FIGS. 7, 8, and 9, we will explain how a reflected object appears in a captured image and the positional relationship between the reflected object and the imaging device 1. FIGS. 7, 8, and 9 are diagrams showing the positional relationship between the imaging device 1 and the reflected object, and an example of a captured image in that positional relationship. FIGS. 7(a), 8(a), and 9(a) are diagrams showing the positional relationship between the imaging device 1 and the reflected object from above the imaging device 1. FIGS. 7(b), 8(b), and 9(b) are diagrams showing the positional relationship between the imaging device 1 and the reflected object from the right of the imaging device 1. FIG. 7(c) is a diagram showing an example of a captured image in the positional relationship between the imaging device 1 and the reflected object shown in FIGS. 7(a) and 7(b). In FIG. 7(c), each reflected area is indicated by a dotted line, as in FIG. 2. The same applies to FIGS. 8(c) and 9(c).

[0047] When a reflected object is close enough to touch the imaging device 1, as described above, the reflected object, such as the photographer's finger, will come into contact with either the left or right lens surface, resulting in a large difference between the left and right captured images. For example, if a reflected object is at position 701 shown in Figures 7(a) and 7(b), the reflected object will appear at position 702 shown in Figure 7(c), i.e., at the right end of the right image. In this case, the reflected object will appear only in the right image, so it can be said that a difference between the left and right captured images is relatively likely to appear.

[0048] In contrast, if the reflected object is at a certain distance from the imaging device 1, the reflected object will appear in both the left and right captured images. Therefore, in this case, the difference between the left and right captured images is often smaller than when the reflected object appears in only one of the left and right captured images. Furthermore, if the reflected object is at a certain distance from the imaging device 1, the difference between the left and right captured images is less likely to appear in the reflected object near the vertical edge compared to the reflected object near the horizontal edge because the lenses are lined up on the left and right.

[0049] For example, if a reflected object is present at position 801 shown in FIGS. 8(a) and 8(b), the reflected object will appear at position 802 shown in FIG. 8(c), i.e., at the right end of each of the left and right images. In the case shown in FIG. 8(a), the distance dL from the left lens to the reflected object is greater than the distance dR from the right lens to the reflected object. Therefore, as shown in FIG. 8(c), the size of the reflected object differs between the left and right images. That is, the area of ​​the reflected object in the captured image that appears in the left image is smaller than the area of ​​the reflected object in the captured image that appears in the right image. This relationship in area is due to the fact that the area of ​​the object in the captured image is inversely proportional to the square of the distance between the object and the lens.

[0050] On the other hand, if a reflected object is present at position 901 shown in FIGS. 9(a) and 9(b), the reflected object appears at position 902 shown in FIG. 9(c), i.e., at the upper end of each of the left and right images. In the case shown in FIG. 9(a), the distance dL from the left lens to the reflected object is approximately equal to the distance dR from the right lens to the reflected object. Therefore, as shown in FIG. 9(c), the size of the reflected object is also approximately equal in the left and right images. That is, the area of ​​the reflected object in the captured image that appears in the left image is approximately equal to the area of ​​the reflected object in the captured image that appears in the right image. Furthermore, as described above, since the area of ​​the subject in the captured image is inversely proportional to the square of the distance between the subject and the lens, the greater the distance from the lens to the reflected object, the less likely a difference appears between the left and right captured images. For these reasons, the difference between the left and right captured images is less likely to appear in the reflection near the vertical edge as shown in FIG. 9(c) than in the reflection near the horizontal edge as shown in FIG. 9(b).

[0051] Returning to the description of FIG. 6, in step S603, the CPU of the imaging device 1 adjusts the threshold value Th according to the position within the captured image using the threshold value calculation unit 104. In the first embodiment, the threshold value calculation unit 104 classifies the currently processed target reflection area pair into a case where the reflection area pair is located at the vertical edge of the captured image and a case where the reflection area pair is located at the horizontal edge of the captured image. Furthermore, the threshold value calculation unit 104 sets the threshold value Th when the currently processed target reflection area pair is located at the vertical edge of the captured image to be smaller than the threshold value Th when the currently processed target reflection area pair is located at the horizontal edge of the captured image. By doing so, the threshold value calculation unit 104 makes a more strict determination of reflection when the reflection area pair is located at the vertical edge of the captured image than when the reflection area pair is located at the horizontal edge of the captured image.

[0052] Specifically, if the current target reflection area pair to be processed is located at the horizontal edge of the captured image, threshold calculation unit 104 leaves threshold Th unchanged. In this case, threshold Th is left unchanged because, as described above, in the first embodiment, the initial value used to initialize threshold Th is set to a value that is suitable for when it is assumed that the reflection object is located at the horizontal edge of the captured image.

[0053] In contrast, if the currently processed reflection area pair is located at the vertical edge of the captured image, threshold calculation unit 104 multiplies threshold Th by a predetermined coefficient smaller than 1, thereby making threshold Th smaller than its initial value. In this case, threshold Th is made smaller than its initial value because, as mentioned above, reflections near the vertical edge are less likely to show differences between the left and right captured images compared to reflections near the horizontal edge. Note that the coefficient is, for example, the average value of the ratio of index values ​​SAD obtained from multiple experiments in which reflections near the horizontal edge and the vertical edge at the reference distance D are generated.

[0054] In step S604, the CPU of the imaging device 1 adjusts the threshold value Th adjusted in step S603 by the threshold calculation unit 104 in accordance with the distance information representative value d generated in step S603. As a result, the threshold value Th is adjusted so that the larger the distance information representative value d, the smaller the threshold value Th. By doing so, the threshold calculation unit 104 makes the reflection determination more strict as the distance indicated by the distance information increases. This is because, as described above, the greater the distance from the lens to the reflected object, the less likely a difference appears between the left and right captured images. In the first embodiment, the threshold value Th is adjusted using the calculation formula shown in equation (2).

[0055] Th=Th*(D / d)^2 (2) In equation (2), * represents multiplication, / represents division, and ^ represents exponentiation. In the first embodiment, equation (2) is used to adjust the threshold value Th according to the distance information representative value d, but this is not limiting. For example, the threshold value Th may be adjusted simply inversely proportional to the distance information representative value d, rather than being adjusted inversely proportional to the square of the distance information representative value d as in equation (2). Furthermore, the equation used to adjust the threshold value Th according to the distance information representative value d may be changed depending on the type of similarity (i.e., index value), etc.

[0056] In step S605, the CPU of the image capture device 1 causes the threshold calculation unit 104 to output the threshold Th adjusted in step S604 to the determination unit 105 as the threshold Th for the current target photographic area pair. This calculates the threshold Th for the current target photographic area pair. In step S606, the CPU of the image capture device 1 determines whether the threshold calculation unit 104 has completed calculation of the threshold Th for all photographic area pairs. If the CPU of the image capture device 1 determines that the threshold calculation unit 104 has not completed calculation of the threshold Th for all photographic area pairs, the process returns to step S600. Thereafter, steps S600 to S606 are repeated until calculation of the threshold Th for all photographic area pairs is completed. On the other hand, if the CPU of the image capture device 1 determines that the threshold calculation unit 104 has completed calculation of the threshold Th for all photographic area pairs, the process of the threshold calculation unit 104 shown in FIG. 6 ends.

[0057] Next, the processing flow of the determination unit 105 will be described in detail using FIG. 10. FIG. 10 is a diagram expressed in the form of a flowchart showing the processing of the determination unit 105. In the processing of the determination unit 105 shown in FIG. 10, a reflection determination is performed for each reflection area pair, and the processing is repeated until reflection determination is completed for all reflection area pairs. When the processing of the determination unit 105 shown in FIG. 10 starts, in step S1000, the CPU of the imaging device 1 initializes the reflection flag F by causing the determination unit 105 to set the reflection flag F to 0. The reflection flag F indicates the determination result of the occurrence of reflection. When the reflection flag F is 0, it means that no reflection contrary to the photographer's intention has occurred, and when the reflection flag F is 1, it means that a reflection contrary to the photographer's intention has occurred. Steps S1001 and S1002 are similar to steps S601 and S602 described above, and therefore detailed description thereof will be omitted.

[0058] In steps S1003 to S1005, a reflection determination is performed on the reflection area pair. In step S1003, the CPU of image capture device 1 determines, via determination unit 105, whether the distance information representative value d of the reflection area pair currently being processed is greater than a predetermined minimum distance D_min. If the CPU of image capture device 1 determines, via determination unit 105, that the distance information representative value d of the reflection area pair currently being processed is greater than the minimum distance D_min, the process proceeds to step S1007, which will be described later. On the other hand, if the CPU of image capture device 1 determines, via determination unit 105, that the distance information representative value d of the reflection area pair currently being processed is not greater than the minimum distance D_min, the process proceeds to step S1004.

[0059] The minimum distance D_min (second predetermined distance) defines the minimum value of the distance of a captured object from the imaging device 1. If the distance information representative value d of the captured area pair currently being processed is greater than the minimum distance D_min, it is determined that no capture contrary to the photographer's intention has occurred, and the captured area pair currently being processed is excluded from the target of capture determination. Generally, captured objects not intended by the photographer are almost always parts of the photographer's body, such as the photographer's hands or head, or parts of the photographer's clothing, such as a hat. Therefore, the minimum distance D_min is determined in advance, for example, based on the range that the photographer's hand can reach while operating the imaging device 1.

[0060] In step S1004, the CPU of image capture device 1 determines, via determination unit 105, whether the index value SAD of the currently targeted image area pair is smaller than a predetermined first threshold value Th_max. If the CPU of image capture device 1 determines, via determination unit 105, that the index value SAD of the currently targeted image area pair is smaller than the first predetermined threshold value Th_max, processing proceeds to step S1007, which will be described later. On the other hand, if the CPU of image capture device 1 determines, via determination unit 105, that the index value SAD of the currently targeted image area pair is not smaller than the first predetermined threshold value Th_max, processing proceeds to step S1005.

[0061] The first predetermined threshold Th_max is the assumed maximum value of the index value SAD of the left and right images in a state where there is no unwanted reflection by the photographer, and is determined in advance for each pair of reflected areas. If the first predetermined threshold Th_max is greater than the index value SAD of the pair of reflected areas currently being processed, it is deemed that there is no unwanted reflection by the photographer, and the pair of reflected areas currently being processed is excluded from the target of reflection judgment. For example, when the index value SAD of the pair of reflected areas in a state where there is no unwanted reflection by the photographer is obtained multiple times through experiments, the maximum value of the index value SAD obtained each time or the average value of the top group is used as the first predetermined threshold Th_max.

[0062] In step S1005, the CPU of the imaging device 1 determines, via the determination unit 105, whether the index value SAD of the current target episcopic area pair is smaller than the threshold value Th of the current target episcopic area pair. The threshold value Th of the current target episcopic area pair is the value supplied from the threshold calculation unit 104. If the CPU of the imaging device 1 determines, via the determination unit 105, that the index value SAD of the current target episcopic area pair is smaller than the threshold value Th of the current target episcopic area pair, the process proceeds to step S1007, which will be described later. On the other hand, if the CPU of the imaging device 1 determines, via the determination unit 105, that the index value SAD of the current target episcopic area pair is not smaller than the threshold value Th of the current target episcopic area pair, the process determines that an unintended image has occurred. Then, the process proceeds to step S1006.

[0063] In step S1006, the CPU of the imaging device 1 sets the reflection flag F to 1 using the determination unit 105. In step S1007, the CPU of the imaging device 1 outputs the reflection flag F set in step S1000 or step S1006 using the determination unit 105 as the reflection flag F of the reflection area pair currently being processed. The reflection flag F of the reflection area pair currently being processed is supplied to the reflection processing unit 106 as the determination result for the reflection area pair currently being processed.

[0064] In step S1008, the CPU of imaging device 1 determines whether or not reflection determination has been completed for all reflection area pairs by determination unit 105. If the CPU of imaging device 1 determines that reflection determination has not been completed for all reflection area pairs by determination unit 105, the process returns to step S1000. Thereafter, steps S1000 to S1008 are repeated until reflection determination has been completed for all reflection area pairs. On the other hand, if the CPU of imaging device 1 determines that reflection determination has been completed for all reflection area pairs by determination unit 105, the process of determination unit 105 shown in FIG. 10 ends.

[0065] Next, the processing contents of the reflection processing unit 106 will be described in detail with reference to FIG. 11. FIG. 11 is a diagram showing an example of processing of a captured image instructed by the reflection processing unit 106. When the reflection processing unit 106 receives the determination result for each reflection area pair (i.e., the reflection flag F) from the determination unit 105, if there is any reflection area pair for which the reflection flag F is 1, the reflection processing unit 106 determines that a reflection contrary to the photographer's intention has occurred. In this case, the reflection processing unit 106 outputs instruction information regarding processing of the captured image to notify a user such as a photographer that a reflection contrary to the photographer's intention has occurred. The instruction information regarding processing of the captured image may be, for example, an instruction to superimpose a warning message 1100 indicating the occurrence of a reflection on the captured image, as shown in FIG. 11. Furthermore, the instruction information regarding processing of the captured image may be, for example, as shown in FIG. 11, an instruction to superimpose a frame line 1101 of the reflection area pair on the captured image as a display object indicating the reflection area pair whose reflection flag F is 1.

[0066] The instruction information regarding the processing of the captured image may be an instruction to simultaneously superimpose both a warning message 1100 and a frame line 1101 on the captured image, as shown in FIG. 11 . Alternatively, the instruction information regarding the processing of the captured image may be an instruction to superimpose either the warning message 1100 or the frame line 1101 on the captured image, which differs from the instruction information shown in FIG. 11 . The instruction information regarding the processing of the captured image is supplied from the reflection processing unit 106 to the display image processing unit 107, the recording image processing unit 109, and the output image processing unit 111. The instruction information regarding the processing of the captured image may be supplied to one or two of the display image processing unit 107, the recording image processing unit 109, and the output image processing unit 111, rather than to all of them. For example, the instruction information regarding the processing of the captured image may be supplied only to the display image processing unit 107. The imaging device 1 may be configured so that a user, such as a photographer, can freely select, by turning on or off, the destination to which the instruction information regarding the processing of the captured image is to be supplied.

[0067] As described above, the imaging device 1 of the first embodiment can improve the accuracy of detecting unwanted reflections by generating and using a threshold value Th for comparison with the similarity depending on the position of the reflected object in the captured image and the distance to the imaging unit 100.

[0068] Second Embodiment The second embodiment will be described below with reference to FIGS. 12 to 13C. In the second embodiment, as in the first embodiment, an imaging device 1 including an information processing device according to the present invention will be described. Therefore, the configuration of the imaging device 1 in the second embodiment is the same as that in the first embodiment, and therefore a description thereof will be omitted. Compared to the first embodiment, the second embodiment differs from the first embodiment in part of the overall processing flow of the imaging device 1 and in the processing contents of some blocks of the imaging device 1. Therefore, the second embodiment will be described focusing on these differences from the first embodiment.

[0069] First, the overall processing flow of the imaging device 1 will be described with reference to FIG. 12. FIG. 12 is a flowchart showing the overall processing of the imaging device 1. When the overall processing of the imaging device 1 shown in FIG. 12 starts, in step S1200, the CPU of the imaging device 1 performs pre-shooting (hereinafter referred to as "pre-shooting") before actual shooting using the imaging unit 100 under operational instructions from the photographer. As a result, the imaging unit 100 performs imaging processing associated with the pre-shooting, and a digital image signal of the pre-shooting is output. The digital image signal of the pre-shooting is supplied from the imaging unit 100 to the development processing unit 102. Note that the pre-shooting in step S1200 is performed to acquire a pre-captured image to be used in the reflection determination. Therefore, the photographer issues operational instructions for the pre-shooting while checking that there is no unintended reflection, for example by viewing a live view image on the display device 108.

[0070] In step S1201, the CPU of the imaging device 1 causes the development processing unit 102 to perform development processing on the digital image signal of the pre-photographed image that is the output of step S1200. As a result, the development processing unit 102 outputs developed image data of the pre-photographed image. The developed image data of the pre-photographed image is supplied from the development processing unit 102 to the similarity acquisition unit 103. In step S1202, the CPU of the imaging device 1 causes the development processing unit 102 to hold the developed image data of the pre-photographed image that is the output of step S1201.

[0071] In step S1203, the CPU of the imaging device 1 performs imaging processing associated with the actual shooting using the imaging unit 100. As a result, the imaging unit 100 outputs a digital image signal of the actual shooting and distance information for each pixel of the actual shooting. The digital image signal of the actual shooting is supplied from the imaging unit 100 to the development processing unit 102. The distance information for each pixel of the actual shooting is supplied from the imaging unit 100 to the distance information calculation unit 101. In step S1204, the CPU of the imaging device 1 causes the development processing unit 102 to perform development processing on the digital image signal of the actual shooting, which is the output of step S1203. As a result, the development processing unit 102 outputs developed image data of the actual shooting. The developed image data of the actual shooting is supplied from the development processing unit 102 to the similarity acquisition unit 103, the display image processing unit 107, the recording image processing unit 109, and the output image processing unit 111.

[0072] In step S1205, the CPU of the imaging device 1 causes the similarity acquisition unit 103 to acquire two types of similarity for each reflected area pair. At this time, the similarity acquisition unit 103 uses the developed image data of the pre-photographing output from step S1201 and the developed image data of the main photography output from step S1204. The first type of similarity (first similarity) is the similarity for each reflected area pair using the developed image data of the main photography (hereinafter referred to as "index value SAD1") and is acquired in the same manner as the index value SAD in the first embodiment. The second type of similarity (second similarity) is the similarity for each reflected area pair (hereinafter referred to as "index value SAD2") acquired using the developed image data of the pre-photographing and the developed image data of the main photography.

[0073] A specific method for obtaining the index value SAD2 will be described below. First, the similarity obtaining unit 103 uses the developed image data from the actual shoot and the developed image data from the preliminary shoot to calculate the index value SAD of the left edge reflected area 202L between the left images and the index value SAD of the left edge reflected area 202R between the right images using the above-mentioned formula (1). The similarity obtaining unit 103 obtains the larger of the index value SAD of the left edge reflected area 202L between the left images of these two image data and the index value SAD of the left edge reflected area 202R between the right images as the index value SAD2 of the pair made up of the left edge reflected areas 202L, 202R.

[0074] Similarly, similarity acquisition unit 103 acquires index values ​​SAD2 for the pair formed by right-edge reflected areas 203L and 203R, the pair formed by top-edge reflected areas 204L and 204R, and the pair formed by bottom-edge reflected areas 205L and 205R. In this way, similarity acquisition unit 103 acquires index value SAD2 for each reflected area pair. Furthermore, similarity acquisition unit 103 supplies index value SAD1 and index value SAD2 for each reflected area pair to determination unit 105 as the similarity for each reflected area pair.

[0075] In step S1206, the CPU of image capture device 1 uses distance information calculation unit 101 to calculate distance information for each reflected area based on the distance information for each pixel of the actual capture output from step S1203. In step S1207, the CPU of image capture device 1 uses threshold calculation unit 104 to calculate a threshold for each reflected area pair in accordance with the distance information for each reflected area output from step S1206. In step S1208, the CPU of image capture device 1 uses determination unit 105 to perform reflection determination for each reflected area pair using the similarity output from step S1205, the distance information output from step S1206, and the threshold value output from step S1207. This will be described in detail later with reference to Figures 13A to 13C.

[0076] In step S1209, the CPU of the imaging device 1 performs processing related to reflections using the reflection processing unit 106. In the reflection processing, instruction information related to processing of the captured image is generated in accordance with the determination result for each reflection area pair, which is the output of step S1208. In step S1210, the CPU of the imaging device 1 performs display image processing using the display image processing unit 107. In the display image processing, the developed image data of the pre-shooting held in step S1202 and the developed image data of the main shooting, which is the output of step S1204, are processed based on the instruction information related to processing of the captured image, which is the output of step S1209. In the display image processing, display image data is generated by further executing predetermined image processing.

[0077] In step S1211, the CPU of the imaging device 1 performs recording image processing using the recording image processing unit 109. In the recording image processing, the developed image data of the pre-shooting held in step S1202 and the developed image data of the main shooting output in step S1204 are processed based on the instruction information related to processing of the captured image output in step S1209. In the recording image processing, a compression process is further executed to generate recording image data, and the generated recording image data is recorded on the recording medium 110.

[0078] In step S1212, the CPU of the imaging device 1 performs output image processing using the output image processing unit 111. In the output image processing, the developed image data of the pre-photographing held in step S1202 and the developed image data of the actual photographing output in step S1204 are processed based on instruction information related to processing of the captured image output in step S1209. In the output image processing, predetermined image processing is further performed to generate image output data, which is output to the outside via the external I / F. In step S1213, the CPU of the imaging device 1 displays the display image data output in step S1210 on the display device 108. When the CPU of the imaging device 1 has completed the processing of all steps, the overall processing of the imaging device 1 shown in FIG. 12 ends.

[0079] Next, the processing flow of determination unit 105 will be described in detail using Fig. 13A. Fig. 13A is a diagram expressed in the form of a flowchart showing the processing of determination unit 105. In the processing of determination unit 105 shown in Fig. 13A, reflection determination is performed for each reflection area pair, and the processing is repeated until reflection determination is completed for all reflection area pairs. When the processing of determination unit 105 shown in Fig. 13A starts, in step S1300, the CPU of imaging device 1 initializes reflection flag F by causing determination unit 105 to set reflection flag F to 0. Steps S1301 and S1302 are similar to steps S601 and S602 described above, and therefore detailed description thereof will be omitted.

[0080] In steps S1303 to S1306, a reflection determination is performed on the reflection area pair. In step S1303, the CPU of image capture device 1 determines, via determination unit 105, whether the distance information representative value d of the reflection area pair currently being processed is greater than the minimum distance D_min. If the CPU of image capture device 1 determines, via determination unit 105, that the distance information representative value d of the reflection area pair currently being processed is greater than the minimum distance D_min, the process proceeds to step S1307, which will be described later. On the other hand, if the CPU of image capture device 1 determines, via determination unit 105, that the distance information representative value d of the reflection area pair currently being processed is not greater than the minimum distance D_min, the process proceeds to step S1304.

[0081] In step S1304, the CPU of the imaging device 1 determines, via the determination unit 105, whether the distance information representative value d of the currently processing target capture area pair is smaller than the predetermined distance D1. If the CPU of the imaging device 1 determines, via the determination unit 105, that the distance information representative value d of the currently processing target capture area pair is not smaller than the predetermined distance D1, the process proceeds to step S1306, which will be described later. On the other hand, if the CPU of the imaging device 1 determines, via the determination unit 105, that the distance information representative value d of the currently processing target capture area pair is smaller than the predetermined distance D1, the process proceeds to step S1305.

[0082] The predetermined distance D1 (first predetermined distance) defines a reference value for determining whether the determination unit 105 should use the index value SAD2 in the image capture determination. When the distance information representative value d is smaller than the predetermined distance D1, the determination unit 105 uses the index value SAD1 but not the index value SAD2. However, as described above, the larger the distance information representative value d, the smaller the index value SAD1 based on the parallax between the left and right images. Therefore, it can be said that the accuracy of the image capture determination using only the index value SAD1 may be reduced. Therefore, when the distance information representative value d is equal to or greater than the predetermined distance D1, the determination unit 105 can compensate for the accuracy of the image capture determination by using the index value SAD2 based on the difference between the pre-captured image and the captured image in addition to the index value SAD1. The predetermined distance D1 is predetermined to a value obtained in advance through experiments or the like that allows the determination unit 105 to perform an appropriate image capture determination.

[0083] Next, the first reflection determination process of step S1305 will be described using FIG. 13B. FIG. 13B is a flowchart showing the first reflection determination process. In the first reflection determination process of step S1305, reflection determination is performed using only index value SAD1. Step S1305 is made up of steps S13050 to S13052. In step S13050, the CPU of image capture device 1 determines, via determination unit 105, whether index value SAD1 of the reflection area pair currently being processed is smaller than a first predetermined threshold value Th_max. If the CPU of image capture device 1 determines, via determination unit 105, that index value SAD1 of the reflection area pair currently being processed is smaller than the first predetermined threshold value Th_max, the process proceeds to step S1307 in FIG. 13A. On the other hand, if the CPU of the image capture device 1 determines through the determination unit 105 that the index value SAD1 of the currently targeted image area pair is not smaller than the first predetermined threshold value Th_max, the process proceeds to step S13051.

[0084] In step S13051, the CPU of the imaging device 1 determines, via the determination unit 105, whether the index value SAD1 of the current target capture area pair is smaller than the threshold value Th of the current target capture area pair. The threshold value Th of the current target capture area pair is the value supplied from the threshold calculation unit 104. If the CPU of the imaging device 1 determines via the determination unit 105 that the index value SAD1 of the current target capture area pair is smaller than the threshold value Th of the current target capture area pair, processing proceeds to step S1307 of FIG. 13A. On the other hand, if the CPU of the imaging device 1 determines via the determination unit 105 that the index value SAD1 of the current target capture area pair is not smaller than the threshold value Th of the current target capture area pair, it is determined that an unintended capture has occurred. Processing then proceeds to step S13052. In step S13052, the CPU of the imaging device 1 sets the reflection flag F to 1 by the determination unit 105. Thereafter, the process proceeds to step S1307 in FIG. 13A.

[0085] Next, the second reflection determination process of step S1306 will be described with reference to FIG. 13C. FIG. 13C is a flowchart showing the second reflection determination process. In the second reflection determination process of step S1306, reflection determination is performed using index values ​​SAD1 and SAD2. Step S1306 is made up of steps S13060 to S13063. In step S13060, the CPU of image capture device 1 determines, via determination unit 105, whether index value SAD1 of the reflection area pair currently being processed is smaller than the first predetermined threshold value Th_max. If the CPU of image capture device 1 determines, via determination unit 105, that index value SAD1 of the reflection area pair currently being processed is smaller than the first predetermined threshold value Th_max, the process proceeds to step S13062, which will be described later. On the other hand, if the CPU of the image capture device 1 determines through the determination unit 105 that the index value SAD1 of the currently targeted image area pair is not smaller than the first predetermined threshold value Th_max, the process proceeds to step S13061.

[0086] In step S13061, the CPU of the imaging device 1 determines, via the determination unit 105, whether the index value SAD1 of the current target episcopic area pair to be processed is smaller than the threshold value Th of the current target episcopic area pair. The threshold value Th of the current target episcopic area pair is the value supplied from the threshold calculation unit 104. If the CPU of the imaging device 1 determines via the determination unit 105 that the index value SAD1 of the current target episcopic area pair to be processed is not smaller than the threshold value Th of the current target episcopic area pair, it is determined that an image contrary to the photographer's intention has occurred. Processing then proceeds to step S13063, which will be described later. On the other hand, if the CPU of the imaging device 1 determines via the determination unit 105 that the index value SAD1 of the current target episcopic area pair to be processed is smaller than the threshold value Th of the current target episcopic area pair to be processed, processing proceeds to step S13062.

[0087] In step S13062, the CPU of the imaging device 1 determines, via the determination unit 105, whether the index value SAD2 of the currently processing target reflected area pair is smaller than the predetermined second threshold value Th_2. If the CPU of the imaging device 1 determines, via the determination unit 105, that the index value SAD2 of the currently processing target reflected area pair is smaller than the second predetermined threshold value Th_2, the process proceeds to step S1307 in FIG. 13A. On the other hand, if the CPU of the imaging device 1 determines, via the determination unit 105, that the index value SAD2 of the currently processing target reflected area pair is not smaller than the second predetermined threshold value Th_2, it is determined that an unintended reflection by the photographer has occurred. As a result, the process proceeds to step S13063.

[0088] As described above, in step S13062, the determination unit 105 uses the index value SAD2 based on the difference between the pre-captured image and the captured image, thereby improving the accuracy of the reflection determination when the distance information representative value d is large. The second predetermined threshold Th_2 is determined in advance as a value that is obtained in advance through experiments or the like and that allows the determination unit 105 to perform a suitable reflection determination. In step S13063, the CPU of the imaging device 1 sets the reflection flag F to 1 using the determination unit 105. Then, the process proceeds to step S1307 in FIG. 13A.

[0089] Returning to the description of Figure 13A, in step S1307, the CPU of the imaging device 1 causes determination unit 105 to output the reflection flag F set in step S1300, step S13052, or step S13063 as the reflection flag F of the reflection area pair currently being processed. The reflection flag F of the reflection area pair currently being processed is supplied to reflection processing unit 106 as the determination result for the reflection area pair currently being processed.

[0090] In step S1308, the CPU of imaging device 1 determines whether determination unit 105 has completed reflection determination for all reflection area pairs. If the CPU of imaging device 1 determines that determination unit 105 has not completed reflection determination for all reflection area pairs, the process returns to step S1300. Thereafter, steps S1300 to S1308 are repeated until reflection determination has been completed for all reflection area pairs. On the other hand, if the CPU of imaging device 1 determines that determination unit 105 has completed reflection determination for all reflection area pairs, the process of determination unit 105 shown in FIG. 13A ends.

[0091] Next, the processing content of the reflection processing unit 106 will be described in detail. The reflection processing unit 106 can execute the processing content described in the first embodiment using FIG. 11 on the developed image data of the actual shooting, but can also execute the processing content described below. When the reflection processing unit 106 receives the determination result for each reflection area pair (i.e., the reflection flag F) from the determination unit 105, if there is even one reflection area pair with the reflection flag F set to 1, it determines that a reflection contrary to the photographer's intention has occurred. In this case, the reflection processing unit 106 generates instruction information for processing the captured image to replace the image of the reflection area pair with the reflection flag F set to 1 with the image of the corresponding reflection area pair in the developed image data of the pre-shooting held by the development processing unit 102.

[0092] Instruction information regarding processing of the captured image is supplied from the reflection processing unit 106 to the display image processing unit 107, the recording image processing unit 109, and the output image processing unit 111. Furthermore, the instruction information regarding processing of the captured image may be supplied to one or two of the display image processing unit 107, the recording image processing unit 109, and the output image processing unit 111 instead of being supplied to all of them. For example, the instruction information regarding processing of the captured image may be supplied only to the display image processing unit 107. Furthermore, the imaging device 1 may be configured so that a user, such as a photographer, can freely select, by turning on or off, the destination to which the instruction information regarding processing of the captured image is to be supplied.

[0093] As described above, the imaging device 1 of the second embodiment can improve the accuracy of detecting unwanted reflections by selectively using the index value SAD1 and the index value SAD2 depending on the distance of the reflected object to the imaging unit 100.

[0094] Note that the determination unit 105 may use the index value SAD1 and not use the index value SAD2 only when the distance information representative value d is smaller than the predetermined distance D1 and the currently processing target reflected area pair is located at the horizontal edge of each of the left and right images. Therefore, in this modification, the determination unit 105 uses the index values ​​SAD1 and SAD2 even if the distance information representative value d is smaller than the predetermined distance D1, if the currently processing target reflected area pair is located at the vertical edge of each of the left and right images. Furthermore, if the distance information representative value d is equal to or greater than the predetermined distance D1, the determination unit 105 uses the index values ​​SAD1 and SAD2 regardless of the positions of the currently processing target reflected area pair in each of the left and right images.

[0095] <Other> While preferred embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and various modifications and variations are possible within the scope of the present invention. For example, the distance measurement calculation unit 403 of the image capture unit 100 uses pupil-division phase difference detection when calculating distance information for each pixel. However, this is not limited to this. For example, parallax between left and right images may be used. Furthermore, the similarity acquisition unit 103 uses a luminance component when calculating similarity. However, this is not limited to this. For example, any one of the color components of the RAW data may be used, or the average value of all color components may be used. Furthermore, the display object indicating a reflection area pair whose reflection flag F is 1 is not limited to the frame line 1101. For example, the reflection area pair may be filled with a highly transparent material. Alternatively, the user may be notified of the reflection by a method other than a display, such as sound or lighting of an LED.

[0096] The present invention can also be realized by supplying a program that realizes one or more functions of each of the above embodiments to a system or device via a network or recording medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A first acquisition means acquires a first similarity indicating the degree of similarity between the left image and the right image of a captured image for each pair of reflection areas formed by two reflection areas located at corresponding positions in the left image and the right image of the captured image, out of a plurality of reflection areas set at corresponding positions in each of the left image and the right image of the captured image having parallax; a second acquisition means for acquiring, for each of the plurality of reflection areas, distance information indicating a distance from an imaging unit that acquired the captured image to a subject in the reflection area; a third acquisition means for acquiring a threshold value for each pair of reflection areas in accordance with the positions of the pair of reflection areas in the left and right images of the captured image and the distance information; a determination means for determining whether or not a reflection occurs in the captured image for each pair of reflection areas using the first similarity and the threshold value; and a reflection processing means for performing processing relating to reflection based on the result of the reflection determination. (Configuration 2) The information processing device described in Configuration 1, wherein the third acquisition means acquires the threshold for each pair of reflection areas so that the reflection judgment is performed more strictly when the positions of the reflection area pair in the left and right images of the captured image are at the vertical ends of the left and right images of the captured image than when they are at the horizontal ends of the left and right images of the captured image. (Configuration 3) The information processing device described in configuration 1 or 2, characterized in that the third acquisition means acquires the threshold value for each pair of reflection areas so that the reflection judgment becomes stricter as the distance indicated by the distance information increases. (Configuration 4) An information processing device described in any one of configurations 1 to 3, characterized in that when the determination means determines that a reflection has occurred in the captured image, the reflection processing means processes the reflection by displaying a warning message indicating that a reflection has occurred in the captured image when the captured image is displayed on a display device. (Configuration 5) The information processing device described in Configuration 4 is characterized in that, when the determination means determines that a reflection has occurred in the captured image, the reflection processing means, as processing related to the reflection, displays the warning message when displaying the captured image on the display device, and also superimposes on the captured image a display object indicating the reflection area pair for which the determination means has determined that a reflection has occurred in the captured image. (Configuration 6) The first acquisition means acquires, for each pair of reflection areas, a second similarity indicating a degree of similarity between a left image and a right image of a pre-captured image acquired by the imaging unit before the captured image and a left image and a right image of the captured image; The information processing device described in any one of configurations 1 to 3, characterized in that the determination means uses the first similarity and the second similarity depending on the distance information to perform the reflection determination for each reflection area pair. (Configuration 7) The information processing device described in Configuration 6 is characterized in that the judgment means uses the first similarity when the distance indicated by the distance information is smaller than a first predetermined distance, and uses the first similarity and the second similarity when the distance indicated by the distance information is not smaller than the first predetermined distance, to perform the reflection judgment for each reflection area pair. (Configuration 8) The information processing device described in Configuration 6 is characterized in that the determination means performs the reflection determination for each reflection area pair using the first similarity if the distance indicated by the distance information is smaller than a first predetermined distance and the positions of the reflection area pair in the left and right images of the captured image are at the horizontal ends of the left and right images of the captured image, and using the first similarity and the second similarity if the distance indicated by the distance information is not smaller than the first predetermined distance and the positions of the reflection area pair in the left and right images of the captured image are not at the horizontal ends of the left and right images of the captured image. (Configuration 9) An information processing device described in any one of configurations 6 to 8, characterized in that when the determination means determines that a reflection has occurred in the captured image, the reflection processing means processes the reflection by displaying a warning message indicating that a reflection has occurred in the captured image when displaying the captured image on a display device. (Configuration 10) The information processing device described in Configuration 9 is characterized in that, when the determination means determines that a reflection has occurred in the captured image, the reflection processing means, as processing related to the reflection, displays the warning message when the captured image is displayed on the display device, and also superimposes on the captured image a display object indicating the reflection area pair for which the determination means has determined that a reflection has occurred in the captured image. (Configuration 11) The information processing device described in configuration 9 or 10 is characterized in that, when the determination means determines that a reflection has occurred in the captured image, the reflection processing means processes the reflection by replacing the reflection area pair of the captured image in which the determination means has determined that a reflection has occurred in the captured image with the reflection area pair at a corresponding position in the pre-captured image when displaying the captured image on the display device. (Configuration 12) An information processing device described in any one of configurations 6 to 11, characterized in that when the determination means determines that a reflection has occurred in the captured image, the reflection processing means processes the reflection by replacing the reflection area pair of the captured image in which the determination means has determined that a reflection has occurred in the captured image with the reflection area pair at a corresponding position in the pre-captured image when recording the captured image on a recording medium. (Configuration 13) An information processing device described in any one of configurations 6 to 12, characterized in that when the determination means determines that a reflection has occurred in the captured image, the reflection processing means processes the reflection by replacing the reflection area pair of the captured image in which the determination means has determined that a reflection has occurred in the captured image with the reflection area pair at a corresponding position in the pre-captured image when outputting the captured image to the outside. (Configuration 14) An information processing device described in any one of configurations 6 to 13, characterized in that the first acquisition means acquires the second similarity of the left image and the right image of the reflection area pair, whichever has the smaller degree of similarity, as the second similarity of the reflection area pair. (Configuration 15) The information processing device according to any one of configurations 1 to 5, wherein the captured image is an image acquired with a fisheye lens. (Configuration 16) The information processing device according to any one of configurations 6 to 14, wherein the captured image and the pre-captured image are images acquired with a fisheye lens. (Configuration 17) The information processing device according to any one of configurations 1 to 16, wherein the reflection processing means performs processing related to the reflection in response to an instruction from a user. (Configuration 18) The information processing apparatus according to any one of configurations 1 to 17, wherein the second acquisition means acquires the distance information based on a phase difference between the captured images. (Configuration 19) The information processing device according to any one of configurations 1 to 18, wherein the determination means uses the distance information indicating the smaller distance among the distance information of each of the two reflection areas that make up the reflection area pair. (Configuration 20) The information processing device described in Configuration 19 is characterized in that, when the distance indicated by the distance information is greater than a second predetermined distance, the judgment means excludes the reflection area pair having the reflection area of ​​the distance information from the target of the reflection judgment. (Configuration 21) An information processing device described in any one of configurations 1 to 20, characterized in that the third acquisition means uses the distance information indicating the smaller distance among the distance information of each of the two reflection areas that make up the reflection area pair. (Configuration 22) The information processing device according to any one of configurations 1 to 21, wherein the plurality of reflection regions are regions along the peripheries of the left and right images of the captured image. (Configuration 23) The imaging unit; An imaging device comprising: an information processing device according to any one of configurations 1 to 22. (Method 1) A first acquisition step of acquiring a first similarity indicating the degree of similarity between the left image and the right image of a captured image for each pair of reflection areas formed by two reflection areas located at corresponding positions in the left image and the right image of the captured image, out of a plurality of reflection areas set at corresponding positions in each of the left image and the right image of the captured image having parallax; a second acquisition step of acquiring distance information indicating a distance from an imaging unit that acquired the captured image to a subject in the reflected area for each of the plurality of reflected areas; a third acquisition step of acquiring a threshold value for each pair of reflection areas according to the positions of the pair of reflection areas in the left image and the right image of the captured image and the distance information; a determination step of determining whether or not a reflection occurs in the captured image for each pair of reflection areas using the first similarity and the threshold value; a reflection processing step of performing processing related to reflection based on the result of the reflection determination. (Program 1) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 22. [Explanation of symbols]

[0098] 1. Imaging device 100 Imaging unit 101 Distance information calculation unit (second acquisition means) 103 Similarity acquisition unit (first acquisition means) 104 Threshold calculation unit (third acquisition means) 105 Judgment unit (judgment means) 106 Reflection processing unit (reflection processing means) SAD index value (first similarity) Th threshold

Claims

1. a first acquisition means for acquiring a first similarity indicating a degree of similarity between the left image and the right image of a captured image for each pair of reflection areas formed by two reflection areas located at corresponding positions in the left image and the right image of the captured image, out of a plurality of reflection areas set at corresponding positions in each of the left image and the right image of the captured image having parallax; a second acquisition means for acquiring, for each of the plurality of reflection areas, distance information indicating a distance from an imaging unit that acquired the captured image to a subject in the reflection area; a third acquisition means for acquiring a threshold value for each pair of reflection areas in accordance with the positions of the pair of reflection areas in the left and right images of the captured image and the distance information; a determination means for determining whether or not a reflection occurs in the captured image for each pair of reflection areas using the first similarity and the threshold value; and a reflection processing means for performing processing relating to reflection based on the result of the reflection determination.

2. The information processing device described in claim 1, characterized in that the third acquisition means acquires the threshold value for each pair of reflection areas so that the reflection judgment is performed more strictly when the positions of the reflection area pair in the left and right images of the captured image are at the vertical ends of the left and right images of the captured image than when they are at the horizontal ends of the left and right images of the captured image.

3. The information processing apparatus according to claim 1 , wherein the third acquisition means acquires the threshold value for each pair of reflection areas so that the reflection determination is performed more strictly as the distance indicated by the distance information increases.

4. The information processing device according to claim 1, characterized in that, when the determination means determines that a reflection has occurred in the captured image, the reflection processing means, as processing related to the reflection, displays a warning message indicating that a reflection has occurred in the captured image when the captured image is displayed on a display device.

5. The information processing device according to claim 4, characterized in that, when the determination means determines that a reflection has occurred in the captured image, the reflection processing means, in addition to displaying the warning message when displaying the captured image on the display device, superimposes on the captured image a display object indicating the reflection area pair for which the determination means has determined that a reflection has occurred in the captured image.

6. the first acquisition means acquires, for each pair of reflection areas, a second similarity indicating a degree of similarity between a left image and a right image of a pre-captured image acquired by the imaging unit before the captured image and a left image and a right image of the captured image; 2 . The information processing apparatus according to claim 1 , wherein the determination means performs the reflection determination for each reflection area pair by selectively using the first similarity and the second similarity depending on the distance information.

7. The information processing device described in claim 6, characterized in that the determination means uses the first similarity when the distance indicated by the distance information is smaller than a first predetermined distance, and uses the first similarity and the second similarity when the distance indicated by the distance information is not smaller than the first predetermined distance, to perform the reflection determination for each reflection area pair.

8. The information processing device described in claim 6, characterized in that the determination means uses the first similarity if the distance indicated by the distance information is smaller than a first predetermined distance and the positions of the reflection area pair in the left and right images of the captured image are at the horizontal ends of the left and right images of the captured image, and uses the first similarity and the second similarity if the distance indicated by the distance information is smaller than the first predetermined distance and the positions of the reflection area pair in the left and right images of the captured image are not at the horizontal ends of the left and right images of the captured image.

9. The information processing device according to claim 6, characterized in that, when the determination means determines that a reflection has occurred in the captured image, the reflection processing means, as processing related to the reflection, displays a warning message indicating that a reflection has occurred in the captured image when the captured image is displayed on a display device.

10. The information processing device according to claim 9, characterized in that, when the determination means determines that a reflection has occurred in the captured image, the reflection processing means, in addition to displaying the warning message when displaying the captured image on the display device, superimposes on the captured image a display object indicating the reflection area pair for which the determination means has determined that a reflection has occurred in the captured image.

11. The information processing device described in claim 10, characterized in that when the determination means determines that a reflection has occurred in the captured image, the reflection processing means processes the reflection by replacing the reflection area pair of the captured image in which the determination means has determined that a reflection has occurred in the captured image with the reflection area pair at a corresponding position in the pre-captured image when displaying the captured image on the display device.

12. The information processing device described in claim 6, characterized in that when the determination means determines that reflection has occurred in the captured image, the reflection processing means processes the reflection by replacing the reflection area pair of the captured image in which the determination means has determined that reflection has occurred in the captured image with the reflection area pair at a corresponding position in the pre-captured image when recording the captured image on a recording medium.

13. The information processing device described in claim 6, characterized in that when the determination means determines that a reflection has occurred in the captured image, the reflection processing means processes the reflection by replacing the reflection area pair of the captured image in which the determination means has determined that a reflection has occurred in the captured image with the reflection area pair at a corresponding position in the pre-captured image when outputting the captured image to an external device.

14. The information processing device according to claim 6, characterized in that the first acquisition means acquires the second similarity of the left image and the right image of the reflection area pair, whichever has the smaller degree of similarity, as the second similarity of the reflection area pair.

15. 2. The information processing apparatus according to claim 1, wherein the captured image is an image captured by a fisheye lens.

16. The information processing apparatus according to claim 6 , wherein the captured image and the pre-captured image are images obtained with a fisheye lens.

17. 2. The information processing apparatus according to claim 1, wherein the reflection processing means performs processing relating to the reflection in response to an instruction from a user.

18. The information processing apparatus according to claim 1 , wherein the second acquisition means acquires the distance information based on a phase difference between the captured images.

19. 2. The information processing apparatus according to claim 1, wherein the determining means uses the distance information indicating the smaller distance out of the distance information of each of the two reflected areas forming the reflected area pair.

20. 20. The information processing apparatus according to claim 19, wherein the determination means excludes the reflection area pair having the reflection area of ​​the distance information from the reflection determination target when the distance indicated by the distance information is greater than a second predetermined distance.

21. 2. The information processing apparatus according to claim 1, wherein the third acquisition means uses the distance information indicating the smaller distance out of the distance information of the two reflection areas that make up the reflection area pair.

22. The information processing device according to claim 1 , wherein the plurality of reflection regions are regions along the peripheries of the left and right images of the captured image.

23. the imaging unit; An imaging device comprising: the information processing device according to claim 1 .

24. a first acquisition step of acquiring a first similarity indicating a degree of similarity between the left image and the right image of the captured image for each pair of reflection areas formed by two reflection areas located at corresponding positions in the left image and the right image of the captured image, out of a plurality of reflection areas set at corresponding positions in each of the left image and the right image of the captured image having parallax; a second acquisition step of acquiring distance information indicating a distance from an imaging unit that acquired the captured image to a subject in the reflected area for each of the plurality of reflected areas; a third acquisition step of acquiring a threshold value for each pair of reflection areas according to the positions of the pair of reflection areas in the left image and the right image of the captured image and the distance information; a determination step of determining whether or not a reflection occurs in the captured image for each pair of reflection areas using the first similarity and the threshold value; a reflection processing step of performing processing related to reflection based on the result of the reflection determination.

25. 2. A program for causing a computer to execute each means of the information processing apparatus according to claim 1.

Images