Imaging device, image processing method, and program

Abstract

Provided are a device and a method for calculating the reliability of a defocus amount and a distance value for each image area of a captured image, and executing a display process for graphic data indicating the reliability, a distance ratio display process of an object, an exposure time control process, and the like. The defocus amount and distance value for each image area of the captured image are calculated, the reliability of the calculated defocus amount and the distance value for each image area is calculated, and control is executed in accordance with the calculated reliability of the defocus amount and the distance value for each image area. For example, a process for superimposing and displaying graphic data indicating the reliability of the defocus amount for each image area on the captured image, a process for displaying the distance ratio of the object, a process for controlling the exposure time, and the like are executed.

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device, an image processing method, and a program. More specifically, the present disclosure relates to an imaging device, an image processing method, and a program that calculates the reliability of a physical quantity for each image region, such as a defocus amount or a distance value for each image region, obtained from detection information of an image-plane phase difference detection pixel, and performs various processes according to the calculated reliability. [Background technology]

[0002] In an imaging device (camera), one method for detecting a focus position (in-focus position) is an image plane phase difference method using image plane phase difference pixels. This image plane phase difference method splits the light passing through the imaging lens into pupils to generate a pair of images, and then analyzes the phase difference between the pair of images to detect the focus position (in-focus position).

[0003] In focus control using the image plane phase difference method, a light beam passing through an imaging lens is split into two, and each of the split light beams is received by a set of image plane phase difference detection pixels that function as a focus detection sensor. The degree of focus is detected based on the amount of deviation in the signals output in accordance with the amount of light received by each of the set of image plane phase difference detection pixels, and the focus lens is adjusted.

[0004] The detection information of the image surface phase difference detection pixel is mainly used for focus control, but can also be applied to other processes in addition to focus control. For example, Patent Document 1 (Japanese Patent Laid-Open No. 2012-142952) discloses a configuration in which detection information from image-plane phase difference detection pixels is used to perform blurring processing on a captured image. Specifically, the detection information of the image plane phase difference detection pixels is used to analyze the distribution of the defocus amount in the captured image, and the analysis results are used to perform blur processing on the captured image.

[0005] Furthermore, Patent Document 2 (JP 2019-035967 A) discloses a configuration in which the gradation characteristics of the defocus amount and distance information obtained from detection information of image plane phase difference detection pixels are changed and output according to the output destination. For example, a configuration is disclosed in which different information based on detection information from image plane phase difference detection pixels is generated and output depending on the information required by the output destination, such as an output destination that requires distance resolution near the focus point or an output destination that requires ranging range information.

[0006] Furthermore, Patent Document 3 (Japanese Patent Laid-Open Publication No. 2011-053378) discloses a configuration for controlling exposure in accordance with the defocus amount and photometric value of each image region of a captured image. Specifically, the present invention discloses a configuration in which exposure control is performed taking into consideration the photometric values ​​of non-focused areas other than the photometric values ​​of the focus detection area to be focused. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-142952 [Patent Document 2] Japanese Patent Application Publication No. 2019-035967 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-053378 [Patent Document 4] Japanese Patent Application Laid-Open No. 2011-004088 Summary of the Invention [Problem to be solved by the invention]

[0008] The present disclosure aims to provide an imaging device, an image processing method, and a program that calculate the reliability of physical quantities per image area, such as defocus amounts and distance values ​​per image area obtained from detection information of image-plane phase difference detection pixels, and perform various processes according to the calculated reliability.

[0009] For example, in an embodiment of the present disclosure, a configuration is realized in which an image in which the reliability of the defocus amount for each image region can be identified is generated and output. Furthermore, in the configuration of one embodiment of the present disclosure, a configuration is realized in which the distance ratio between the focused subject and the background subject is output. For example, in one embodiment of the present disclosure, a configuration is realized in which an image is generated and output that makes it possible to identify the stability of a mask that is set in a partial area of ​​a captured image, depending on the reliability of the defocus amount per image area. Furthermore, in the configuration according to the embodiment of the present disclosure, a configuration is realized in which exposure control is performed for each image area in accordance with the reliability of the defocus amount for each image area. [Means for solving the problem]

[0010] A first aspect of the present disclosure provides: an image region physical quantity calculation unit that calculates a physical quantity that changes depending on the subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; The imaging device includes an image region unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image region unit calculated by the image region unit physical quantity reliability calculation unit.

[0011] Furthermore, a second aspect of the present disclosure is An image processing method executed in an imaging device, an image region physical quantity calculation step in which an image region physical quantity calculation unit calculates a physical quantity that changes depending on a subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation step in which an image region unit physical quantity reliability calculation unit calculates the reliability of the physical quantity for each image region calculated in the image region physical quantity calculation step; The image processing method includes an image region unit physical quantity reliability corresponding processing execution step in which an image region unit physical quantity reliability corresponding processing execution unit executes control processing according to the reliability of the physical quantity for each image region calculated in the image region unit physical quantity reliability calculation step.

[0012] Furthermore, a third aspect of the present disclosure is A program for causing an imaging device to perform image processing, an image region physical quantity calculation step of causing an image region physical quantity calculation unit to calculate a physical quantity that changes depending on the subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation step of causing an image region unit physical quantity reliability calculation unit to calculate the reliability of the physical quantity for each image region calculated in the image region physical quantity calculation step; The program causes an image region unit physical quantity reliability corresponding processing execution step to cause an image region unit physical quantity reliability corresponding processing execution unit to execute control processing according to the reliability of the physical quantity for each image region calculated in the image region unit physical quantity reliability calculation step.

[0013] The program of the present disclosure is a program that can be provided, for example, via a storage medium or communication medium in a computer-readable format to an information processing device or computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or computer system.

[0014] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0015] According to the configuration of one embodiment of the present disclosure, an apparatus and method are realized that calculate the reliability of the defocus amount and distance value for each image area of ​​a captured image, and perform display processing of graphic data indicating the reliability, display processing of the distance ratio of a subject, exposure time control processing, etc. Specifically, for example, the defocus amount and distance value are calculated for each image region of the captured image, and the reliability of the calculated defocus amount and distance value for each image region is further calculated, and control is performed according to the reliability of the calculated defocus amount and distance value for each image region. For example, the control may perform a process of displaying graphic data indicating the reliability of the defocus amount for each image region by superimposing it on the captured image, a process of displaying the distance ratio of the subject, a process of controlling the exposure time, etc. This configuration realizes an apparatus and method that calculates the reliability of the defocus amount and distance value for each image area of ​​a captured image, and performs display processing of graphic data indicating the reliability, display processing of the distance ratio of the subject, exposure time control processing, etc. The effects described in this specification are merely examples and are not limiting, and additional effects may also be present. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of the configuration of an imaging device according to the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of an image sensor having phase difference detection pixels. [Figure 3] 1A and 1B are diagrams illustrating an overview of focus detection processing using a phase difference detection method. [Figure 4] 1A and 1B are diagrams illustrating an overview of focus detection processing using a phase difference detection method. [Figure 5] 1A and 1B are diagrams illustrating an overview of focus detection processing using a phase difference detection method. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging device. [Figure 7] 1A and 1B are diagrams illustrating a specific example of a pixel configuration of an image sensor and an image area as a unit for calculating a defocus amount. [Figure 8]10A and 10B are diagrams illustrating a specific example of a defocus map generated by a digital signal processing unit of an imaging device. [Figure 9] 2 is a diagram illustrating an example of the configuration of a digital signal processing unit of the imaging apparatus according to the first embodiment. FIG. [Figure 10] 3A to 3C are diagrams illustrating a specific example of processing executed by a digital signal processing unit of the imaging device according to the first embodiment. [Figure 11] 3A to 3C are diagrams illustrating a specific example of processing executed by a digital signal processing unit of the imaging device according to the first embodiment. [Figure 12] 3A to 3C are diagrams illustrating a specific example of processing executed by a digital signal processing unit of the imaging device according to the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging apparatus according to a second embodiment. [Figure 14] 10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an image pickup apparatus according to a second embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging apparatus according to a third embodiment. [Figure 16] 10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an image pickup apparatus according to a third embodiment. [Figure 17] 10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an image pickup apparatus according to a third embodiment. [Figure 18] 10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an image pickup apparatus according to a third embodiment. [Figure 19] FIG. 10 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging apparatus according to a fourth embodiment. [Figure 20] 10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an image pickup apparatus according to a fourth embodiment. [Figure 21] 10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an image pickup apparatus according to a fourth embodiment. [Figure 22] FIG. 10 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging apparatus according to a fifth embodiment. [Figure 23]10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an imaging device according to a fifth embodiment. [Figure 24] 10A and 10B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an imaging device according to a fifth embodiment. [Figure 25] FIG. 13 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging apparatus according to a sixth embodiment. [Figure 26] 13A and 13B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an imaging device according to a sixth embodiment. [Figure 27] FIG. 13 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging apparatus according to a sixth embodiment. [Figure 28] 13A and 13B are diagrams illustrating a specific example of processing executed by a digital signal processing unit of an imaging device according to a sixth embodiment. [Figure 29] FIG. 13 is a diagram illustrating an example of the configuration of a digital signal processing unit of an imaging apparatus according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The imaging device, image processing method, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items. 1. Configuration example of the imaging device of the present disclosure 2. Overview of the configuration of phase difference detection pixels and focus control using detection signals from phase difference detection pixels 3. Basic configuration example for calculating the reliability of physical quantities for each image region, such as the defocus amount and distance value, and performing various processes according to the calculated reliability 4. Specific examples of calculating the reliability of physical quantities for each image region, such as the defocus amount and distance value for each image region, and performing various processes according to the calculated reliability 4-1. (Example 1) Example of generating and outputting an image in which the reliability of the defocus amount in image area units can be identified 4-2. (Example 2) Example of generating and outputting an image in which the distance ratio between the focused subject and other background subjects is superimposed on the captured image 4-3. (Example 3) Example of generating and outputting an image in which mask stability information is superimposed on a captured image 4-4. (Example 4) Example of generating and outputting an image in which a color map outputting colors according to the defocus amount in image area units is superimposed on a captured image 4-5. (Example 5) An example in which a color corresponding to the defocus amount in each image area is output, and an image in which a color map that enables identification of the reliability of the defocus amount in each image area is superimposed on a captured image is generated and output. 4-6. (Example 6) Example of image capture by controlling exposure time according to the reliability of defocus amount in image area units 5. Other Examples 6. Summary of the Disclosure

[0018] 1. Configuration example of the imaging device of the present disclosure First, a configuration example of an imaging device according to the present disclosure will be described.

[0019] FIG. 1 is a block diagram showing an example configuration of an imaging device 100 according to the present disclosure. An example configuration of an imaging device 100 according to the present disclosure will be described with reference to FIG. Incident light passing through a focus lens 101 and a zoom lens 102 is input to an image pickup element 103 such as a CMOS or CCD, and is photoelectrically converted in the image pickup element 103 .

[0020] The image sensor 103 has a plurality of pixels, each having a photodiode, arranged two-dimensionally in a matrix, and includes normal pixels on the light receiving surface of which color filters, each having different spectral characteristics, for example, R (red), G (green), and B (blue), are arranged, and phase difference detection pixels for pupil-dividing the subject light and detecting the focus.

[0021] The normal pixels of the image sensor 103 generate analog electrical signals (image signals) of the R (red), G (green), and B (blue) color components of the subject image and output them as R, G, and B color image signals. The phase difference detection pixels of the image sensor 103 output a phase difference detection signal (detection information). The phase difference detection signal (detection information) is a signal that is mainly used for autofocus control. The configuration of the phase difference detection pixel, the phase difference detection signal generated by the phase difference detection pixel, and the focus control using the phase difference detection signal will be described in detail later.

[0022] In this way, the photoelectric conversion data output from the image sensor 103 includes an RGB image signal and a phase difference detection signal from the phase difference detection pixel. These signals are input to analog signal processing section 104, where they are subjected to processing such as noise removal, and then converted into digital signals by A / D conversion section 105.

[0023] The digital signal converted by the A / D converter 105 is input to a digital signal processor (DSP) 108, where various signal processes are performed. The RGB image signal undergoes various image signal processing such as demosaic processing, white balance adjustment, and gamma correction, and the processed image is recorded in a recording device 115 that is configured, for example, by a flash memory. Furthermore, the image is displayed on the monitor 117 and the viewfinder (EVF) 116. The monitor 117 and the viewfinder (EVF) 116 display an image through the lens as a through image, regardless of whether or not an image is being captured.

[0024] Phase difference detection pixel information (detection signal) output from the phase difference detection pixel of the image sensor 103 is also input to the digital signal processor (DSP) 108 via the AD converter 106. The digital signal processing unit (DSP) 108 analyzes the phase difference between a pair of images generated by the phase difference detection pixel information (detection signal) and calculates the amount of focus deviation for the subject (focus object) to be focused on, i.e., the amount of deviation between the focus distance and the subject distance (defocus amount (DF)).

[0025] An input unit (operation unit) 118 is an operation unit including an input unit for inputting various operation information, such as a shutter and zoom button on the camera body, and a mode dial for setting the shooting mode.

[0026] The control unit 110 has a CPU and controls various processes executed by the imaging device according to programs stored in advance in a memory (ROM) 120. A memory (EEPROM) 119 is a non-volatile memory that stores image data, various auxiliary information, programs, and the like.

[0027] The memory (ROM) 120 stores programs and calculation parameters used by the control unit (CPU) 110. The memory (RAM) 121 stores programs used by the control unit (CPU) 110, the AF control unit 112a, etc., and parameters that change as appropriate during the execution of the programs.

[0028] The gyro 131 is a sensor that measures the tilt, angle, tilt speed (angular speed), etc. of the imaging device 100. The detection information of the gyro 131 is used, for example, to calculate the amount of camera shake when capturing an image.

[0029] AF control unit 112a drives focus lens drive motor 113a set in correspondence with focus lens 101 to perform autofocus control (AF control) processing. For example, focus lens 101 is moved to a focus position of focus lens 101 for a subject included in an area selected by the user, thereby obtaining a focused state.

[0030] The zoom control unit 112b drives a zoom lens drive motor 113b set in correspondence with the zoom lens 102. The vertical driver 107 drives the image sensor (CCD) 103. The timing generator 106 generates control signals for the processing timing of the image sensor 103 and the analog signal processing unit 104, and controls the processing timing of each of these processing units. The focus lens 101 is driven in the optical axis direction under the control of the AF control unit 112a.

[0031] [2. Overview of the configuration of phase difference detection pixels and focus control using detection signals from phase difference detection pixels] Next, the configuration of the phase difference detection pixel and an overview of focus control using the detection signal from the phase difference detection pixel will be described.

[0032] As described above, the image sensor 103 of the image sensor 100 shown in FIG. 1 generates analog electrical signals (image signals) for each of the R (red), G (green), and B (blue) color components of a subject image and outputs them as image signals for each of the R, G, and B colors, while also outputting a phase difference detection signal (detection information) from a phase difference detection pixel, which is a signal used for autofocus control. A specific example of the pixel configuration of the image sensor 103 will be described with reference to FIG.

[0033] FIG. 2 is a diagram showing an example of the pixel configuration of the image sensor 103. As shown in FIG. FIG. 2 shows (A) an example of the pixel configuration of the image sensor 103 corresponding to a partial area of ​​a captured image. The vertical direction is the Y axis, and the horizontal direction is the X axis. In Figure 2, one pixel is represented by one square. The RGB pixels shown in Fig. 2 are pixels used for normal image capture, and have, for example, a Bayer array configuration.

[0034] Detection information acquisition pixels applied to autofocus processing, that is, phase difference detection pixels 151 for acquiring phase difference information, are discretely set in some (rows) of the RGB pixels having a Bayer array. The phase difference detection pixel is configured by a pair of a right-opening phase difference detection pixel Pa and a left-opening phase difference detection pixel Pb.

[0035] The image sensor 103 outputs the following two types of data individually. (1) Pixel information (image signal) output from pixels (RGB pixels) for captured images; (2) Output of phase difference detection pixel information ((AF) detection signal) by the phase difference detection pixel 151;

[0036] "(1) Output of pixel information (image signal) from pixels (RGB pixels) for captured image" is output in accordance with the timing of image capture by the user (photographer), and even when not capturing an image, a display image (live view image) is output to be displayed on the monitor 117, etc. The display image (live view image) is output at a frame rate according to the image display rate of the monitor 117, etc.

[0037] "(2) Output of phase difference detection pixel information (detection signal) by the phase difference detection pixel 151" is performed at intervals equal to or shorter than the image output intervals, for example, at (1 / 60) sec intervals (=16.7 msec intervals).

[0038] The phase difference detection pixel information (detection signal) output by the phase difference detection pixel 151 is input to a digital signal processing unit (DSP) 108 via an AD conversion unit 106. The digital signal processing unit (DSP) 108 analyzes the phase difference between a pair of images generated by the phase difference detection pixel information (detection signal) and calculates the amount of focus deviation for the subject (focus object) to be focused on, i.e., the amount of deviation between the focus distance and the subject distance (defocus amount (DF)).

[0039] An outline of the focus detection process using the phase difference detection method will be described with reference to FIGS. As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the deviation amount of the signals output according to the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor, and the focus lens is set to the focal position (focus position) based on this defocus amount.

[0040] The pair of phase difference detection pixels described with reference to FIG. 2 are designated as pixel Pa and pixel Pb, and the details of the incident light to these pixels will be described with reference to FIG.

[0041] 3, the phase difference detection unit has phase difference detection pixels Pa and Pb arranged in the horizontal direction, each of which is composed of a pair of photodetectors (PD) that receive a light beam Ta from a right portion (also referred to as a "right partial pupil region" or simply "right pupil region") Qa of the exit pupil EY of the imaging optical system and a light beam Tb from a left portion (also referred to as a "left partial pupil region" or simply "left pupil region") Qb. Note that here, the +X direction side in the drawing is expressed as the right side, and the -X direction side is expressed as the left side.

[0042] Of the pair of phase difference detection pixels Pa, Pb, one phase difference detection pixel (hereinafter referred to as the "first phase difference detection pixel") Pa is composed of a microlens ML that collects light incident on the first phase difference detection pixel Pa, a first light-shielding plate AS1 having a slit (rectangular) first opening OP1, and a photoelectric conversion unit PD that is arranged below the first light-shielding plate AS1 and receives light that passes through a second light-shielding plate AS2 having a slit (rectangular) second opening OP2.

[0043] The first opening OP1 in the first phase difference detection pixel Pa is provided at a position offset in a specific direction (here, rightward (+X direction)) with respect to a central axis CL that passes through the center of the light receiving element PD and is parallel to the optical axis LT. The second opening OP2 in the first phase difference detection pixel Pa is provided at a position offset in a direction opposite to the specific direction (also referred to as the "anti-specific direction") with respect to the central axis CL.

[0044] Furthermore, of the pair of phase difference detection pixels Pa, Pb, the other phase difference detection pixel Pb (hereinafter referred to as the "second phase difference detection pixel") includes a first light-shielding plate AS1 having a slit-shaped first opening OP1 and a second light-shielding plate AS2 disposed below the first light-shielding plate AS1 and having a slit-shaped second opening OP2. The first opening OP1 in the second phase difference detection pixel Pb is provided at a position offset in the direction opposite to the specific direction with respect to the central axis CL. The second opening OP2 in the second phase difference detection pixel Pb is provided at a position offset in the specific direction with respect to the central axis CL.

[0045] That is, in the pair of phase difference detection pixels Pa and Pb, the first openings OP1 are arranged to be biased in different directions from each other, and the second openings OP2 are arranged to be shifted in different directions from the corresponding first openings OP1 in the phase difference detection pixels Pa and Pb.

[0046] The pair of phase difference detection pixels Pa and Pb having the above-described configuration acquires subject light that has passed through different regions (portions) in the exit pupil EY. Specifically, the light beam Ta that passes through the right pupil region Qa of the exit pupil EY passes through the microlens ML corresponding to the phase difference detection pixel Pa and the first opening OP1 of the first light shielding plate AS1, and is further restricted (limited) by the second light shielding plate AS2, before being received by the light receiving element PD of the first phase difference detection pixel Pa.

[0047] Furthermore, the light beam Tb that passes through the left pupil region Qb of the exit pupil EY passes through the microlens ML corresponding to the phase difference detection pixel Pb and the first opening OP1 of the second light shielding plate AS2, and is then further restricted by the second light shielding plate AS2 before being received by the light receiving element PD of the second phase difference detection pixel Pb.

[0048] An example of the output of the light receiving element acquired at each pixel Pa and Pb is shown in Fig. 4. As shown in Fig. 4, the output line from pixel Pa and the output line from pixel Pb are signals with a predetermined shift amount Sf.

[0049] FIG. 5(a) shows the shift amount Sfa that occurs between the pixels Pa and Pb when the focus lens is set at a position according to the subject distance and is in focus, that is, in an in-focus state. Figures 5(b1) and (b2) show the shift amount Sfa that occurs between the Pa and Pb pixels when the focus lens is not set to a position corresponding to the subject distance and is not in focus, i.e., in an out-of-focus state. (b1) is an example where the shift amount is larger than when in focus, and (b2) is an example where the shift amount is smaller than when in focus.

[0050] In the cases shown in FIGS. 5(b1) and 5(b2), it is possible to focus by moving the focus lens so that the shift amount corresponds to that during focusing. This process is the focusing process according to the "phase difference detection method." The focus lens can be set to the in-focus position by focusing processing based on this "phase difference detection method," and the focus lens can be set to a position according to the subject distance.

[0051] The shift amount described with reference to FIG. 5 can be measured for each set of pixels Pa and Pb, which are phase difference detection pixels configured in the image sensor shown in FIG. 2, and it becomes possible to individually calculate the focus position (focus point) and defocus amount for the subject image captured in this fine region (the region where the pixels Pa and Pb are combined).

[0052] In addition, the shift amount in cases such as those shown in Figures 5(b1) and (b2) differs from the shift amount during focusing, and the defocus amount, i.e., the defocus amount corresponding to the amount of deviation between the in-focus distance and the subject distance, can be calculated from this deviation amount.

[0053] [3. Basic configuration example for calculating the reliability of physical quantities for each image region, such as the defocus amount and distance value for each image region, and performing various processes according to the calculated reliability] Next, a basic configuration example will be described in which the reliability of a physical quantity in an image region unit, such as a defocus amount or distance value in an image region unit, is calculated, and various processes are executed according to the calculated reliability.

[0054] FIG. 6 shows an example of the internal configuration of the digital signal processing unit 108, which is a component of the imaging device 100 described with reference to FIG. The digital signal processing unit 108 shown in FIG. 6 calculates the reliability of physical quantities in units of image regions, such as the defocus amount and distance value in units of image regions, using detection information from the phase difference detection pixels, and performs various processes according to the calculated reliability.

[0055] As shown in FIG. 6 , the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, and an image output unit 213.

[0056] The digital signal processing unit 108 receives as input the RGB image signal from the A / D conversion unit 105 at the previous stage and the phase difference detection signal (detection information) that is the output of the phase difference detection pixel.

[0057] The phase difference information acquisition unit 201 shown in FIG. 6 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105. On the other hand, the image information acquisition unit 211 shown in FIG. 6 selects only the image signal (for example, the RGB image signal) from the input signal from the A / D conversion unit 105.

[0058] The image signal acquired by the image information acquisition unit 211 is input to the image signal processing unit 212 . The image signal processing unit 212 performs various image signal processes such as demosaic processing, white balance adjustment, gamma correction, etc. on the image signal, and outputs the processed image (for example, an RGB image) to the image output unit 213.

[0059] Note that the image (e.g., RGB image) generated by the image signal processing unit 212 may further undergo image control processing by the image region unit physical quantity reliability corresponding processing execution unit 206. For example, a process of superimposing reliability identification data of the defocus amount on the RGB image is performed. Specific examples of these processes will be described later.

[0060] An image (e.g., an RGB image) generated by the image signal processing unit 212, or an image that has been modified and processed by the image region unit physical quantity reliability corresponding processing execution unit 206 with respect to the image generated by the image signal processing unit 212, is output to the image output unit 213.

[0061] The image output unit 213 outputs the image input from the image signal processing unit 212. For example, the image output unit 213 outputs the image to the monitor 117, the viewfinder (EVF) 116, and the recording device 115.

[0062] The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105, and outputs the selected phase difference detection signal (detection information) to the defocus amount calculation unit 202.

[0063] The defocus amount calculation unit 202 calculates the amount of focus deviation in units of minute image areas, for example, in units of image areas each consisting of a plurality of pixels, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)). The defocus amount calculation unit 202 calculates the defocus amount, which is a physical amount that changes depending on the subject distance, for each image region.

[0064] As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the amount of deviation in the signals output in accordance with the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor.

[0065] The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to, for example, a focus position for a subject specified by the user based on this defocus amount, and outputs the generated AF control signal to the AF control unit 112a.

[0066] The AF control unit 112a drives the focus lens in accordance with an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and sets the focus lens to a focus position (focus position) for a subject designated by the user, for example.

[0067] Note that the subject set at the in-focus position (focus position) is not the entire image area of ​​the captured image, but a subject designated by the user, such as a person, etc. Images of other subjects, such as the background, are not in focus and appear blurred.

[0068] For example, the shift amount described above with reference to FIG. 5 is measured for each pair of pixels, i.e., the pixel Pa and the pixel Pb, which are phase difference detection pixels configured in the image sensor shown in FIG. That is, the defocus amount calculation unit 202 can calculate the defocus amount of the subject image for each minute image region of the captured image.

[0069] FIG. 7 shows an example of an image region that serves as a unit for calculating the defocus amount. FIG. 7 shows the pixel configuration of the image sensor 103 similar to that shown in FIG. 2 described above. As described above with reference to FIG. 2, the phase difference detection pixels 151 for acquiring phase difference information are discretely set in some (rows) of the RGB pixels having a Bayer array. The phase difference detection pixel is configured by a pair of a right-opening phase difference detection pixel Pa and a left-opening phase difference detection pixel Pb.

[0070] The image area that is the unit for calculating the defocus amount can be set, for example, like an image area 152 shown in FIG. In the example shown in FIG. 7, the image area 152 that is the calculation unit for the defocus amount is a minute image area of ​​n×m pixels, such as 6×5 pixels. This fine image region of n×m pixels includes a plurality of sets of phase difference detection pixels. The shift amount described above with reference to FIG. 5 is measured from each of these plurality of sets of phase difference detection pixels.

[0071] 7, the defocus amount calculation unit 202 calculates the average value of multiple sets of shift amounts within the image area 152 of n×m pixels as the shift amount of the image area 152. Furthermore, the defocus amount calculation unit 202 calculates the defocus amount from the deviation between the calculated shift amount and the shift amount of the focused image area, that is, the defocus amount corresponding to the deviation between the focus distance and the subject distance. In this way, the defocus amount calculation unit 202 calculates the defocus amount for each image region 152 as shown in FIG.

[0072] The defocus map generation unit 204 inputs the defocus amount for each fine image area of ​​the captured image calculated by the defocus amount calculation unit 202, and generates a defocus map that makes it possible to identify the defocus amount for each image area based on the defocus amount for each image area.

[0073] The defocus map generation unit 204 calculates the defocus amount for each image region 152 as shown in Figure 7, for example, based on the defocus amount for each image region 152 as shown in Figure 7 calculated by the defocus amount calculation unit 202, and generates a defocus map.

[0074] An example of a defocus map generated by the defocus map generating unit 204 will be described with reference to FIG.

[0075] Fig. 8(A) is an example of an image captured by the imaging device 100. Note that the captured image shown in Fig. 8(A) is not limited to only captured images recorded by a user's shutter operation on the imaging device 100, but also includes so-called through images that are input through the lens of the imaging device 100 and displayed on the monitor 117 or the like, regardless of whether the shutter operation is performed or not.

[0076] The defocus map in FIG. 8(B) is a defocus map corresponding to the captured image in (A), and is generated by the defocus map generating unit 204. As described above, the defocus map generation unit 204 calculates the defocus amount for each image region 152 as shown in Figure 7 based on the defocus amount for each image region 152 as shown in Figure 7 calculated by the defocus amount calculation unit 202, and generates a defocus map.

[0077] The rectangular area shown in the defocus map of FIG. 8(B) is an image area in units of defocus amount calculation, and corresponds to image area 152 shown in FIG. 7. Note that the rectangular area shown in the defocus map of FIG. 8(B) is shown in a larger size for ease of understanding. The image area as an actual unit of defocus amount calculation can be set as an even smaller image area. That is, as previously described with reference to FIG. 7, it can be set as a pixel area in units of several pixels to several tens of pixels.

[0078] The defocus map shown in FIG. 8B is a map in which luminance values ​​(pixel values) are set according to the defocus amount for each pixel region (rectangular region). The higher the luminance (white) of an area, the smaller the amount of defocus, that is, the higher the degree of focus of the pixel area. On the other hand, the lower the brightness (black) of an area, the larger the amount of defocus, that is, the lower the degree of focus of the pixel area.

[0079] For example, if the brightness value (pixel value) is set to 0 to 255, the closer the brightness value (pixel value) is to 255 (highest brightness (white)), the smaller the defocus amount, i.e., the higher the degree of focus of the pixel area, and the closer the brightness value (pixel value) is to 0 (lowest brightness (black)), the larger the defocus amount, i.e., the lower the degree of focus of the pixel area.

[0080] 8, (A) shows an example in which a person or a house shown in a captured image is set as a subject to be focused, and (B) shows a defocus map in which image regions corresponding to the person or the house are set as high-luminance (white) regions, indicating that they are pixel regions with small defocus amounts and high degrees of focus. On the other hand, background regions other than the person or the house are set as low-luminance (black) regions or gray regions, indicating that they are pixel regions with large defocus amounts and low degrees of focus.

[0081] In this way, the defocus map generation unit 204 calculates the defocus amount per image area based on the defocus amount per image area 152 calculated by the defocus amount calculation unit 202, for example, as shown in Figure 7, and generates a defocus map such as that shown in Figure 8(B).

[0082] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of a physical quantity in an image region unit that changes depending on the subject distance, such as a defocus amount or distance value in an image region unit.

[0083] For example, the image region unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and calculates the reliability of the defocus amount calculated for each image region based on these input data. Furthermore, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the distance value calculated from the defocus amount in image region units. Specific examples of these processes will be described later.

[0084] The reliability of the defocus amount in image region unit or the reliability of the distance value in image region unit calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability corresponding process execution unit 206.

[0085] The image region unit physical quantity reliability corresponding processing execution unit 206 executes various controls such as image control and shooting control on the captured image (e.g., RGB image) generated by the image signal processing unit 212, depending on the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit 205 or the reliability of the distance value for each image region.

[0086] As image control processing for the captured image (for example, an RGB image) generated by the image signal processing unit 212, for example, the following processing is executed. (a) Generation of an output image that allows identification of the reliability of the defocus amount for each image region; (b) generating an output image by superimposing the distance ratio data of the focused subject and the background subject on the captured image; (c) Generation of an image that allows identification of the stability of a mask set in a partial area of ​​the captured image; (d) generating an output image in which a color map that enables identification of the defocus amount for each image region is superimposed on the captured image; Furthermore, as a photographing control process, exposure control for each image area is performed according to the reliability of the defocus amount for each image area.

[0087] [4. Specific examples of calculating the reliability of physical quantities for each image region, such as defocus amount and distance value for each image region, and performing various processes according to the calculated reliability] Next, a specific example will be described in which the reliability of a physical quantity for each image region, such as a defocus amount or distance value for each image region, is calculated, and various processes are executed according to the calculated reliability.

[0088] The following several examples will be described as specific examples in which the imaging device of the present disclosure performs various processes according to the reliability of physical quantities per image area, such as the defocus amount and distance value per image area. (Example 1) Example of generating and outputting an image in which the reliability of the defocus amount in image area units can be identified (Example 2) Example of generating and outputting an image in which the distance ratio between the focused subject and other background subjects is superimposed on the captured image (Embodiment 3) An embodiment in which an image in which mask stability information is superimposed on a photographed image is generated and output (Embodiment 4) An embodiment in which an image is generated by superimposing a color map, which outputs colors according to the defocus amount in image area units, on a captured image and outputting the image (Embodiment 5) An embodiment in which a color corresponding to the defocus amount in image area units is output, and an image in which a color map that makes it possible to identify the reliability of the defocus amount in image area units is superimposed on the captured image is generated and output. (Embodiment 6) An embodiment in which an image is captured by controlling the exposure time according to the reliability of the defocus amount in each image area

[0089] [4-1. (Example 1) Example of generating and outputting an image in which the reliability of the defocus amount in image region units can be identified] First, as a first embodiment, an embodiment in which an image in which the reliability of the defocus amount in image region units can be identified is generated and output will be described.

[0090] FIG. 9 is a block diagram illustrating the configuration of the first embodiment. FIG. 9 shows an example of the internal configuration of the digital signal processing unit 108, which is a component of the imaging device 100 described with reference to FIG.

[0091] As shown in FIG. 9 , the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, and an image output unit 213.

[0092] The digital signal processing unit 108 receives as input the RGB image signal from the A / D conversion unit 105 at the previous stage and the phase difference detection signal (detection information) that is the output of the phase difference detection pixel.

[0093] The phase difference information acquisition unit 201 shown in FIG. 9 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105. On the other hand, the image information acquisition unit 211 shown in FIG. 9 selects only the image signal (for example, the RGB image signal) from the input signal from the A / D conversion unit 105.

[0094] The image signal acquired by the image information acquisition unit 211 is input to the image signal processing unit 212 . The image signal processing unit 212 performs various image signal processes such as demosaic processing, white balance adjustment, gamma correction, etc. on the image signal, and outputs the processed image (for example, an RGB image) to the image output unit 213.

[0095] In the first embodiment, the image region unit physical quantity reliability corresponding processing execution unit 206 further performs image control processing on the image (for example, an RGB image) generated by the image signal processing unit 212.

[0096] In the first embodiment, a process of superimposing the reliability identification data of the defocus amount onto the image (for example, an RGB image) generated by the image signal processing unit 212 is performed.

[0097] An image (e.g., an RGB image) generated by the image signal processing unit 212, or an image that has been modified and processed by the image region unit physical quantity reliability corresponding processing execution unit 206 with respect to the image generated by the image signal processing unit 212, is output to the image output unit 213.

[0098] The image output unit 213 outputs the image input from the image signal processing unit 212. For example, the image output unit 213 outputs the image to the monitor 117, the viewfinder (EVF) 116, and the recording device 115.

[0099] The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105, and outputs the selected phase difference detection signal (detection information) to the defocus amount calculation unit 202.

[0100] The defocus amount calculation unit 202 calculates the amount of focus deviation in units of minute image areas, for example, in units of image areas each consisting of a plurality of pixels, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)). The defocus amount calculation unit 202 calculates the defocus amount, which is a physical amount that changes depending on the subject distance, for each image region.

[0101] As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the amount of deviation in the signals output in accordance with the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor.

[0102] The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to, for example, a focus position for a subject specified by the user based on this defocus amount, and outputs the generated AF control signal to the AF control unit 112a.

[0103] The AF control unit 112a drives the focus lens in accordance with an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and sets the focus lens to a focus position (focus position) for a subject designated by the user, for example.

[0104] Note that the subject set at the in-focus position (focus position) is not the entire image area of ​​the captured image, but a subject designated by the user, such as a person, etc. Images of other subjects, such as the background, are not in focus and appear blurred.

[0105] As described above with reference to FIG. 7, the defocus amount calculation unit 202 calculates the defocus amount in units of a small image area, such as n×m pixels, i.e., the defocus amount corresponding to the deviation between the focus distance and the subject distance.

[0106] The defocus map generation unit 204 inputs the defocus amount for each fine image area of ​​the captured image calculated by the defocus amount calculation unit 202, and generates a defocus map that makes it possible to identify the defocus amount for each image area based on the defocus amount for each image area.

[0107] For example, a defocus map such as that described above with reference to FIG. 8(B) is generated. For example, in the case of a defocus map with a brightness value (pixel value) set to 0 to 255, the closer the brightness value (pixel value) is to 255 (highest brightness (white)), the smaller the defocus amount, i.e., the higher the degree of focus of the pixel area, and the closer the brightness value (pixel value) is to 0 (lowest brightness (black)), the larger the defocus amount, i.e., the lower the degree of focus of the pixel area. In this way, the defocus map generating unit 204 generates a defocus map such as that shown in FIG. 8(B), for example.

[0108] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region, which is a physical quantity for each image region. The image region unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and calculates the reliability of the defocus amount calculated for each image region based on this input data.

[0109] Note that various methods can be applied to the process of calculating the reliability of the defocus amount for each image region. For example, a method of calculating the reliability of the defocus amount for each image region using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, and the like can be used. It is also possible to use a method of calculating the reliability of the defocus amount for each image region using a cross-correlation function between two waveforms of parallax data used to calculate the defocus amount.

[0110] In the following, as one specific example of the reliability calculation process of the defocus amount for each image region executed by the image region unit physical quantity reliability calculation unit 205, a method using the cross-correlation function of two waveforms of disparity data used for calculating the defocus amount will be described.

[0111] This defocus amount reliability calculation method uses two waveforms of the disparity data shown in FIG. 4 used to calculate the defocus amount, i.e., a waveform indicating the output from pixel Pa and a waveform indicating the output from pixel Pb shown in FIG. 4, and the cross-correlation function of these two waveforms.

[0112] As described above with reference to FIG. 3, the pair of phase difference detection pixels Pa and Pb acquire subject light that has passed through different regions (portions) in the exit pupil EY. As shown in FIG. 4, the output of the light receiving element acquired at each pixel Pa and Pb is a signal having a predetermined shift amount Sf between the output line from pixel Pa and the output line from pixel Pb.

[0113] FIG. 5(a) shows the shift amount Sfa that occurs between the pixels Pa and Pb when the focus lens is set at a position according to the subject distance and is in focus, that is, in an in-focus state. Figures 5(b1) and (b2) show the shift amount Sfa that occurs between the Pa and Pb pixels when the focus lens is not set to a position corresponding to the subject distance and is not in focus, i.e., in an out-of-focus state.

[0114] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region using two waveforms of the disparity data used in the defocus amount calculation, i.e., a waveform indicating the output from pixel Pa and a waveform indicating the output from pixel Pb shown in Fig. 4, and a cross-correlation function between these two waveforms. Note that these two waveform data, the waveform indicating the output from pixel Pa and the waveform indicating the output from pixel Pb, are acquired by the phase information acquisition unit 201 and input to the image region unit physical quantity reliability calculation unit 205 via the defocus amount calculation unit 202.

[0115] The image region unit physical quantity reliability calculation unit 205 calculates the defocus amount reliability from the two waveform data as follows. The waveform data representing the output from pixel Pa shown in Figure 4 is f(t), and the waveform data representing the output from pixel Pb is g(t). The cross-correlation function h(τ) of these two waveform data f(t) and g(t) can be calculated using the following equation (1), where t represents the position within each pixel Pa or Pb.

[0116]

number

[0117] In the above (Equation 1), the time (τ) at which the cross-correlation function h(τ) reaches its maximum value is defined as τ max As h(τ max ) can be calculated as the reliability of the defocus amount.

[0118] The defocus amount reliability calculated in this example is the reliability of the defocus amount in pixel region units. As described above with reference to Fig. 7, a plurality of sets of phase difference detection pixels exist in one image region 152. The defocus amount calculation unit 202 calculates the average value of the shift amounts of the plurality of sets in the image region 152 as shown in Fig. 7, for example, as the shift amount of the image region 152 of n × m pixels. Furthermore, the defocus amount is calculated from the deviation between the calculated shift amount and the shift amount of the focused image region, that is, the defocus amount corresponding to the deviation between the focus distance and the subject distance.

[0119] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount per image region using the defocus amount per image region calculated by the defocus amount calculation unit 202, i.e., the defocus amount per image region, which is one rectangular region in the defocus map shown in Figure 8(B). The waveform data f(t) and g(t) that form the basis for calculating the reliability are obtained by calculating, for example, an average waveform of the waveform data f(t) and an average waveform of the waveform data g(t) of each set of a plurality of phase difference detection pixels included in the image region that is the target of reliability calculation.

[0120] In this way, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount calculated by the defocus amount calculation unit 202 for each image region, for example, by using the cross-correlation function of the two waveforms of the disparity data used to calculate the defocus amount.

[0121] The reliability data of the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability corresponding process execution unit 206 .

[0122] The image region unit physical quantity reliability corresponding processing execution unit 206 executes image control on the captured image (e.g., RGB image) generated by the image signal processing unit 212, according to the reliability data of the defocus amount on an image region unit basis calculated by the image region unit physical quantity reliability calculation unit 205. Specifically, the image generating unit 203 performs processing to generate an image in which the reliability of the defocus amount calculated by the image region unit physical quantity reliability calculation unit 205 can be identified.

[0123] That is, the image region unit physical quantity reliability corresponding processing execution unit 206 executes processing to generate an output image in which low reliability regions and high reliability regions of the defocus amount calculated on an image region unit basis can be identified as image control processing for the image (e.g., RGB image) generated by the image signal processing unit 212. A specific example of this process will be described with reference to FIG. 10 and subsequent figures.

[0124] Figure 10 shows the following data: (A) Photographed image (B) Defocus map

[0125] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The (B) defocus map is a defocus map generated by the defocus map generation unit 204. That is, it is a defocus map in which the defocus amount for each image region generated by the defocus map generation unit 204 is output as a brightness value (for example, 0 to 255) based on the defocus amount for each image region calculated by the defocus amount calculation unit 202 based on the pixel value signals of the phase difference detection pixels acquired by the phase difference information acquisition unit 201.

[0126] As described above, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region unit of the defocus map generated by the defocus map generation unit 204, and outputs the calculated reliability data to the image region unit physical quantity reliability corresponding processing execution unit 206.

[0127] The image region unit physical quantity reliability corresponding processing execution unit 206 extracts, for example, a defocus amount low reliability region and a defocus amount high reliability region as shown in FIG. 10(B) based on the reliability data of the defocus amount in image region units input from the image region unit physical quantity reliability calculation unit 205.

[0128] Specifically, the defocus amount reliability for each image region input from the image region unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold value to extract a low defocus amount reliability region and a high defocus amount reliability region. For example, using a predetermined low-reliability threshold Th1 and a high-reliability threshold Th2, (Determination formula 1) Defocus amount reliability≦Th1 An image region that satisfies the above-mentioned judgment formula 1 is extracted as a defocus amount low reliability region.

[0129] moreover, (Determination formula 2) Th2≦Defocus amount reliability An image region that satisfies the above-mentioned judgment formula 2 is extracted as a defocus amount high reliability region.

[0130] The image region unit physical quantity reliability corresponding processing execution unit 206 extracts low-reliability regions of defocus amount and high-reliability regions of defocus amount in accordance with these judgment formulas, and executes processing to generate an output image in which low-reliability regions and high-reliability regions of the defocus amount calculated in image region units can be identified based on the extraction results. A specific example will be described with reference to FIG. 11 and subsequent figures.

[0131] Figure 11 shows the following data: (A) Photographed image (B) Defocus map (C) Output image

[0132] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . (B) The defocus map is a defocus map generated by the defocus map generating unit 204. (C) Output image is an example of an output image generated by the image region unit physical quantity reliability correspondence processing execution unit 206 based on (A) captured image and (B) defocus map.

[0133] The example of the output image (C) shown in FIG. 11 is an example of an output image in which the defocus amount low reliability region is identifiable.

[0134] The image region unit physical quantity reliability corresponding process execution unit 206 (B) determines whether the reliability of the defocus amount in image region unit set in the defocus map is equal to or lower than the low reliability threshold value Th1 described above, that is, (Determination formula 1) Defocus amount reliability≦Th1 An image region that satisfies the above-mentioned judgment formula 1 is extracted as a defocus amount low reliability region, and graphic data that makes this low reliability region identifiable is superimposed on the captured image (A) to generate an output image (C).

[0135] The defocus amount low reliability region shown in the output image of Figure 11(C) is composed of graphic data of, for example, multiple semi-transparent red rectangular blocks. Each rectangular block corresponds to one image region, which is a unit for calculating the defocus amount. Note that red is just an example, and other colors may be used.

[0136] This (C) output image is output to the monitor 117 of the imaging device 100, for example. The user can view the image output on the monitor 117 and easily check areas where the defocus amount may not have been calculated correctly.

[0137] The output image (C) in Fig. 11 is an example of an output image in which a low-reliability defocus amount region is identifiable. Next, with reference to Fig. 12, an example of an output image in which a high-reliability defocus amount region is identifiable will be described. As in FIG. 11, FIG. 12 shows the following data: (A) Photographed image (B) Defocus map (C) Output image

[0138] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . (B) The defocus map is a defocus map generated by the defocus map generating unit 204. (C) Output image is an example of an output image generated by the image region unit physical quantity reliability correspondence processing execution unit 206 based on (A) captured image and (B) defocus map.

[0139] The example of the output image (C) shown in FIG. 12 is an example of an output image in which a highly reliable defocus amount region can be identified.

[0140] The image region unit physical quantity reliability corresponding process execution unit 206 (B) selects a region where the reliability of the defocus amount in image region unit set in the defocus map is equal to or higher than the high reliability threshold value Th2 described above, that is, (Determination formula 2) Th2≦Defocus amount reliability An image region that satisfies the above-mentioned judgment formula 2 is extracted as a defocus amount high reliability region, and graphic data that makes this high reliability region identifiable is superimposed on the captured image (A) to generate an output image (C).

[0141] The defocus amount high reliability region shown in the output image of Figure 12(C) is configured, for example, by graphic data of multiple semi-transparent blue rectangular blocks. Each rectangular block corresponds to one image region, which is a unit for calculating the defocus amount. Note that blue is just an example, and other colors may be set.

[0142] This (C) output image is output to the monitor 117 of the imaging device 100, for example. The user can view the image output on the monitor 117 and easily check the area where the defocus amount has been calculated correctly.

[0143] In this way, this embodiment 1 is an embodiment in which an image is generated and output in which graphic data that makes it possible to identify the reliability of the defocus amount for each image area is superimposed on the captured image, and the user can distinguish and confirm areas in which the defocus amount has been correctly calculated from areas in which it has not been correctly calculated by using the graphic data superimposed on the captured image.

[0144] Note that the digital signal processing unit 108 described with reference to FIG. 9 is configured to include all of the components, namely, the phase difference information acquisition unit 201, the defocus amount calculation unit 202, the AF control signal generation unit 203, the defocus map generation unit 204, the image region unit physical quantity reliability calculation unit 205, the image region unit physical quantity reliability corresponding processing execution unit 206, the image information acquisition unit 211, the image signal processing unit 212, and the image output unit 213, within the digital signal processing unit 108; however, this configuration is merely an example.

[0145] 9 may be configured as a part of the digital signal processing unit 108 of the image capturing device 100. Also, the data processing may be performed in an external device other than the image capturing device 100. Specifically, for example, the image signal processing unit 212 can be configured outside the digital signal processing unit 108. Furthermore, the image signal processing may be performed in an external device other than the imaging device 100, such as a PC.

[0146] [4-2. (Example 2) Example of generating and outputting an image in which the distance ratio between a focused subject and other background subjects is superimposed on a captured image] Next, as a second embodiment, an embodiment in which an image in which the distance ratio between the focused object and other background objects is superimposed on a captured image is generated and output will be described.

[0147] FIG. 13 is a block diagram illustrating the configuration of the second embodiment. FIG. 13 shows an example of the internal configuration of the digital signal processing unit 108, which is a component of the image pickup device 100 described with reference to FIG.

[0148] As shown in FIG. 13 , the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, an image output unit 213, and a distance information calculation unit 221.

[0149] The digital signal processing unit 108 receives as input the RGB image signal from the A / D conversion unit 105 at the previous stage and the phase difference detection signal (detection information) that is the output of the phase difference detection pixel.

[0150] The phase difference information acquisition unit 201 shown in FIG. 13 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105. On the other hand, the image information acquisition unit 211 shown in FIG. 13 selects only the image signal (for example, the RGB image signal) from the input signal from the A / D conversion unit 105.

[0151] The image signal acquired by the image information acquisition unit 211 is input to the image signal processing unit 212 . The image signal processing unit 212 performs various image signal processes such as demosaic processing, white balance adjustment, gamma correction, etc. on the image signal, and outputs the processed image (for example, an RGB image) to the image output unit 213.

[0152] In the second embodiment, similarly to the first embodiment described above, the image control processing by the image region unit physical quantity reliability corresponding processing execution unit 206 is performed on the image (for example, an RGB image) generated by the image signal processing unit 212.

[0153] In the second embodiment, an image in which the distance ratio between the focused subject and other background subjects is superimposed on an image (for example, an RGB image) generated by the image signal processing unit 212 is generated and output.

[0154] An image (e.g., an RGB image) generated by the image signal processing unit 212, or an image that has been modified and processed by the image region unit physical quantity reliability corresponding processing execution unit 206 with respect to the image generated by the image signal processing unit 212, is output to the image output unit 213.

[0155] The image output unit 213 outputs the image input from the image signal processing unit 212. For example, the image output unit 213 outputs the image to the monitor 117, the viewfinder (EVF) 116, and the recording device 115.

[0156] The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105, and outputs the selected phase difference detection signal (detection information) to the defocus amount calculation unit 202.

[0157] The defocus amount calculation unit 202 calculates the amount of focus deviation in units of minute image areas, for example, in units of image areas each consisting of a plurality of pixels, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)). As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the amount of deviation in the signals output in accordance with the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor.

[0158] The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to, for example, a focus position for a subject specified by the user based on this defocus amount, and outputs the generated AF control signal to the AF control unit 112a.

[0159] The AF control unit 112a drives the focus lens in accordance with an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and sets the focus lens to a focus position (focus position) for a subject designated by the user, for example.

[0160] Note that the subject set at the in-focus position (focus position) is not the entire image area of ​​the captured image, but a subject designated by the user, such as a person, etc. Images of other subjects, such as the background, are not in focus and appear blurred.

[0161] As described above with reference to FIG. 7, the defocus amount calculation unit 202 calculates the defocus amount in units of a small image area, such as n×m pixels, i.e., the defocus amount corresponding to the deviation between the focus distance and the subject distance.

[0162] The distance information calculation unit 221 receives the defocus amount for each image area of ​​the captured image calculated by the defocus amount calculation unit 202, and calculates distance information for each image area of ​​the captured image based on the defocus amount for each image area. The distance information calculation unit 221 calculates a distance value, which is a physical quantity that changes depending on the subject distance, for each image region. The distance information calculation unit 221 generates a depth map that indicates, for example, distance values ​​in image region units using pixel values ​​(for example, 0 to 255).

[0163] The (B) distance information (depth map) for each image area generated by the distance information calculation unit 221 is a depth map that shows the value of the subject distance for each image area calculated based on the defocus amount for each tiny image area, such as n x m pixels, calculated by the defocus amount calculation unit 202, as a pixel value (e.g., 0 to 255). A high brightness (high pixel value) region is a region where the subject distance is close, and a low brightness (low pixel value) region is a region where the subject distance is far.

[0164] The process of calculating the subject distance from the defocus amount can be performed by applying parameters such as the focal length of the lens (focus lens) of the imaging device. Specifically, the subject distance for each image region is calculated according to the following (Equation 2).

[0165]

number

[0166] The distance information calculation unit 221 calculates the subject distance for each image region in accordance with the above (Equation 2).

[0167] The subject distance information for each image region calculated by the distance information calculation unit 221 is output to the image region unit physical quantity reliability calculation unit 205 .

[0168] The defocus map generation unit 204 inputs the defocus amount for each fine image area of ​​the captured image calculated by the defocus amount calculation unit 202, and generates a defocus map that makes it possible to identify the defocus amount for each image area based on the defocus amount for each image area.

[0169] For example, a defocus map such as that described above with reference to FIG. 8(B) is generated. For example, in the case of a defocus map with a brightness value (pixel value) set to 0 to 255, the closer the brightness value (pixel value) is to 255 (highest brightness (white)), the smaller the defocus amount, i.e., the higher the degree of focus of the pixel area, and the closer the brightness value (pixel value) is to 0 (lowest brightness (black)), the larger the defocus amount, i.e., the lower the degree of focus of the pixel area. In this way, the defocus map generating unit 204 generates a defocus map such as that shown in FIG. 8(B), for example.

[0170] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region, which is a physical quantity for each image region, and the reliability of the distance information for each image region.

[0171] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the distance information for each image region as a reliability corresponding to the reliability of the defocus amount for each image region. That is, image areas with low reliability of the defocus amount are judged to have low reliability of the distance value calculated based on the defocus amount, and image areas with high reliability of the defocus amount are judged to have high reliability of the distance value calculated based on the defocus amount.

[0172] As in the previous first embodiment, the image region unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and calculates the reliability of the defocus amount calculated for each image region based on these input data.

[0173] As explained above in the first embodiment, various methods can be applied as a method for calculating the reliability of the defocus amount for each image region. For example, a method of calculating the reliability of the defocus amount for each image region using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, and the like can be used. Also, the method using (Equation 1) described in the first embodiment, that is, the method of calculating the defocus amount reliability for each image region using the cross-correlation function of two waveforms of disparity data used for calculating the defocus amount, can be used.

[0174] Note that the defocus amount reliability calculated in the second embodiment is also the reliability of the defocus amount in pixel region units, as explained in the first embodiment. The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount per image region using the defocus amount per image region calculated by the defocus amount calculation unit 202, i.e., the defocus amount per image region, which is one rectangular region in the defocus map shown in FIG. 8(B).

[0175] In this way, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount calculated by the defocus amount calculation unit 202 for each image region, for example, by using the cross-correlation function of the two waveforms of the disparity data used to calculate the defocus amount.

[0176] Furthermore, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the distance information in image region units according to the reliability of the defocus amount in image region units. As described above, the reliability of the distance information for each image region is calculated as a reliability corresponding to the reliability of the defocus amount for each image region. That is, image areas with low reliability of the defocus amount are judged to have low reliability of the distance value calculated based on the defocus amount, and image areas with high reliability of the defocus amount are judged to have high reliability of the distance value calculated based on the defocus amount.

[0177] The reliability data of the distance information in image region units calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability corresponding process execution unit 206 .

[0178] The image region unit physical quantity reliability corresponding processing execution unit 206 executes image control on the captured image (e.g., RGB image) generated by the image signal processing unit 212, according to the reliability data of the distance information on an image region unit basis calculated by the image region unit physical quantity reliability calculation unit 205. Specifically, depending on the reliability of the distance information for each image region calculated by the image region unit physical quantity reliability calculation unit 205, distance ratio data between regions having highly reliable distance information is generated and displayed superimposed on the captured image. For example, image areas having distance values ​​with reliability equal to or greater than a predetermined threshold value are selected, and distance ratio data between subjects in the selected image areas is generated and displayed superimposed on the captured image.

[0179] A specific example of this process will be described with reference to FIG.

[0180] Figure 14 shows the following data: (A) Photographed image (B) Output image (p) Separation degree (distance ratio) calculation example

[0181] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The output image (B) is an example of an output image generated by the image region unit physical quantity reliability correspondence processing execution unit 206 based on the captured image (A).

[0182] (B) The degree of separation (= distance ratio) between the focused subject and the background subject is superimposed on the upper left of the output image. This separation degree (=distance ratio) data is data generated by the image region unit physical quantity reliability calculation unit 205. The image region unit physical quantity reliability calculation unit 205 selects the focused subject region and a part of the background region as regions having highly reliable distance information according to the reliability of the distance information in image region units, calculates the distance ratio between these regions, and displays this as separation degree (distance ratio) data by superimposing it on the captured image.

[0183] An example of calculation of the separation degree (distance ratio) data is shown in Fig. 14(p) Separation degree (distance ratio) calculation example. That is, for example, let a be the distance from the camera to the focused subject, and b be the distance from the camera to the background area. In this case, the degree of separation (distance ratio) between the focused object and the background area is b / a. The upper left corner of the output image in Figure 14(B) shows the calculated Separation degree (distance ratio) = b / a is displayed.

[0184] This (B) output image is output to the monitor 117 of the imaging device 100, for example. The user can view the image output on the monitor 117 and easily confirm the degree of separation (distance ratio) between regions whose distances have been correctly calculated.

[0185] In this way, this second embodiment is an embodiment in which the reliability of distance information is calculated for each image region, and an image is generated and output in which data that enables confirmation of the degree of separation (distance ratio) between regions with highly reliable distance information is superimposed on the captured image.By looking at the separation (distance ratio) data superimposed on the captured image, the user can easily confirm the degree of separation (distance ratio) between regions whose distances have been correctly calculated.

[0186] Note that the digital signal processing unit 108 described with reference to FIG. 13 is configured to include all of the following components within the digital signal processing unit 108: a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, an image output unit 213, and a distance information calculation unit 221; however, this configuration is merely an example.

[0187] 13 may be configured as a part of the digital signal processing unit 108 of the imaging device 100. Also, a configuration may be adopted in which data processing is performed in an external device different from the imaging device 100. Specifically, for example, the image signal processing unit 212 can be configured outside the digital signal processing unit 108. Furthermore, the image signal processing may be performed in an external device other than the imaging device 100, such as a PC.

[0188] [4-3. (Example 3) Example of generating and outputting an image in which mask stability information is superimposed on a captured image] Next, as a third embodiment, an embodiment in which an image in which mask stability information is superimposed on a captured image is generated and output will be described.

[0189] For example, a masking process may be performed in which the background area other than a specific subject selected from the photographed image is set to a uniform color such as a green background or a blue background. By compositing another image, for example, a new background image, into the green or blue background area of ​​the image that has been subjected to such masking processing, it is possible to generate a composite image in which the selected specific subject is displayed on the new background image. In this embodiment, when such mask processing is performed, mask stability information indicating whether or not highly accurate mask setting is possible is generated and output by being superimposed on the captured image.

[0190] FIG. 15 is a block diagram illustrating the configuration of the third embodiment. FIG. 15 shows an example of the internal configuration of the digital signal processing unit 108, which is a component of the imaging device 100 described with reference to FIG.

[0191] As shown in FIG. 15 , the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, and an image output unit 213.

[0192] The digital signal processing unit 108 receives as input the RGB image signal from the A / D conversion unit 105 at the previous stage and the phase difference detection signal (detection information) that is the output of the phase difference detection pixel.

[0193] The phase difference information acquisition unit 201 shown in FIG. 15 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105. On the other hand, the image information acquisition unit 211 shown in FIG. 15 selects only the image signal (for example, the RGB image signal) from the input signal from the A / D conversion unit 105.

[0194] The image signal acquired by the image information acquisition unit 211 is input to the image signal processing unit 212 . The image signal processing unit 212 performs various image signal processes such as demosaic processing, white balance adjustment, gamma correction, etc. on the image signal, and outputs the processed image (for example, an RGB image) to the image output unit 213.

[0195] Also in this embodiment 3, similarly to the previously described embodiments 1 and 2, the image region unit physical quantity reliability corresponding processing executing unit 206 further performs image control processing on the image (e.g., RGB image) generated by the image signal processing unit 212.

[0196] In the third embodiment, a process is performed in which mask stability information according to the reliability of the defocus amount is superimposed on an image (for example, an RGB image) generated by the image signal processing unit 212 and output.

[0197] An image (e.g., an RGB image) generated by the image signal processing unit 212, or an image that has been modified and processed by the image region unit physical quantity reliability corresponding processing execution unit 206 with respect to the image generated by the image signal processing unit 212, is output to the image output unit 213.

[0198] The image output unit 213 outputs the image input from the image signal processing unit 212. For example, the image output unit 213 outputs the image to the monitor 117, the viewfinder (EVF) 116, and the recording device 115.

[0199] The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105, and outputs the selected phase difference detection signal (detection information) to the defocus amount calculation unit 202.

[0200] The defocus amount calculation unit 202 calculates the amount of focus deviation in units of minute image areas, for example, in units of image areas each consisting of a plurality of pixels, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)). As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the amount of deviation in the signals output in accordance with the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor.

[0201] The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to, for example, a focus position for a subject specified by the user based on this defocus amount, and outputs the generated AF control signal to the AF control unit 112a.

[0202] The AF control unit 112a drives the focus lens in accordance with an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and sets the focus lens to a focus position (focus position) for a subject designated by the user, for example.

[0203] Note that the subject set at the in-focus position (focus position) is not the entire image area of ​​the captured image, but a subject designated by the user, such as a person, etc. Images of other subjects, such as the background, are not in focus and appear blurred.

[0204] As described above with reference to FIG. 7, the defocus amount calculation unit 202 calculates the defocus amount in units of a small image area, such as n×m pixels, i.e., the defocus amount corresponding to the deviation between the focus distance and the subject distance.

[0205] The defocus map generation unit 204 inputs the defocus amount for each fine image area of ​​the captured image calculated by the defocus amount calculation unit 202, and generates a defocus map that makes it possible to identify the defocus amount for each image area based on the defocus amount for each image area.

[0206] For example, a defocus map such as that described above with reference to FIG. 8(B) is generated. For example, in the case of a defocus map with a brightness value (pixel value) set to 0 to 255, the closer the brightness value (pixel value) is to 255 (highest brightness (white)), the smaller the defocus amount, i.e., the higher the degree of focus of the pixel area, and the closer the brightness value (pixel value) is to 0 (lowest brightness (black)), the larger the defocus amount, i.e., the lower the degree of focus of the pixel area. In this way, the defocus map generating unit 204 generates a defocus map such as that shown in FIG. 8(B), for example.

[0207] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region, which is a physical quantity for each image region. As in the previous first embodiment, the image region unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and calculates the reliability of the defocus amount calculated for each image region based on these input data.

[0208] As explained above in the first embodiment, various methods can be applied as a method for calculating the reliability of the defocus amount for each image region. For example, a method of calculating the reliability of the defocus amount for each image region using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, and the like can be used. Also, the method using (Equation 1) described in the first embodiment, that is, the method of calculating the defocus amount reliability for each image region using the cross-correlation function of two waveforms of disparity data used for calculating the defocus amount, can be used.

[0209] Note that the defocus amount reliability calculated in the third embodiment is also the reliability of the defocus amount in pixel region units, as explained in the first embodiment. The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount per image region using the defocus amount per image region calculated by the defocus amount calculation unit 202, i.e., the defocus amount per image region, which is one rectangular region in the defocus map shown in FIG. 8(B).

[0210] In this way, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount calculated by the defocus amount calculation unit 202 for each image region, for example, by using the cross-correlation function of the two waveforms of the disparity data used to calculate the defocus amount.

[0211] The reliability data of the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability corresponding process execution unit 206 .

[0212] The image region unit physical quantity reliability corresponding processing execution unit 206 executes image control on the captured image (e.g., RGB image) generated by the image signal processing unit 212, according to the reliability data of the defocus amount on an image region unit basis calculated by the image region unit physical quantity reliability calculation unit 205. Specifically, mask stability information is generated according to the reliability of the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205, and is output by being superimposed on the captured image.

[0213] The mask stability information is information indicating whether or not a mask can be accurately set when a mask is set in an area other than the selected subject of a photographed image, for example, in a background area. As mentioned above, for example, by performing a masking process to set the background area other than a specific subject selected from the captured image to a uniform color such as a green or blue background, and then compositing another image, for example a new background image, into the green or blue background area of ​​the masked image, it is possible to generate a composite image in which the selected specific subject is displayed on the new background image.

[0214] In this embodiment, when such mask processing is performed, mask stability information indicating whether or not highly accurate mask setting is possible is generated and output by being superimposed on the captured image. The image region unit physical quantity reliability corresponding processing execution unit 206 generates mask stability information according to the reliability of the defocus amount in image region unit calculated by the image region unit physical quantity reliability calculation unit 205, and outputs the mask stability information by superimposing it on the captured image. A specific example of this process will be described with reference to FIG. 16 and subsequent figures.

[0215] First, an example of mask processing for a captured image and composite image generation processing will be described with reference to FIG. For example, the captured image in Figure 16(A) is an image in which a person and a house are set as the focused subjects. The defocus amount of this focused subject is almost 0. A masking process is performed in which the background area outside this focused subject area is set as a mask area and set to a uniform color such as a green or blue background. This mask setting process generates the masked image (B).

[0216] Next, another image, for example, a new background image, is composited into the green or blue screen area of ​​the masked image. This process makes it possible to generate a composite image such as that shown in Figure 16(C). To generate such a composite image, it is essential to set an accurate mask area.

[0217] For example, the following process can be performed as a specific mask area determination process when an area where no mask is set is set as a focused area such as a person or a house, and a background area other than the focused area is set as a mask area. The area where the defocus amount of the image area unit is equal to or less than a predetermined threshold value Th, i.e., (Determination formula a) Defocus amount≦Th An area that satisfies this determination formula a is determined to be an in-focus area and is determined to be an area that is not to be masked. On the other hand, an area that does not satisfy the determination formula a is determined to be an out-of-focus area and is determined as a mask area to be masked. By this process, the mask area can be determined.

[0218] However, if the defocus amount is an inaccurate value, it will not be possible to determine an accurate mask area even if the mask area determination process is performed using the above (Determination formula a). In this embodiment, when performing mask processing, mask stability information indicating whether high-precision mask setting is possible is generated and output by superimposing it on the captured image, thereby making it possible to notify the user whether stable mask area setting is possible.

[0219] The image region unit physical quantity reliability corresponding processing execution unit 206 generates mask stability information according to the reliability of the defocus amount in image region unit calculated by the image region unit physical quantity reliability calculation unit 205, and outputs the mask stability information by superimposing it on the captured image. A specific processing example will be described with reference to FIG. 17 and subsequent figures.

[0220] FIG. 17 shows the following data: (A) Photographed image (B) Defocus map (C) Output image

[0221] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The (B) defocus map is a defocus map generated by the defocus map generation unit 204. That is, it is a defocus map in which the defocus amount for each image region generated by the defocus map generation unit 204 is output as a brightness value (for example, 0 to 255) based on the defocus amount for each image region calculated by the defocus amount calculation unit 202 based on the pixel value signals of the phase difference detection pixels acquired by the phase difference information acquisition unit 201.

[0222] (C) Output image is an example of an output image generated by the image region unit physical quantity reliability correspondence processing execution unit 206 based on (A) captured image. (C) Mask stability information is displayed in the top right corner of the output image.

[0223] As described above, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region unit of the defocus map generated by the defocus map generation unit 204, and outputs the calculated reliability data to the image region unit physical quantity reliability corresponding processing execution unit 206.

[0224] The image region unit physical quantity reliability corresponding processing execution unit 206 extracts, for example, a defocus amount low reliability region as shown in FIG. 17(B) based on the reliability data of the defocus amount in image region unit input from the image region unit physical quantity reliability calculation unit 205.

[0225] Specifically, the defocus amount reliability for each image region input from the image region unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold value to extract a defocus amount low reliability region. For example, using a predefined low-reliability threshold Th1, (Determination formula 1) Defocus amount reliability≦Th1 An image region that satisfies the above-mentioned judgment formula 1 is extracted as a defocus amount low reliability region.

[0226] The defocus map shown in FIG. 17(B) includes a region with low reliability of the defocus amount. In this way, when a defocus amount low reliability region is detected, the image region unit physical quantity reliability corresponding process executing unit 206 determines that stable mask region determination process is difficult. Furthermore, in accordance with this determination, an output image is generated in which mask stability information indicating "mask unstable" is superimposed on the captured image, as shown in Fig. 17(C) and output.

[0227] The example of output image (C) shown in Figure 17 is an example of an output image in which a low-reliability defocus amount area has been detected and mask stability information (= "Mask Unstable") indicating that it is difficult to determine a stable mask area has been superimposed on the captured image.

[0228] This (C) output image is output to the monitor 117 of the imaging device 100, for example. The user can see the image output on the monitor 117 and recognize that it is difficult to set a stable mask.

[0229] Next, a processing example in which stable mask setting is possible will be described with reference to FIG. As in FIG. 17, FIG. 18 also shows the following data: (A) Photographed image (B) Defocus map (C) Output image

[0230] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The (B) defocus map is a defocus map generated by the defocus map generation unit 204. That is, it is a defocus map in which the defocus amount for each image region generated by the defocus map generation unit 204 is output as a brightness value (for example, 0 to 255) based on the defocus amount for each image region calculated by the defocus amount calculation unit 202 based on the pixel value signals of the phase difference detection pixels acquired by the phase difference information acquisition unit 201.

[0231] (C) Output image is an example of an output image generated by the image region unit physical quantity reliability correspondence processing execution unit 206 based on (A) captured image. (C) Mask stability information is displayed in the top right corner of the output image.

[0232] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region unit of the defocus map generated by the defocus map generation unit 204, and outputs the calculated reliability data to the image region unit physical quantity reliability corresponding processing execution unit 206.

[0233] The example shown in FIG. 18 is an example in which a low-reliability defocus amount region is not detected from the defocus map shown in FIG. 18(B). That is, as explained earlier, (Determination formula 1) Defocus amount reliability≦Th1 This is an example of a case where an image area that satisfies the above-mentioned criterion 1 is not detected.

[0234] The defocus map shown in FIG. 18(B) does not include any low-reliability defocus amount regions. In this way, when no defocus amount low reliability region is detected, the image region unit physical quantity reliability corresponding process executing unit 206 determines that stable mask region determination process is possible. Furthermore, in accordance with this determination, an output image is generated in which mask stability information indicating "mask stable" is superimposed on the captured image, as shown in Fig. 18(C) and output.

[0235] The example of output image (C) shown in Figure 18 is an example of an output image in which mask stability information (="Mask Stable") indicating that no low-reliability defocus area was detected and a stable mask area can be determined is superimposed on the captured image.

[0236] This (C) output image is output to the monitor 117 of the imaging device 100, for example. The user can see the image output on the monitor 117 and recognize that a stable mask setting is possible.

[0237] The examples of mask stability information shown in FIGS. 17 and 18 are merely examples, and it is possible to display the mask stability information using various other characters, symbols, icons, icon displays of different colors, etc.

[0238] Furthermore, the digital signal processing unit 108 described with reference to FIG. 15 is configured to include all of the components, namely, the phase difference information acquisition unit 201, the defocus amount calculation unit 202, the AF control signal generation unit 203, the defocus map generation unit 204, the image region unit physical quantity reliability calculation unit 205, the image region unit physical quantity reliability corresponding processing execution unit 206, the image information acquisition unit 211, the image signal processing unit 212, and the image output unit 213, within the digital signal processing unit 108, but this configuration is just one example.

[0239] 15 may be configured as a part of the digital signal processing unit 108 of the imaging device 100. Also, a configuration may be adopted in which data processing is performed in an external device different from the imaging device 100. Specifically, for example, the image signal processing unit 212 can be configured outside the digital signal processing unit 108. Furthermore, the image signal processing may be performed in an external device other than the imaging device 100, such as a PC.

[0240] [4-4. (Example 4) Example of generating and outputting an image in which a color map outputting colors according to defocus amounts in image area units is superimposed on a captured image] Next, as a fourth embodiment, an embodiment will be described in which an image is generated and output by superimposing a color map, which outputs colors according to defocus amounts in image region units, on a captured image.

[0241] For example, in a captured image, there are subjects with various defocus amounts, such as a focused subject with a defocus amount of almost 0, and a background subject with a large defocus amount. In this embodiment, an output image in which a color corresponding to the defocus amount is set for a captured image is generated and output.

[0242] FIG. 19 is a block diagram illustrating the configuration of the fourth embodiment. FIG. 19 shows an example of the internal configuration of the digital signal processing unit 108, which is a component of the imaging device 100 described with reference to FIG.

[0243] As shown in FIG. 19 , the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, and an image output unit 213.

[0244] The digital signal processing unit 108 receives as input the RGB image signal from the A / D conversion unit 105 at the previous stage and the phase difference detection signal (detection information) that is the output of the phase difference detection pixel.

[0245] The phase difference information acquisition unit 201 shown in FIG. 19 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105. On the other hand, the image information acquisition unit 211 shown in FIG. 19 selects only the image signal (for example, the RGB image signal) from the input signal from the A / D conversion unit 105.

[0246] The image signal acquired by the image information acquisition unit 211 is input to the image signal processing unit 212 . The image signal processing unit 212 performs various image signal processes such as demosaic processing, white balance adjustment, gamma correction, etc. on the image signal, and outputs the processed image (for example, an RGB image) to the image output unit 213.

[0247] Also in this embodiment 4, similarly to the previously described embodiments 1 to 3, the image region unit physical quantity reliability corresponding processing executing unit 206 further performs image control processing on the image (e.g., RGB image) generated by the image signal processing unit 212.

[0248] In this fourth embodiment, a process is performed to generate and output an image in which a color map that outputs colors according to the defocus amount in image area units is superimposed on an image (e.g., an RGB image) generated by the image signal processing unit 212.

[0249] An image (e.g., an RGB image) generated by the image signal processing unit 212, or an image that has been modified and processed by the image region unit physical quantity reliability corresponding processing execution unit 206 with respect to the image generated by the image signal processing unit 212, is output to the image output unit 213.

[0250] The image output unit 213 outputs the image input from the image signal processing unit 212. For example, the image output unit 213 outputs the image to the monitor 117, the viewfinder (EVF) 116, and the recording device 115.

[0251] The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105, and outputs the selected phase difference detection signal (detection information) to the defocus amount calculation unit 202.

[0252] The defocus amount calculation unit 202 calculates the amount of focus deviation in units of minute image areas, for example, in units of image areas each consisting of a plurality of pixels, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)). As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the amount of deviation in the signals output in accordance with the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor.

[0253] The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to, for example, a focus position for a subject specified by the user based on this defocus amount, and outputs the generated AF control signal to the AF control unit 112a.

[0254] The AF control unit 112a drives the focus lens in accordance with an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and sets the focus lens to a focus position (focus position) for a subject designated by the user, for example.

[0255] Note that the subject set at the in-focus position (focus position) is not the entire image area of ​​the captured image, but a subject designated by the user, such as a person, etc. Images of other subjects, such as the background, are not in focus and appear blurred.

[0256] As described above with reference to FIG. 7, the defocus amount calculation unit 202 calculates the defocus amount in units of a small image area, such as n×m pixels, i.e., the defocus amount corresponding to the deviation between the focus distance and the subject distance.

[0257] The defocus map generation unit 204 inputs the defocus amount for each fine image area of ​​the captured image calculated by the defocus amount calculation unit 202, and generates a defocus map that makes it possible to identify the defocus amount for each image area based on the defocus amount for each image area.

[0258] For example, a defocus map such as that described above with reference to FIG. 8(B) is generated. For example, in the case of a defocus map with a brightness value (pixel value) set to 0 to 255, the closer the brightness value (pixel value) is to 255 (highest brightness (white)), the smaller the defocus amount, i.e., the higher the degree of focus of the pixel area, and the closer the brightness value (pixel value) is to 0 (lowest brightness (black)), the larger the defocus amount, i.e., the lower the degree of focus of the pixel area. In this way, the defocus map generating unit 204 generates a defocus map such as that shown in FIG. 8(B), for example.

[0259] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region, which is a physical quantity for each image region. As in the previous first embodiment, the image region unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and calculates the reliability of the defocus amount calculated for each image region based on these input data.

[0260] As explained above in the first embodiment, various methods can be applied as a method for calculating the reliability of the defocus amount for each image region. For example, a method of calculating the reliability of the defocus amount for each image region using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, and the like can be used. Also, the method using (Equation 1) described in the first embodiment, that is, the method of calculating the defocus amount reliability for each image region using the cross-correlation function of two waveforms of disparity data used for calculating the defocus amount, can be used.

[0261] Note that the defocus amount reliability calculated in the fourth embodiment is also the reliability of the defocus amount in pixel region units, as explained in the first embodiment. The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount per image region using the defocus amount per image region calculated by the defocus amount calculation unit 202, i.e., the defocus amount per image region, which is one rectangular region in the defocus map shown in FIG. 8(B).

[0262] In this way, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount calculated by the defocus amount calculation unit 202 for each image region, for example, by using the cross-correlation function of the two waveforms of the disparity data used to calculate the defocus amount.

[0263] The reliability data of the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability corresponding process execution unit 206 .

[0264] The image region unit physical quantity reliability corresponding processing execution unit 206 executes image control on the captured image (e.g., RGB image) generated by the image signal processing unit 212, according to the reliability data of the defocus amount on an image region unit basis calculated by the image region unit physical quantity reliability calculation unit 205. Specifically, an image is generated by superimposing a color map, which outputs colors according to the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205, on the captured image, and output the image. A specific example of this process will be described with reference to FIG. 20 and subsequent figures.

[0265] Figure 20 shows the following data: (A) Photographed image (B) Defocus map (C) Defocus color map

[0266] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The (B) defocus map is a defocus map generated by the defocus map generation unit 204. That is, it is a defocus map in which the defocus amount for each image region generated by the defocus map generation unit 204 is output as a brightness value (for example, 0 to 255) based on the defocus amount for each image region calculated by the defocus amount calculation unit 202 based on the pixel value signals of the phase difference detection pixels acquired by the phase difference information acquisition unit 201.

[0267] (C) The defocus color map is a map obtained by coloring a defocus map in which the amount of defocus is output as a brightness value (for example, 0 to 255), and is a map in which different colors are set according to the amount of defocus. For example, this is a color map in which a region where the defocus amount is 0 to small is yellow, a region where the defocus amount is medium is green, and a region where the defocus amount is large is blue.

[0268] Various color settings are possible. For example, in addition to the above-mentioned three or five color settings, it is also possible to set the color to change smoothly from yellow to blue according to the change in the defocus amount.

[0269] Furthermore, for in-focus areas where the defocus amount is approximately 0, no color may be set, and the area may be made transparent so that the color of the original captured image is output as is. Furthermore, it is also possible to use a configuration in which subject objects are identified by, for example, face recognition or semantic segmentation, and a specific color that is different only for a human face area or a specific object is output.

[0270] In this way, the image region unit physical quantity reliability corresponding processing execution unit 206 first generates a color map that outputs colors according to the defocus amounts of the image region units calculated by the image region unit physical quantity reliability calculation unit 205. Furthermore, the image region unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image. A specific example of this process will be described with reference to FIG.

[0271] FIG. 21 shows the following data: (A) Photographed image (C) Defocus color map (D) Output image

[0272] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . (C) The defocus color map is a map obtained by coloring a defocus map in which the amount of defocus is output as a brightness value (for example, 0 to 255), and is a map in which different colors are set according to the amount of defocus. For example, this is a color map in which a region where the defocus amount is 0 to small is yellow, a region where the defocus amount is medium is green, and a region where the defocus amount is large is blue.

[0273] (C) Output image is an output image generated by superimposing (B) defocus color map on (A) captured image.

[0274] In this way, the image region unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image. This (C) output image is output to the monitor 117 of the imaging device 100, for example. The user can easily and reliably determine the amount of defocus in each area of ​​the captured image by looking at the image output on the monitor 117.

[0275] Note that the digital signal processing unit 108 described with reference to FIG. 19 is configured to include all of the components, namely, the phase difference information acquisition unit 201, the defocus amount calculation unit 202, the AF control signal generation unit 203, the defocus map generation unit 204, the image region unit physical quantity reliability calculation unit 205, the image region unit physical quantity reliability corresponding processing execution unit 206, the image information acquisition unit 211, the image signal processing unit 212, and the image output unit 213, within the digital signal processing unit 108; however, this configuration is merely an example.

[0276] 19 may be configured as a part of the digital signal processing unit 108 of the imaging device 100. Also, a configuration may be adopted in which data processing is performed in an external device different from the imaging device 100. Specifically, for example, the image signal processing unit 212 can be configured outside the digital signal processing unit 108. Furthermore, the image signal processing may be performed in an external device other than the imaging device 100, such as a PC.

[0277] [4-5. (Example 5) Example of generating and outputting an image in which a color corresponding to the defocus amount for each image region is output and a color map that enables identification of the reliability of the defocus amount for each image region is superimposed on the captured image] Next, as Example 5, an example will be described in which a color corresponding to the defocus amount for each image area is output, and an image in which a color map that makes it possible to identify the reliability of the defocus amount for each image area is superimposed on the captured image and output.

[0278] This Example 5 is a modified example of the above-mentioned Example 4, and is an example in which the color map described in Example 4 is changed to a color map that makes it possible to identify the defocus amount reliability described in Example 1.

[0279] FIG. 22 is a block diagram illustrating the configuration of the fifth embodiment. FIG. 22 shows an example of the internal configuration of the digital signal processing unit 108, which is a component of the imaging device 100 described with reference to FIG.

[0280] As shown in FIG. 22 , the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, and an image output unit 213.

[0281] The digital signal processing unit 108 receives as input the RGB image signal from the A / D conversion unit 105 at the previous stage and the phase difference detection signal (detection information) that is the output of the phase difference detection pixel.

[0282] The phase difference information acquisition unit 201 shown in FIG. 22 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105. On the other hand, the image information acquisition unit 211 shown in FIG. 22 selects only the image signal (for example, the RGB image signal) from the input signal from the A / D conversion unit 105.

[0283] The image signal acquired by the image information acquisition unit 211 is input to the image signal processing unit 212 . The image signal processing unit 212 performs various image signal processes such as demosaic processing, white balance adjustment, gamma correction, etc. on the image signal, and outputs the processed image (for example, an RGB image) to the image output unit 213.

[0284] Also in this embodiment 5, similarly to the previously described embodiments 1 to 4, the image (e.g., RGB image) generated by the image signal processing unit 212 is subjected to image control processing by the image region unit physical quantity reliability corresponding processing execution unit 206.

[0285] In this embodiment 5, a color corresponding to the defocus amount per image area is output for an image (e.g., an RGB image) generated by the image signal processing unit 212, and further, an image is generated and output on which a color map that makes it possible to identify the reliability of the defocus amount per image area is superimposed.

[0286] An image (e.g., an RGB image) generated by the image signal processing unit 212, or an image that has been modified and processed by the image region unit physical quantity reliability corresponding processing execution unit 206 with respect to the image generated by the image signal processing unit 212, is output to the image output unit 213.

[0287] The image output unit 213 outputs the image input from the image signal processing unit 212. For example, the image output unit 213 outputs the image to the monitor 117, the viewfinder (EVF) 116, and the recording device 115.

[0288] The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105, and outputs the selected phase difference detection signal (detection information) to the defocus amount calculation unit 202.

[0289] The defocus amount calculation unit 202 calculates the amount of focus deviation in units of minute image areas, for example, in units of image areas each consisting of a plurality of pixels, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)). As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the amount of deviation in the signals output in accordance with the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor.

[0290] The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to, for example, a focus position for a subject specified by the user based on this defocus amount, and outputs the generated AF control signal to the AF control unit 112a.

[0291] The AF control unit 112a drives the focus lens in accordance with an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and sets the focus lens to a focus position (focus position) for a subject designated by the user, for example.

[0292] Note that the subject set at the in-focus position (focus position) is not the entire image area of ​​the captured image, but a subject designated by the user, such as a person, etc. Images of other subjects, such as the background, are not in focus and appear blurred.

[0293] As described above with reference to FIG. 7, the defocus amount calculation unit 202 calculates the defocus amount in units of a small image area, such as n×m pixels, i.e., the defocus amount corresponding to the deviation between the focus distance and the subject distance.

[0294] The defocus map generation unit 204 inputs the defocus amount for each fine image area of ​​the captured image calculated by the defocus amount calculation unit 202, and generates a defocus map that makes it possible to identify the defocus amount for each image area based on the defocus amount for each image area.

[0295] For example, a defocus map such as that described above with reference to FIG. 8(B) is generated. For example, in the case of a defocus map with a brightness value (pixel value) set to 0 to 255, the closer the brightness value (pixel value) is to 255 (highest brightness (white)), the smaller the defocus amount, i.e., the higher the degree of focus of the pixel area, and the closer the brightness value (pixel value) is to 0 (lowest brightness (black)), the larger the defocus amount, i.e., the lower the degree of focus of the pixel area. In this way, the defocus map generating unit 204 generates a defocus map such as that shown in FIG. 8(B), for example.

[0296] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region, which is a physical quantity for each image region. As in the previous first embodiment, the image region unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and calculates the reliability of the defocus amount calculated for each image region based on these input data.

[0297] As explained above in the first embodiment, various methods can be applied as a method for calculating the reliability of the defocus amount for each image region. For example, a method of calculating the reliability of the defocus amount for each image region using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, and the like can be used. Also, the method using (Equation 1) described in the first embodiment, that is, the method of calculating the defocus amount reliability for each image region using the cross-correlation function of two waveforms of disparity data used for calculating the defocus amount, can be used.

[0298] Note that the defocus amount reliability calculated in the fifth embodiment is also the reliability of the defocus amount in pixel region units, as explained in the first embodiment. The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount per image region using the defocus amount per image region calculated by the defocus amount calculation unit 202, i.e., the defocus amount per image region, which is one rectangular region in the defocus map shown in FIG. 8(B).

[0299] In this way, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount calculated by the defocus amount calculation unit 202 for each image region, for example, by using the cross-correlation function of the two waveforms of the disparity data used to calculate the defocus amount.

[0300] The reliability data of the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability corresponding process execution unit 206 .

[0301] The image region unit physical quantity reliability corresponding processing execution unit 206 executes image control on the captured image (e.g., RGB image) generated by the image signal processing unit 212, according to the reliability data of the defocus amount on an image region unit basis calculated by the image region unit physical quantity reliability calculation unit 205. Specifically, the image area unit physical quantity reliability calculation unit 205 outputs a color according to the defocus amount calculated for each image area, and further generates and outputs an image in which a color map that makes it possible to identify the reliability of the defocus amount for each image area is superimposed on the captured image. A specific example of this process will be described with reference to FIG. 23 and subsequent figures.

[0302] FIG. 23 shows the following data: (A) Photographed image (B) Defocus map (C) Defocus color map

[0303] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The (B) defocus map is a defocus map generated by the defocus map generation unit 204. That is, it is a defocus map in which the defocus amount for each image region generated by the defocus map generation unit 204 is output as a brightness value (for example, 0 to 255) based on the defocus amount for each image region calculated by the defocus amount calculation unit 202 based on the pixel value signals of the phase difference detection pixels acquired by the phase difference information acquisition unit 201.

[0304] The defocus map (B) shown in FIG. 23 includes a region with low reliability of the defocus amount. As described above, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region unit of the defocus map generated by the defocus map generation unit 204, and outputs the calculated reliability data to the image region unit physical quantity reliability corresponding processing execution unit 206.

[0305] The image region unit physical quantity reliability corresponding processing execution unit 206 extracts a defocus amount low reliability region, for example, as shown in FIG. 23(B), based on the reliability data of the defocus amount in image region unit input from the image region unit physical quantity reliability calculation unit 205.

[0306] Specifically, the defocus amount reliability for each image region input from the image region unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold value to extract a defocus amount low reliability region. For example, using a predefined low-reliability threshold Th1, (Determination formula 1) Defocus amount reliability≦Th1 An image region that satisfies the above-mentioned judgment formula 1 is extracted as a defocus amount low reliability region.

[0307] The defocus map shown in FIG. 23(B) includes a region with low reliability of the defocus amount. In this way, when a defocus amount low reliability region is detected, the image region unit physical quantity reliability corresponding process execution unit 206 generates a defocus color map that makes it possible to identify the defocus amount low reliability region.

[0308] The defocus color map of Figure 23(C) is a map in which a defocus map in which the defocus amount is output as a brightness value (for example, 0 to 255) is colored, and different colors are set according to the defocus amount, and further, a specific color is set in the defocus amount low reliability area.

[0309] For example, this is a color map in which areas with a defocus amount of 0 to small are set to yellow, areas with a medium defocus amount are set to green, areas with a large defocus amount are set to blue, and areas with low reliability of defocus amount are set to red. Various color settings are possible.

[0310] In this way, the image region unit physical quantity reliability corresponding process executing unit 206 first sets a color according to the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit 205, and further generates a color map in which a defocus amount low reliability region is set to red. Furthermore, the image region unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image. A specific example of this process will be described with reference to FIG.

[0311] FIG. 24 shows the following data: (A) Photographed image (C) Defocus color map (D) Output image

[0312] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . (C) The defocus color map is a color map in which different colors are set according to the defocus amount in a defocus map that outputs the defocus amount as a brightness value (for example, 0 to 255), and low-reliability defocus amount areas are set in a specific color (for example, red) that can be identified.

[0313] For example, the color map may be set such that areas with a defocus amount of 0 to small are yellow, areas with a medium defocus amount are green, areas with a large defocus amount are blue, and areas with low reliability of the defocus amount are red.

[0314] (D) The output image is generated by superimposing (C) the defocus color map on (A) the captured image.

[0315] In this way, the image region unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image. This (D) output image is output to the monitor 117 of the imaging device 100, for example. By viewing the image output on the monitor 117, the user can easily and reliably determine the amount of defocus in each area of ​​the captured image, and can also reliably recognize areas with low reliability of the defocus amount.

[0316] Note that the digital signal processing unit 108 described with reference to FIG. 22 is configured to include all of the components, namely, the phase difference information acquisition unit 201, the defocus amount calculation unit 202, the AF control signal generation unit 203, the defocus map generation unit 204, the image region unit physical quantity reliability calculation unit 205, the image region unit physical quantity reliability corresponding processing execution unit 206, the image information acquisition unit 211, the image signal processing unit 212, and the image output unit 213, within the digital signal processing unit 108; however, this configuration is one example.

[0317] 22 may be configured as a part of the digital signal processing unit 108 of the imaging device 100. Also, the data processing may be performed in an external device other than the imaging device 100. Specifically, for example, the image signal processing unit 212 can be configured outside the digital signal processing unit 108. Furthermore, the image signal processing may be performed in an external device other than the imaging device 100, such as a PC.

[0318] 4-6. (Example 6) Example of image capture by controlling exposure time according to reliability of defocus amount for each image region Next, as a sixth embodiment, an embodiment in which an image is captured by controlling the exposure time in accordance with the reliability of the defocus amount for each image region will be described.

[0319] Image areas with low reliability of the defocus amount are often areas where the subject distance is not measured accurately, and are likely to be image areas with poor S / N of the captured image. For such image areas, the S / N can be improved by setting a longer exposure time. The embodiment described below is an embodiment that performs shooting control to set a long exposure time for image areas where the reliability of the defocus amount is low, for example, and executes image shooting, thereby enabling high-quality image capture.

[0320] FIG. 25 is a block diagram illustrating the configuration of the sixth embodiment. FIG. 25 shows an example of the internal configuration of the digital signal processing unit 108, which is a component of the imaging device 100 described with reference to FIG.

[0321] As shown in FIG. 25 , the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, and an image output unit 213.

[0322] The digital signal processing unit 108 receives as input the RGB image signal from the A / D conversion unit 105 at the previous stage and the phase difference detection signal (detection information) that is the output of the phase difference detection pixel.

[0323] The phase difference information acquisition unit 201 shown in FIG. 25 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105. On the other hand, the image information acquisition unit 211 shown in FIG. 25 selects only the image signal (for example, the RGB image signal) from the input signal from the A / D conversion unit 105.

[0324] The image signal acquired by the image information acquisition unit 211 is input to the image signal processing unit 212 . The image signal processing unit 212 performs various image signal processes such as demosaic processing, white balance adjustment, gamma correction, etc. on the image signal, and outputs the processed image (for example, an RGB image) to the image output unit 213.

[0325] The image (for example, an RGB image) generated by the image signal processing unit 212 is output to the image output unit 213. The image output unit 213 outputs the image input from the image signal processing unit 212. For example, the image output unit 213 outputs the image to the monitor 117, the viewfinder (EVF) 116, and the recording device 115.

[0326] In this sixth embodiment, unlike the first to fifth embodiments described above, the image region unit physical quantity reliability corresponding processing execution unit 206 does not perform image processing on the image (for example, an RGB image) generated by the image signal processing unit 212. The image region unit physical quantity reliability corresponding processing execution unit 206 outputs an imaging control command to the control unit 110. Specifically, an exposure time control command is output to perform image capture using exposure control in which the exposure time for an image region with a low reliability of the defocus amount is set to be long.

[0327] The exposure time during image capture can be controlled to change on a pixel-by-pixel basis. For example, by applying the configuration described in Patent Document 4 (JP 2011-004088 A), exposure time control on a pixel-by-pixel basis can be performed.

[0328] The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A / D conversion unit 105, and outputs the selected phase difference detection signal (detection information) to the defocus amount calculation unit 202.

[0329] The defocus amount calculation unit 202 calculates the amount of focus deviation in units of minute image areas, for example, in units of image areas each consisting of a plurality of pixels, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)). As described above, in the phase difference detection method, the defocus amount of the focus lens is calculated based on the amount of deviation in the signals output in accordance with the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor.

[0330] The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to, for example, a focus position for a subject specified by the user based on this defocus amount, and outputs the generated AF control signal to the AF control unit 112a.

[0331] The AF control unit 112a drives the focus lens in accordance with an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and sets the focus lens to a focus position (focus position) for a subject designated by the user, for example.

[0332] Note that the subject set at the in-focus position (focus position) is not the entire image area of ​​the captured image, but a subject designated by the user, such as a person, etc. Images of other subjects, such as the background, are not in focus and appear blurred.

[0333] As described above with reference to FIG. 7, the defocus amount calculation unit 202 calculates the defocus amount in units of a small image area, such as n×m pixels, i.e., the defocus amount corresponding to the deviation between the focus distance and the subject distance.

[0334] The defocus map generation unit 204 inputs the defocus amount for each fine image area of ​​the captured image calculated by the defocus amount calculation unit 202, and generates a defocus map that makes it possible to identify the defocus amount for each image area based on the defocus amount for each image area.

[0335] For example, a defocus map such as that described above with reference to FIG. 8(B) is generated. For example, in the case of a defocus map with a brightness value (pixel value) set to 0 to 255, the closer the brightness value (pixel value) is to 255 (highest brightness (white)), the smaller the defocus amount, i.e., the higher the degree of focus of the pixel area, and the closer the brightness value (pixel value) is to 0 (lowest brightness (black)), the larger the defocus amount, i.e., the lower the degree of focus of the pixel area. In this way, the defocus map generating unit 204 generates a defocus map such as that shown in FIG. 8(B), for example.

[0336] The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region, which is a physical quantity for each image region. As in the previous first embodiment, the image region unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and calculates the reliability of the defocus amount calculated for each image region based on these input data.

[0337] As explained above in the first embodiment, various methods can be applied as a method for calculating the reliability of the defocus amount for each image region. For example, a method of calculating the reliability of the defocus amount for each image region using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, and the like can be used. Also, the method using (Equation 1) described in the first embodiment, that is, the method of calculating the defocus amount reliability for each image region using the cross-correlation function of two waveforms of disparity data used for calculating the defocus amount, can be used.

[0338] Note that the defocus amount reliability calculated in the sixth embodiment is also the reliability of the defocus amount in pixel region units, as explained in the first embodiment. The image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount per image region using the defocus amount per image region calculated by the defocus amount calculation unit 202, i.e., the defocus amount per image region, which is one rectangular region in the defocus map shown in FIG. 8(B).

[0339] In this way, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount calculated by the defocus amount calculation unit 202 for each image region, for example, by using the cross-correlation function of the two waveforms of the disparity data used to calculate the defocus amount.

[0340] The reliability data of the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability corresponding process execution unit 206 .

[0341] The image region unit physical quantity reliability corresponding processing execution unit 206 outputs an imaging control command to the control unit 110 according to the reliability data of the defocus amount in image region units calculated by the image region unit physical quantity reliability calculation unit 205. Specifically, for an image region where the defocus amount is low in reliability, an exposure time control command is output to set a long exposure time and perform image capture. A specific example of this process will be described with reference to FIG.

[0342] FIG. 26 shows the following data: (A) Photographed image (B) Defocus map (C) Example of shooting control (example of exposure time control for each image area)

[0343] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The (B) defocus map is a defocus map generated by the defocus map generation unit 204. That is, it is a defocus map in which the defocus amount for each image region generated by the defocus map generation unit 204 is output as a brightness value (for example, 0 to 255) based on the defocus amount for each image region calculated by the defocus amount calculation unit 202 based on the pixel value signals of the phase difference detection pixels acquired by the phase difference information acquisition unit 201.

[0344] The defocus map (B) shown in FIG. 26 includes a region with low reliability of the defocus amount. As described above, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region unit of the defocus map generated by the defocus map generation unit 204, and outputs the calculated reliability data to the image region unit physical quantity reliability corresponding processing execution unit 206.

[0345] The image region unit physical quantity reliability corresponding processing execution unit 206 extracts, for example, a defocus amount low reliability region as shown in FIG. 26(B) based on the reliability data of the defocus amount in image region unit input from the image region unit physical quantity reliability calculation unit 205.

[0346] Specifically, the defocus amount reliability for each image region input from the image region unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold value to extract a defocus amount low reliability region. For example, using a predefined low-reliability threshold Th1, (Determination formula 1) Defocus amount reliability≦Th1 An image region that satisfies the above-mentioned judgment formula 1 is extracted as a defocus amount low reliability region.

[0347] The defocus map shown in FIG. 26(B) includes a region with low reliability of the defocus amount. In this way, when a defocus amount low reliability region is detected, the image region unit physical quantity reliability corresponding process executing unit 206 outputs an exposure time control command to the control unit 110 that performs exposure control during image capture. Specifically, the exposure time control command is output to set the exposure time for an image area with a low reliability of the defocus amount longer than the exposure time for other image areas and perform image capture.

[0348] That is, as shown in the example of shooting control (example of exposure time control per image area) in Figure 26 (C), an image area with a low reliability of defocus amount is designated, and an exposure time control command is output to set the exposure time of this designated area longer than the exposure time of other image areas and perform image shooting.

[0349] By controlling the exposure time in this way, an image is captured with the exposure time for an image region with a low reliability of the defocus amount being longer than the exposure time for other image regions. As a result, it is possible to take a high-quality image with an improved S / N ratio in an image region where the defocus amount is low in reliability.

[0350] In this way, the image region unit physical quantity reliability corresponding process execution unit 206 executes image capture control for controlling the exposure time according to the reliability of the defocus amount in image region units and performing image capture.

[0351] As mentioned above, image regions with low reliability of the defocus amount are often regions where the subject distance is not measured accurately, and are likely to be image regions with poor S / N ratios in the captured image. For such image regions, the S / N ratio can be improved by setting a longer exposure time. As described above, the sixth embodiment is an embodiment that enables capturing of a high-quality image by performing shooting control to set a long exposure time for an image region where the reliability of the defocus amount is low and execute image capture.

[0352] The sixth embodiment described with reference to FIGS. 25 and 26 is an embodiment in which shooting control is performed to set a long exposure time for an image area in which the reliability of the defocus amount is low and perform image shooting. However, as a modified example of the sixth embodiment, a configuration in which shooting control is performed as follows is also possible. (Modification 1) A configuration for performing shooting control to set a short exposure time for an image area where the reliability of the defocus amount is high. (Variation 2) A configuration for performing shooting control in which a long exposure time is set for an image region where the reliability of the defocus amount is low, and a short exposure time is set for an image region where the reliability of the defocus amount is high.

[0353] FIG. 27 shows an example of the configuration of the digital signal processing unit 108 of the imaging device that performs the above-mentioned (Modification 1) shooting control. The configuration of the digital signal processing unit 108 shown in FIG. 27 has the same components as the configuration of the digital signal processing unit 108 shown in FIG. 25 described above. However, the processing executed by the image region unit physical quantity reliability corresponding processing execution unit 206 is different.

[0354] The image region unit physical quantity reliability corresponding processing execution unit 206 of the digital signal processing unit 108 shown in FIG. 27 outputs an exposure time control command for setting a short exposure time and performing image capturing for an image region in which the defocus amount in image region unit calculated by the image region unit physical quantity reliability calculation unit 205 is highly reliable. A specific example of this process will be described with reference to FIG.

[0355] FIG. 28 shows the following data: (A) Photographed image (B) Defocus map (C) Example of shooting control (example of exposure time control for each image area)

[0356] The (A) photographed image is a photographed image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 . The (B) defocus map is a defocus map generated by the defocus map generation unit 204. That is, it is a defocus map in which the defocus amount for each image region generated by the defocus map generation unit 204 is output as a brightness value (for example, 0 to 255) based on the defocus amount for each image region calculated by the defocus amount calculation unit 202 based on the pixel value signals of the phase difference detection pixels acquired by the phase difference information acquisition unit 201.

[0357] The defocus map (B) shown in FIG. 28 includes a region with high reliability of the defocus amount. As described above, the image region unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image region unit of the defocus map generated by the defocus map generation unit 204, and outputs the calculated reliability data to the image region unit physical quantity reliability corresponding processing execution unit 206.

[0358] The image region unit physical quantity reliability corresponding processing execution unit 206 extracts a defocus amount high reliability region, for example, as shown in FIG. 28(B), based on the reliability data of the defocus amount in image region unit input from the image region unit physical quantity reliability calculation unit 205.

[0359] Specifically, the defocus amount reliability for each image region input from the image region unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold value to extract a defocus amount high reliability region. For example, using a predefined low-reliability threshold Th2, (Determination formula 2) Th2≦Defocus amount reliability An image region that satisfies the above-mentioned judgment formula 2 is extracted as a defocus amount high reliability region.

[0360] The defocus map shown in FIG. 28(B) includes a region with high reliability of the defocus amount. In this way, when a defocus amount highly reliable region is detected, the image region unit physical quantity reliability corresponding process executing unit 206 outputs an exposure time control command to the control unit 110 that performs exposure control during image capture. Specifically, the exposure time control command is output to set the exposure time for an image area with a highly reliable defocus amount shorter than the exposure time for other image areas and perform image capture.

[0361] That is, as shown in the example of shooting control (example of exposure time control per image area) in Figure 28 (C), an image area with a highly reliable defocus amount is designated, and an exposure time control command is output to set the exposure time of this designated area shorter than the exposure time of other image areas and perform image shooting.

[0362] By controlling the exposure time in this way, an image is captured with the exposure time for an image area with a highly reliable defocus amount being shorter than the exposure time for other image areas. As a result, it is possible to capture a high-quality image with reduced subject blurring in image areas where the defocus amount is highly reliable.

[0363] FIG. 29 shows an example configuration of a digital signal processing unit 108 of an imaging device that performs the above-mentioned (Variant 2) shooting control, i.e., sets a long exposure time for image areas where the reliability of the defocus amount is low, and sets a short exposure time for image areas where the reliability of the defocus amount is high. The configuration of the digital signal processing unit 108 shown in FIG. 29 has the same components as the configuration of the digital signal processing unit 108 shown in FIG. 25 described above. However, the processing executed by the image region unit physical quantity reliability corresponding processing execution unit 206 is different.

[0364] The image region unit physical quantity reliability corresponding processing execution unit 206 of the digital signal processing unit 108 shown in FIG. 29 outputs an exposure time control command for executing image capturing by setting a long exposure time for an image region where the defocus amount calculated by the image region unit physical quantity reliability calculation unit 205 is low in reliability and setting a short exposure time for an image region where the defocus amount is high in reliability.

[0365] As a result of these processes, the effects of both the sixth embodiment described with reference to FIGS. 25 and 26 and the (first modified example) described with reference to FIGS. 27 and 28 can be obtained. That is, for image areas where the reliability of the defocus amount is low, it is possible to improve the S / N ratio by setting a long exposure time. Furthermore, in image areas where the defocus amount is highly reliable, it is possible to reduce subject blur.

[0366] In this way, the sixth embodiment is an embodiment that makes it possible to capture a high-quality image by controlling the exposure time for each image region in accordance with the reliability of the defocus amount and then capturing an image.

[0367] [5. Other Examples] Next, other embodiments will be described.

[0368] Up to now, the following six embodiments have been described with reference to FIGS. (Example 1) Example of generating and outputting an image in which the reliability of the defocus amount in image area units can be identified (Example 2) Example of generating and outputting an image in which the distance ratio between the focused subject and other background subjects is superimposed on the captured image (Embodiment 3) An embodiment in which an image in which mask stability information is superimposed on a photographed image is generated and output (Embodiment 4) An embodiment in which an image is generated by superimposing a color map, which outputs colors according to the defocus amount in image area units, on a captured image and outputting the image (Embodiment 5) An embodiment in which a color corresponding to the defocus amount in image area units is output, and an image in which a color map that makes it possible to identify the reliability of the defocus amount in image area units is superimposed on the captured image is generated and output. (Embodiment 6) An embodiment in which an image is captured by controlling the exposure time according to the reliability of the defocus amount in each image area

[0369] These six embodiments can be configured independently, but can also be configured as an apparatus or system having a combination of any number of embodiments.

[0370] In the second embodiment and the like, the image region unit physical quantity reliability calculation unit 205 performs the following process to calculate the reliability of the distance value in image region units. That is, the image area unit physical quantity reliability calculation unit 205 determines that an image area with a low reliability of the defocus amount also has a low reliability of the distance value calculated based on the defocus amount, and determines that an image area with a high reliability of the defocus amount also has a high reliability of the distance value calculated based on the defocus amount. Such a processing example has been described.

[0371] Regarding the reliability determination process of the distance value in image region units, if the imaging device is a twin-lens stereo camera, for example, the following process may be performed. That is, the amount of error in the matching process of the disparity map generated from the images captured by the stereo camera is calculated for each image region, and the distance value reliability for each image region is calculated based on the calculated amount of error for each image region. In this process, an image region with a large amount of error in an image region unit is determined to have low distance value reliability, and an image region with a small amount of error in an image region unit is determined to have high distance value reliability.

[0372] Furthermore, the processes of the above-described first to sixth embodiments may be configured to be executed continuously while the imaging device 100 is in operation, or may be configured to be executed in response to a specific operation on the input unit (operation unit) 118 by the user, for example. Furthermore, the processing may be stopped if the focus position has changed significantly, or if it is determined from the detection value of the gyroscope 131 that the imaging device 100 has moved significantly. Conversely, if the focus position has changed significantly, new processing may be started. Also, if it is determined from the detection value of the gyroscope 131 that the image capturing device 100 has moved significantly, new processing may be started.

[0373] 6. Summary of the Disclosure The embodiments of the present disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0374] The technology disclosed in this specification can be configured as follows. (1) an image region physical quantity calculation unit that calculates a physical quantity that changes depending on the subject distance for each image region that is a divided region of a captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; an image area unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image area units calculated by the image area unit physical quantity reliability calculation unit;

[0375] (2) The image region physical quantity calculation unit a defocus amount calculation unit that calculates a defocus amount for each image region; The image region unit physical quantity reliability calculation unit The imaging device according to (1), wherein the defocus amount calculation unit calculates a reliability of the defocus amount calculated for each image region.

[0376] (3) The image region unit physical quantity reliability correspondence processing execution unit Based on the reliability of the defocus amount calculated by the image region unit physical quantity reliability calculation unit, The imaging device according to (2), wherein graphic data indicating the reliability of the defocus amount for each image region is displayed superimposed on the captured image.

[0377] (4) The image region unit physical quantity reliability correspondence processing execution unit comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold Th1; The imaging device according to (3), wherein graphic data that enables identification of image regions where the reliability of the defocus amount in image region units is equal to or less than the low reliability threshold value Th1 is superimposed on the captured image and displayed.

[0378] (5) The image region unit physical quantity reliability correspondence processing execution unit comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined high reliability threshold value Th2; The imaging device according to (3) or (4), wherein graphic data that enables identification of image regions where the reliability of the defocus amount in image region units is equal to or greater than the high reliability threshold Th2 is superimposed on the captured image and displayed.

[0379] (6) The image region unit physical quantity reliability correspondence processing execution unit The imaging device according to any one of (2) to (5) displays mask stability information indicating whether or not highly accurate mask setting is possible when masking an area other than the specific subject.

[0380] (7) The image region unit physical quantity reliability correspondence processing execution unit comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold Th1; When an image area having a reliability equal to or less than the low reliability threshold Th1 is detected in the captured image, The imaging device according to (6), wherein mask stability information indicating that highly accurate mask setting is difficult is displayed on the captured image.

[0381] (8) The image region unit physical quantity reliability corresponding processing execution unit comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold Th1; If an image area with a reliability equal to or less than the low reliability threshold Th1 is not detected in the captured image, The imaging device according to (6) or (7), wherein mask stability information indicating that highly accurate mask setting is possible is displayed on the captured image.

[0382] (9) The image region unit physical quantity reliability correspondence processing execution unit The imaging device according to any one of (2) to (8), wherein a color map in which different colors are set according to the defocus amount in units of the image region is displayed superimposed on the captured image.

[0383] (10) The image region unit physical quantity reliability correspondence processing execution unit comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold Th1; The imaging device according to (9), wherein a color map that enables identification of image regions where the reliability of the defocus amount in image region units is equal to or less than the low reliability threshold Th1 is superimposed on the captured image and displayed.

[0384] (11) The image region unit physical quantity reliability correspondence processing execution unit The imaging device according to any one of (2) to (10), wherein exposure time control is performed for each image region in accordance with the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation section.

[0385] (12) The image region unit physical quantity reliability correspondence processing execution unit comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold Th1; The imaging device according to (11), wherein exposure time control is performed to make the exposure time of an image area where the reliability of the defocus amount in image area unit is equal to or less than the low reliability threshold Th1 longer than that of other image areas.

[0386] (13) The image region unit physical quantity reliability correspondence processing execution unit comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined high reliability threshold value Th2; The imaging device according to (11) or (12), wherein exposure time control is performed to make the exposure time of an image area where the reliability of the defocus amount in image area unit is equal to or greater than the high reliability threshold Th2 shorter than that of other image areas.

[0387] (14) The image region physical quantity calculation unit a distance information calculation unit that calculates a distance value in units of the image region; The image region unit physical quantity reliability calculation unit The imaging device according to any one of (1) to (13), wherein the distance information calculation unit calculates a reliability of the distance information calculated for each image region.

[0388] (15) The image region unit physical quantity reliability correspondence processing execution unit The imaging device according to (14) displays distance ratios of a plurality of subjects at different distances.

[0389] (16) The image region unit physical quantity reliability correspondence processing execution unit The imaging device according to (14) or (15), wherein an image area is selected in which the reliability of the distance information of the image area unit calculated by the image area unit physical quantity reliability calculation unit is equal to or greater than a predetermined threshold, and a distance ratio between the subjects of the selected image area is calculated and displayed on an image captured by the imaging device.

[0390] (17) An image processing method executed in an imaging device, an image region physical quantity calculation step in which an image region physical quantity calculation unit calculates a physical quantity that changes depending on a subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation step in which an image region unit physical quantity reliability calculation unit calculates the reliability of the physical quantity for each image region calculated in the image region physical quantity calculation step; an image region unit physical quantity reliability corresponding processing execution step in which an image region unit physical quantity reliability corresponding processing execution unit executes control processing according to the reliability of the physical quantity for each image region calculated in the image region unit physical quantity reliability calculation step.

[0391] (18) A program for causing an imaging device to perform image processing, an image region physical quantity calculation step of causing an image region physical quantity calculation unit to calculate a physical quantity that changes depending on the subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation step of causing an image region unit physical quantity reliability calculation unit to calculate the reliability of the physical quantity for each image region calculated in the image region physical quantity calculation step; a program that causes an image region unit physical quantity reliability corresponding processing execution step to cause an image region unit physical quantity reliability corresponding processing execution unit to execute control processing according to the reliability of the physical quantity for each image region calculated in the image region unit physical quantity reliability calculation step.

[0392] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed in a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from a recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0393] Furthermore, the various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices in the same housing. [Industrial Applicability]

[0394] As described above, according to the configuration of one embodiment of the present disclosure, an apparatus and method are realized that calculate the reliability of the defocus amount and distance value for each image area of ​​a captured image, and perform processing such as displaying graphic data indicating the reliability, displaying the distance ratio of a subject, and controlling exposure time. Specifically, for example, the defocus amount and distance value are calculated for each image region of the captured image, and the reliability of the calculated defocus amount and distance value for each image region is further calculated, and control is performed according to the reliability of the calculated defocus amount and distance value for each image region. For example, the control may perform a process of displaying graphic data indicating the reliability of the defocus amount for each image region by superimposing it on the captured image, a process of displaying the distance ratio of the subject, a process of controlling the exposure time, etc. This configuration realizes an apparatus and method that calculates the reliability of the defocus amount and distance value for each image area of ​​a captured image, and performs display processing of graphic data indicating the reliability, display processing of the distance ratio of the subject, exposure time control processing, etc. [Explanation of symbols]

[0395] 100 Imaging device 101 Focus Lens 102 Zoom Lens 103 Image sensor 104 Analog signal processing section 105 A / D conversion section 106 Timing Generator (TG) 107 Vertical Driver 108 Digital Signal Processing Unit (DSP) 110 control section 112a AF control unit 112b Zoom control section 113 Motor 115 Recording Devices 116 Viewfinder 117 Monitor 118 Input section (operation section) 122 Image Area 131 Gyro 151 Phase difference detection pixels 152 Image Area 201 Phase difference information acquisition section 202 Defocus amount calculation unit 203 AF control signal generation unit 204 Defocus map generation unit 205 Image region unit physical quantity reliability calculation unit 206 Image region unit physical quantity reliability correspondence processing execution unit 211 Image information acquisition unit 212 Image signal processing section 213 Image output unit 221 Distance information calculation unit

Claims

1. an image region physical quantity calculation unit that calculates a physical quantity that changes depending on the subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; an image region unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image region unit calculated by the image region unit physical quantity reliability calculation unit, The image region physical quantity calculation unit a defocus amount calculation unit that calculates a defocus amount for each image region; The image region unit physical quantity reliability calculation unit calculating a reliability of the defocus amount calculated by the defocus amount calculation unit for each image region; The image region unit physical quantity reliability corresponding processing execution unit A configuration that executes a process of displaying mask stability information indicating whether or not a highly accurate mask setting is possible when masking an area other than the specific subject, comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1; When an image area having a reliability equal to or less than the low reliability threshold Th1 is detected in the captured image, An imaging device that displays mask stability information on the captured image, which indicates that highly accurate mask setting is difficult.

2. An image region physical quantity calculation unit that calculates a physical quantity that changes depending on the subject distance for each image region that is a divided region of a captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; an image region unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image region unit calculated by the image region unit physical quantity reliability calculation unit, The image region physical quantity calculation unit a defocus amount calculation unit that calculates a defocus amount for each image region; The image region unit physical quantity reliability calculation unit calculating a reliability of the defocus amount calculated by the defocus amount calculation unit for each image region; The image region unit physical quantity reliability corresponding processing execution unit A configuration that executes a process of displaying mask stability information indicating whether or not a highly accurate mask setting is possible when masking an area other than the specific subject, comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1; If an image area with a reliability equal to or less than the low reliability threshold Th1 is not detected in the captured image, An imaging device that displays mask stability information on the captured image, which indicates that highly accurate mask setting is possible.

3. An image region physical quantity calculation unit that calculates a physical quantity that changes depending on the subject distance for each image region that is a divided region of a captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; an image region unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image region unit calculated by the image region unit physical quantity reliability calculation unit, The image region physical quantity calculation unit a defocus amount calculation unit that calculates a defocus amount for each image region; The image region unit physical quantity reliability calculation unit calculating a reliability of the defocus amount calculated by the defocus amount calculation unit for each image region; The image region unit physical quantity reliability corresponding processing execution unit a configuration for executing a process of superimposing a color map, in which different colors are set according to the defocus amount of each image region, on the captured image and displaying the color map; comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1; An imaging apparatus that displays a color map, which enables identification of image regions where the reliability of the defocus amount in image region units is equal to or less than the low reliability threshold value Th1, superimposed on the captured image.

4. An image region physical quantity calculation unit that calculates a physical quantity that changes depending on a subject distance for each image region that is a divided region of a captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; an image region unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image region unit calculated by the image region unit physical quantity reliability calculation unit, The image region physical quantity calculation unit a defocus amount calculation unit that calculates a defocus amount for each image region; The image region unit physical quantity reliability calculation unit calculating a reliability of the defocus amount calculated by the defocus amount calculation unit for each image region; The image region unit physical quantity reliability corresponding processing execution unit an exposure time control is performed for each image region in accordance with the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit; comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1; An imaging apparatus that performs exposure time control to make the exposure time of an image area where the reliability of the defocus amount in image area units is equal to or less than the low reliability threshold value Th1 longer than that of other image areas.

5. An image region physical quantity calculation unit that calculates a physical quantity that changes depending on the subject distance for each image region that is a divided region of a captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; an image region unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image region unit calculated by the image region unit physical quantity reliability calculation unit, The image region physical quantity calculation unit a defocus amount calculation unit that calculates a defocus amount for each image region; The image region unit physical quantity reliability calculation unit calculating a reliability of the defocus amount calculated by the defocus amount calculation unit for each image region; The image region unit physical quantity reliability corresponding processing execution unit an exposure time control is performed for each image region in accordance with the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit; comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined high reliability threshold value Th2; An imaging apparatus that performs exposure time control to make the exposure time of an image area where the reliability of the defocus amount in image area units is equal to or greater than the high reliability threshold value Th2 shorter than that of other image areas.

6. An image region physical quantity calculation unit that calculates a physical quantity that changes depending on the subject distance for each image region that is a divided region of a captured image; an image region unit physical quantity reliability calculation unit that calculates the reliability of the physical quantity calculated by the image region physical quantity calculation unit; an image region unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image region unit calculated by the image region unit physical quantity reliability calculation unit, The image region physical quantity calculation unit a distance information calculation unit that calculates a distance value in units of the image region; The image region unit physical quantity reliability calculation unit executes a process of calculating reliability of the distance information calculated by the distance information calculation unit for each image region; The image region unit physical quantity reliability corresponding processing execution unit Performing a process to display distance ratios of objects at different distances; an imaging device that selects image areas where the reliability of the distance information for each image area calculated by the image area unit physical quantity reliability calculation unit is equal to or greater than a predetermined threshold, calculates a distance ratio between subjects in the selected image areas, and displays the calculated ratio on an image captured by the imaging device.

7. An image processing method executed in an imaging device, an image region physical quantity calculation step in which an image region physical quantity calculation unit calculates a physical quantity that changes depending on a subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation step in which an image region unit physical quantity reliability calculation unit calculates the reliability of the physical quantity for each image region calculated in the image region physical quantity calculation step; an image region unit physical quantity reliability corresponding processing execution unit executes an image region unit physical quantity reliability corresponding processing execution step of executing a control process according to the reliability of the physical quantity for each image region calculated in the image region unit physical quantity reliability calculation step; The image region physical quantity calculation unit a defocus amount calculation unit that calculates a defocus amount for each image region; The image region unit physical quantity reliability calculation unit calculating a reliability of the defocus amount calculated by the defocus amount calculation unit for each image region; The image region unit physical quantity reliability corresponding processing execution unit A configuration that executes a process of displaying mask stability information indicating whether or not a highly accurate mask setting is possible when masking an area other than the specific subject, comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1; When an image area having a reliability equal to or less than the low reliability threshold Th1 is detected in the captured image, An image processing method that displays mask stability information on the captured image, which indicates that highly accurate mask setting is difficult.

8. A program for causing an imaging device to perform image processing, an image region physical quantity calculation step of causing an image region physical quantity calculation unit to calculate a physical quantity that changes depending on the subject distance for each image region that is a divided region of the captured image; an image region unit physical quantity reliability calculation step of causing an image region unit physical quantity reliability calculation unit to calculate the reliability of the physical quantity for each image region calculated in the image region physical quantity calculation step; an image region unit physical quantity reliability corresponding processing execution step of causing an image region unit physical quantity reliability corresponding processing execution unit to execute a control process according to the reliability of the physical quantity for each image region calculated in the image region unit physical quantity reliability calculation step; and the image region physical quantity calculation unit, executes a defocus amount calculation process for calculating a defocus amount for each image region; the image region unit physical quantity reliability calculation unit, Calculating the reliability of the defocus amount calculated in the defocus amount calculation process for each image region; the image region unit physical quantity reliability correspondence processing execution unit, Execute a process to display mask stability information indicating whether or not a highly accurate mask setting is possible when masking an area other than the specific subject; executes a process of comparing the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1; When an image area having a reliability equal to or less than the low reliability threshold Th1 is detected in the captured image, A program for displaying mask stability information on the captured image, which indicates that highly accurate mask setting is difficult.

Images