Display control device and control method for the same

Abstract

To provide a display control device capable of displaying a suitable focal state and a control method for the same.SOLUTION: The display control device has a determination step for determining whether a plurality of pupils can be contained within a specified depth of focus when a plurality of pieces of pupil information of the same subject is acquired. The display control device controls to display different indicators for each of the pupils within the specified depth of focus and for each of the pupils outside the specified depth of focus when it is determined by the determination step that the plurality of pupils of the same subject is not possible to be contained within the specified depth of focus.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a display function that indicates the focus state of an image. [Background technology]

[0002] A technology known as eye AF has been known for some time in imaging devices such as digital cameras, which detects the eyes of people or animals (dogs, cats, birds, etc.) in an image captured by an imaging element and focuses on the detected eyes. Conventional eye AF functions focus on either the left or right eye, and do not guarantee that eyes other than the target eye will be in focus. However, in situations such as portrait photography, where eye AF is frequently used, there is a need not only to focus on one eye, but also to capture images in which both the left and right eyes are in focus. In such use cases, there is also a need to know whether both the left and right eyes can be focused on.

[0003] In response to such needs, Patent Document 1 discloses a technology for controlling depth of field so that both pupils of multiple people are within the depth of field. When it is not possible to fit all pupils within the depth of field, focus control is performed so that the largest number of pupils are within the depth of field. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-215403 Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 1 does not mention a suitable display showing the focus state for multiple pupils or a display for a pupil that cannot be included within the depth of field. Furthermore, because there is no suitable display showing whether multiple pupils can be included within the depth of field, it is difficult for the user to select whether to focus on multiple pupils or only on a specific pupil.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a display control device and a control method thereof that enable display of a suitable focus state. [Means for solving the problem]

[0007] A technical feature of the present invention is a display control method comprising: an acquisition step of acquiring pupil information from an image obtained by an imaging means that captures an image of a subject obtained through an imaging optical system; a focus detection step of detecting the focus state of the subject; a display control step of controlling the display of an index on a display screen based on the pupil information obtained in the acquisition step; and a judgment step of judging whether or not it is possible to fit multiple pupils of the same subject within a specific depth when multiple pupils of the same subject are acquired by the acquisition step; wherein the display control step controls the display of different indexes for pupils within the specific depth and pupils outside the specific depth when it is judged by the judgment step that it is not possible to fit multiple pupils of the same subject within the specific depth. [Effects of the Invention]

[0008] According to the present invention, it is possible to display a suitable focus state. [Brief explanation of the drawings]

[0009] [Figure 1] 1A is a side cross-sectional view showing the configuration of a digital single-lens reflex camera according to a first embodiment, and FIG. 1B is a block diagram showing the electrical configuration of the digital single-lens reflex camera. [Figure 2] FIG. 2 is an explanatory diagram showing a partial area of ​​an image sensor. [Figure 3]10A and 10B are diagrams illustrating image signals obtained from a detection area for detecting a defocus amount. [Figure 4] 10A and 10B are diagrams illustrating a correlation amount waveform, a correlation change amount waveform, and a focus deviation amount. [Figure 5] FIG. 10 is a diagram illustrating a method for calculating the degree of coincidence between two images. [Figure 6] 10 is a flowchart of a phase-difference AF process. [Figure 7] FIG. 2 is a diagram showing a processing flow in the first embodiment. [Figure 8] FIG. 3 is a diagram showing an example of a frame display in the first embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a frame display in the second embodiment. [Figure 10] FIG. 10 is a diagram showing a processing flow in the second embodiment. [Figure 11] 10A to 10C are diagrams illustrating an example of frame display and control in the third embodiment. [Figure 12] FIG. 10 is a diagram for explaining control in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0011] First Embodiment [Configuration of display control device] In this embodiment, an example in which the present invention is applied to a digital single-lens reflex camera (hereinafter also simply referred to as a camera) 100, which is an embodiment of a display control device of the present invention, is shown.

[0012] Fig. 1(A) is a side cross-sectional view showing the configuration of a camera 100. Fig. 1(B) is a block diagram showing the electrical configuration of the camera 100 of Fig. 1(A).

[0013] As shown in FIG. 1A, camera 100 of this embodiment has a detachable, interchangeable lens unit 120 attached to the front side (subject side) of camera body 101. Lens unit 120 includes a focus lens 121, an aperture 122, a focus lens driver (not shown), and an aperture driver (not shown). It is electrically connected to camera body 101 via mount contacts 123. Focus lens 121 is moved by the focus lens driver to adjust the focus of an image formed on image sensor 104 (described later). Aperture 122 is driven by the aperture driver to control the amount of light incident on image sensor 104 (described later). The focus lens driver and aperture driver are controlled by control unit 201 of camera body 101. The focus lens 121 can also be manually adjusted by the user.

[0014] The image sensor 104 is composed of a CMOS sensor or the like, and includes an infrared cut filter, a low-pass filter, etc. The image sensor 104 photoelectrically converts the subject image formed through the photographing optical system of the lens unit 120 during photographing, and outputs a signal for generating a photographed image and a signal for performing image plane phase difference AF to the arithmetic unit 102. The arithmetic unit 102 generates a photographed image from the acquired signal, stores the image in the image storage unit 107, and displays it on the display unit 105, such as an LCD. The arithmetic unit 102 also controls the shutter 103, which shields the image sensor 104 from light when not photographing and opens it during photographing to expose the image sensor 104 to light.

[0015] Next, the configuration related to control will be explained using Fig. 1(B). The arithmetic unit 102 is equipped with a multi-core CPU capable of parallel processing of multiple tasks, RAM, ROM, and dedicated circuits for executing specific arithmetic processing at high speed. With this hardware, the arithmetic unit 102 comprises a control unit 201, a main subject calculation unit 202, a tracking calculation unit 203, a focus calculation unit 204, and an exposure calculation unit 205. The control unit 201 controls each unit of the camera body 101 and the lens unit 120.

[0016] The main subject calculation unit 202 is configured to include a detector 213 and a main subject determination unit 214. The detector 213 performs processing to detect specific regions (for example, a human face or eyes, or an animal face or eyes) from an image. There are cases where no specific regions are detected, and cases where multiple specific regions are detected. The detector for human or animal eyes is included in the detector 213. Any known method such as AdaBoost or a convolutional neural network may be used as the detection method. Furthermore, the implementation form may be a program running on a CPU, dedicated hardware, or a combination of these.

[0017] The subject detection results obtained from detector 213 are sent to main subject determination unit 214, which determines the main subject from among the detected subjects and sets the main subject region. The determination of the main subject is performed using a known calculation method based on size, position, reliability of the detection result, etc. If detector 213 does not detect a specific region, the main subject region to be used as the main subject is determined based on past detection results, feature amounts such as edges of the target frame, defocus information of the subject, etc.

[0018] Tracking calculation section 203 tracks the main subject region based on the detection information of the main subject.

[0019] The focus calculation unit 204 performs correlation calculation based on the signal for performing image plane phase difference AF output from the image sensor 104, and calculates defocus information (defocus amount, reliability information (degree of coincidence between two images, degree of steepness between two images, contrast information, saturation information, scratch information, etc.)). The calculated defocus information (defocus amount and reliability information) is output to the control unit 201. Furthermore, the control unit 201 notifies the focus calculation unit 204 of a change in the settings for calculating the defocus amount and reliability information based on the acquired defocus amount and reliability information. Details of the correlation calculation will be described later using FIGS. 3 to 5.

[0020] An exposure calculation unit 205 calculates control values ​​for the aperture 122 and the image sensor 104 to achieve proper exposure for the main subject area.

[0021] Next, the control unit 201 receives the results of the exposure calculation unit 205 and the focus calculation unit 204 and controls the focus lens 121, the aperture 122, the display unit 105, etc. The control unit 201 includes a depth-priority control means 215. When multiple pupils of a person or animal are detected, the depth-priority control means 215 uses the detection information from the main subject calculation unit 202 and the defocus information from the focus calculation unit 204 to determine whether the multiple pupils can be contained within a set specific depth. If possible, the depth-priority control means 215 calculates control values ​​for the lens and aperture in that case. Based on the calculated control values, the focus lens 121 and the aperture 122 are controlled. In addition, in response to the control results, the display unit 105 displays a frame on the display screen indicating whether the subject is in focus, out of focus, etc. Here, the specific depth generally refers to the depth of field, but it may be an arbitrarily set depth. Furthermore, a subject that is contained within the specific depth (depth of field) is defined as being in focus.

[0022] The control unit 201 exchanges information with and controls the entire camera body 101. In addition to processing within the camera body 101, it also executes various camera functions operated by the user in response to input from the operation unit 106, such as turning the power on / off, changing settings, starting recording, starting AF control, and checking recorded video.

[0023] [Image sensor configuration] Figure 2 shows a portion of the light receiving surface of the image sensor 104, which serves as an image sensor. To enable image plane phase-difference AF, the image sensor 104 has an array of pixel units, each of which has two photodiodes serving as light receiving units that act as photoelectric conversion means for each microlens. This makes it possible for each pixel unit to receive a light beam that has been split by the exit pupil of the lens barrel.

[0024] For reference, Figure 2(A) is a schematic diagram of a portion of the image sensor surface in an example of a Bayer array of red (R), blue (B), and green (Gb, Gr). Figure 2(B) is an example of a pixel section in which two photodiodes serving as photoelectric conversion means are held for one microlens, corresponding to the color filter array in Figure 2(A).

[0025] An image sensor having such a configuration is capable of outputting two signals for phase-difference AF (hereinafter also referred to as image A signal and image B signal) from each pixel portion.

[0026] It is also possible to output a recording signal (image signal A+image signal B) obtained by adding the signals from the two photodiodes. This added signal is equivalent to the output of the image sensor in the Bayer array example outlined in FIG. 2A. Using the output signal from the image sensor 104 as this image sensor, the focus calculation unit 204 performs correlation calculations on the two image signals and calculates information such as the defocus amount and various types of reliability.

[0027] In this embodiment, a total of three signals, including a signal for imaging and two signals for phase-difference AF, are output from the image sensor 104. This is not limited to the above method. For example, a total of two signals, including a signal for imaging and one of the two image signals for phase-difference AF, may be output. In this case, after output, the other of the two image signals for phase-difference AF is calculated using the two output signals from the image sensor 104.

[0028] Also, in Figure 2, an example is shown in which pixel units each having two photodiodes as photoelectric conversion means are arranged in an array for one microlens. However, pixel units each having three or more photodiodes as photoelectric conversion means for one microlens may be arranged in an array. Furthermore, a plurality of pixel units each having a light-receiving portion with an aperture positioned differently relative to the microlens may be provided. In other words, it is sufficient as long as two signals for phase-difference AF capable of phase difference detection, such as an image A signal and an image B signal, are obtained as a result.

[0029] [Correlation calculation of image plane phase detection AF method] 3(D) is a conceptual diagram of the pixel array of the image sensor 104 as an image sensor, illustrating an example of an area from which an image signal is acquired. The area to be calculated, which will be described below, is area 304, in pixel array 303, where pixel units (not shown) are arranged in an array. Shift area 305, which is required for correlation calculation when calculating the defocus amount for area 304, is combined with area 304 to form shift area 306, which is required for correlation calculation.

[0030] 3 and 4, p, q, s, and t respectively represent coordinates in the x-axis direction, p to q represent the shift region 306, and s to t represent the region 304.

[0031] 3A, 3B, and 3C show image signals acquired from the shift region 306 set in FIG. 3D. s to t correspond to region 304, and p to q are image signals corresponding to shift region 306, the range required for calculation to calculate the defocus amount based on the shift amount. FIG. 3A is a diagram conceptually showing waveforms of the image A and B signals before shifting for correlation calculation. Solid line 301 represents image A signal A, and dashed line 302 represents image B signal.

[0032] 3B is a conceptual diagram showing the case where the image waveforms in (A) are mutually shifted in the positive direction relative to the image waveforms before the shift, and (C) is a conceptual diagram showing the case where the image waveforms in (A) are mutually shifted in the negative direction relative to the image waveforms before the shift. When calculating the correlation amount, which indicates the correlation between the two images, for example, the image A signal 301 and the image B signal 302 are shifted by one bit in the directions of the arrows, respectively.

[0033] Next, we will explain how to calculate the correlation amount COR. First, as shown in Figures 3(B) and 3(C), for example, image signals A and B are shifted by one bit at a time, and the sum of the absolute values ​​of the differences between image signals A and B in each state is calculated. In this case, the shift amount is represented by i, the minimum shift number is ps in Figure 4, and the maximum shift number is qt in Figure 4. Furthermore, x is the start coordinate of the focus detection area, and y is the end coordinate of the focus detection area. Using these, the correlation amount COR can be calculated using the following equation (1).

[0034]

number

[0035] FIG. 4(A) is a conceptual diagram showing the correlation amount in the form of a waveform graph. The horizontal axis of the graph represents the shift amount, and the vertical axis represents the correlation amount. This is an example of a correlation amount waveform 401 having extreme values ​​402 and 403. The smaller the correlation amount, the higher the degree of match between image A and image B.

[0036] Next, we will explain how to calculate the correlation change amount ΔCOR. First, using the conceptual diagram of the correlation amount waveform in Figure 4(A), the correlation change amount is calculated from the difference in the correlation amount for every other shift. In this case, the shift amount is represented by i, the minimum number of shifts is ps in Figure 3(D), and the maximum number of shifts is qt in Figure 3(D). Using these, the calculation can be performed using the following equation (2).

[0037]

number

[0038] 4B is a conceptual diagram illustrating the correlation change amount ΔCOR in the form of a waveform graph. The horizontal axis of the graph represents the shift amount, and the vertical axis represents the correlation change amount. The correlation change amount waveform 404 has points 405 and 406 where the correlation change amount changes from positive to negative. The state where the correlation change amount becomes 0 from point 405 is the shift amount between the image A and image B signals where the degree of match between the images A and B is relatively high. The shift amount at this time corresponds to the defocus amount.

[0039] FIG. 5(A) is an enlarged view of point 405 in FIG. 4(B), with waveform 501 being a portion of the correlation change amount waveform 404. Using FIG. 5(A), we will illustrate a method for calculating the defocus amount PRD corresponding to the defocus amount. The defocus amount is conceptualized as being divided into an integer part β and a decimal part α. The decimal part α can be calculated using the following equation (3) based on the similarity relationship between triangles ABC and ADE in the figure.

[0040]

number

[0041] Next, the decimal part β can be calculated from FIG. 5(A) using the following formula (4). β=k-1 (4)

[0042] As described above, the defocus amount PRD can be calculated from the sum of α and β.

[0043] Furthermore, when there are multiple zero crossings, as in Figure 4(B), the point where the steepness (maxder) of the change in correlation amount at the zero crossing is large is defined as the first zero crossing. This steepness is an index that indicates the ease of AF, and the larger the value, the easier it is to AF. Steepness can be calculated using the following equation (5):

[0044]

number

[0045] As described above, when there are multiple zero crossings, the first zero crossing is determined based on steepness.

[0046] Next, an example of a method for calculating the reliability level of the defocus amount will be described. This corresponds to the reliability of the defocus amount, but the following explanation is an example and other well-known methods may be used to calculate the reliability level. The reliability can be defined by the aforementioned steepness and the degree of coincidence fnclvl between the two images, the A image signal and the B image signal (hereinafter referred to as the two-image coincidence). The two-image coincidence is an index that represents the accuracy of the defocus amount, with the smaller the value, the better the accuracy. Figure 5(B) is an enlarged view of the area near the extreme value 402 in Figure 4(A), showing waveform 502, which is a portion of the correlation amount waveform 401. This will be used to illustrate a method for calculating the steepness and the two-image coincidence. The two-image coincidence can be calculated using the following equation (6).

[0047]

number

[0048] [Defocus amount calculation] FIG. 6 shows the flow of a series of processes up to calculation of the defocus amount. In the following explanation of the examples, the defocus amount and the defocus amount are distinguished from each other. In this regard, the defocus amount in the technical concept of the present application may be conceptualized as an absolute distance or number of pulses from the in-focus position, or may be a concept with a different dimension or unit from such concepts, or a relative concept. It is a concept that indicates how far away from the in-focus state it can be determined that the focus state is reached, and how much focus control is required to determine whether the focus state can be achieved. Obtaining defocus information in this way will be described as "obtaining focus information."

[0049] In step S601, image signals A and B are acquired from pixels at positions on the image sensor 104 corresponding to the respective areas set as exemplified above. Next, a correlation amount is calculated from the acquired image signals (step S602). Subsequently, a correlation change amount is calculated from the calculated correlation amount (step S603). Then, a focus error amount is calculated from the calculated correlation change amount (step S604). Furthermore, a reliability level indicating how reliable the calculated focus error amount is is calculated (step S605). These processes are performed a number of times corresponding to the number of focus detection areas. Then, the focus error amount is converted into a defocus amount for each focus detection area (step S606).

[0050] [Defocus amount confidence level] Next, the reliability level of the defocus amount in this embodiment will be described.

[0051] The reliability level of the defocus amount is an index indicating the likelihood of accuracy of the calculated defocus amount, and is calculated by the focus calculation unit 204. Basically, the reliability level is high when the calculated defocus amount is judged to be reliable, and the reliability level decreases as the amount becomes less reliable. In this embodiment, the reliability level is expressed as a number from 1 to 4, with 1 indicating the highest reliability and 4 indicating the lowest reliability. Details of each reliability level are as follows:

[0052] When the reliability level of the defocus amount is "1," this occurs when the contrast between the A and B image signals is high and the shapes of the A and B image signals are similar (the two-image coincidence level is high), or when the main subject image is already in focus. In this case, the defocus amount is trusted and driving is performed.

[0053] A defocus amount reliability level of "2" indicates a state in which the contrast between the A and B image signals is high and the shapes of the A and B image signals are similar, although not as high as a reliability level of "1." Alternatively, it indicates a state in which the main subject image is already positioned close to in-focus within a certain error range. In this case, the target position is determined based on the defocus amount and driving is performed.

[0054] When the reliability level of the defocus amount is "3," this means that although the two-image coincidence level calculated by the focus calculation unit 204 is lower than a predetermined value, there is a certain tendency in the correlation obtained by relatively shifting the A and B image signals, and the defocus direction is reliable. For example, this is often the case when the main subject is slightly blurred.

[0055] When the defocus amount and defocus direction are unreliable, the reliability level is determined to be "4." For example, this occurs when the contrast between the A and B image signals is low and the image coincidence level is also low. This often occurs when the subject is significantly out of focus, making it difficult to calculate the defocus amount.

[0056] [Process flow for displaying focus state] Next, the processing flow within the depth priority control means 215 in this embodiment will be described with reference to FIG.

[0057] In step S701, it is checked whether the detector 213 has detected the eyes of a person or animal from the image data generated by the image sensor 104. If an eye is detected, the process proceeds to step S702. If an eye is not detected, the process of the depth-priority control means 215 ends.

[0058] In step S702, it is checked whether both the left and right eyes are detected in the face of the subject detected in step S701. If both the left and right eyes are detected, the process proceeds to step S703. If only one eye is detected, the process proceeds to step S708.

[0059] In step S703, the depth difference between the left and right pupils is measured, which is detected from the pupil information of both pupils and the information from the focus calculation unit 204. Next, the process proceeds to step S704.

[0060] In step S704, the control values ​​for the position of the focus lens 121 and the aperture 122 are calculated so that the depth difference measured in step S703 falls within a specific depth (depth of field), and the calculated control values ​​are returned. If there are no control values ​​that fall within the specific depth (depth of field), or if there are restrictions such as the control value of the aperture 122 not being able to be changed due to settings of the display control device, a calculation result that indicates that control is not possible is returned. Next, the process proceeds to step S705.

[0061] In step S705, it is determined whether the result of step S704 indicates that both the left and right pupils can be accommodated within a specific depth (depth of field). If the determination result indicates that the pupils can be accommodated within the specific depth (depth of field), the process proceeds to step S706, and if not, the process proceeds to step S708.

[0062] In step S706, the position of the focus lens 121 and the aperture 122 are operated based on the control value calculated in step S704, and control is performed so that both pupils fall within a specific depth (depth of field). Next, the process proceeds to step S707.

[0063] In step S707, the display unit 105 is controlled to display a frame on the operation screen of the display control device indicating that both the left and right pupils are within a specific depth (depth of field), and the depth priority control means 215 then completes the process.

[0064] Step S708 is executed when both left and right pupils are not detected in the same subject in S702, or when both left and right pupils are detected in step S705 but control cannot be performed to include them within a specific depth (depth of field). If both left and right pupils are detected, control is performed to focus on the foreground pupil. If only one pupil is detected, focus control is performed on that pupil. Next, proceed to step S709.

[0065] In step S709, a frame indicating that the pupil is in focus is displayed for the pupil that was in focus in step S708. Furthermore, if both the left and right pupils are detected, a frame indicating that the pupil is out of focus and is outside the specific depth is displayed for the pupil that is not in focus, and the processing of the depth priority control means 215 is completed.

[0066] [Focus status display example] Next, an example of frame display in this embodiment in steps S707 and S709 in FIG. 7 will be described with reference to FIG. 8. The upper figures (8-A-1) and (8-B-1) in FIG. 8 show front views, while the lower figures (8-A-2) and (8-B-2) show top views from the top of the head. Column (A) shows a case where both pupils are within a specific depth (depth of field), while column (B) shows a case where only one pupil is within the specific depth. In FIGS. (8-A-1) and (8-A-2), both the left and right pupils are in focus, so frames 800 and 801 indicating focus are displayed for both the left and right pupils (S707 in FIG. 7). In the cases of FIGS. (8-A-1) and (8-A-2), the depth difference 804 between the left and right pupils is within the depth of field. On the other hand, in the cases of (8-B-1) and (8-B-2), only the left pupil is in focus. A frame 803 indicating that the left pupil is in focus is displayed, and a frame 802 indicating that the right pupil, which is out of focus, is displayed (S709 in FIG. 7). The depth difference 805 between the left and right pupils is greater than the depth of field. In the example of FIG. 8, the frame displays 800, 801, and 803 indicating that the pupil is in focus are displayed using solid lines, and the frame display 802 indicating that the pupil is out of the specific depth (depth of field) is displayed using dashed lines, but the display method is not limited to this. For example, the meaning of the frame may be expressed by using different colors for the frame.

[0067] As described above, in this embodiment, when both the left and right eyes of the same subject are detected, it is possible to clearly indicate to the user whether both eyes are within a specific depth (depth of field), thereby meeting the need to capture an image with both eyes in focus, which is required in use cases such as portrait photography.

[0068] <Second embodiment> Next, a second embodiment of the present invention will be described with reference to FIGS.

[0069] [Focus status display example] An example of frame display and operation of this embodiment will be described using Figure 9. In the first embodiment, when it is possible to focus on both the left and right pupils, the focus lens 121 and the aperture 122 are operated and a frame is displayed so as to focus on both pupils. On the other hand, in this embodiment, there are a mode in which focusing is prioritized on only one pupil (referred to as single pupil priority mode in the figure) and a mode in which focusing is prioritized on both pupils if possible (referred to as double pupil priority mode in the figure), and a display control device is assumed that can switch between these modes.

[0070] In Figure 9, the upper figures (9-A-1) and (9-B-1) show examples of frame display in (1) both-pupil priority mode, and the lower figures (9-A-2) and (9-B-2) show examples in (2) one-pupil priority mode. Also, the figures in column (A) show cases where it is possible to focus on both the left and right pupils, and the figures in column (B) show cases where it is possible to focus on only one pupil.

[0071] The (1) both-pupil priority mode shown in the upper part of FIG. 9 is the same as that of the first embodiment. FIG. (9-A-1) shows an example in which the position of the focus lens 121 and the aperture 122 are operated to control both pupils to fall within a specific depth (depth of field), and then a frame is displayed on the operation screen of the display control device to indicate that both left and right pupils are within the specific depth (depth of field). The frame display indicating that both left and right pupils are within the specific depth (depth of field) means that it was possible to focus on both pupils. FIG. (9-B-1) shows a case in which both left and right pupils are detected, but control cannot be performed to include them within the specific depth (depth of field), in which control is performed to focus on the foreground pupil. Then, a frame display 905 indicating focus is achieved is displayed for the left pupil, and a frame display 904 indicating that focus is impossible for the right pupil, which is not in focus, because it is outside the specific depth (depth of field).

[0072] In the (2) one-pupil priority mode shown in the lower part of Fig. 9, in the case where the depth priority control means 215 has determined that focusing on both pupils is possible (Fig. 9-A-2), focusing control is performed on the left pupil located in the foreground, and a frame display 903 indicating focusing is performed is displayed. For the other pupil, a frame display 902 is displayed, indicating that focusing control is possible to keep the other pupil within a specific depth (depth of field). Also, in the case where focusing on both pupils is not possible (Fig. 9-B-2), focusing control is performed on the left pupil located in the foreground, and a frame display 903 indicating focusing is performed is displayed. Because the other pupil has not been detected, no frame display is displayed.

[0073] 9, in-focus frame displays 900, 901, 903, and 905 are represented by solid black lines, frame display 902 indicating that focus control is possible is represented by a dashed black line, and frame display 904 indicating that focus is not possible is represented by a dashed gray line. However, the display method is not limited to this as long as the three types of frame displays can be distinguished. For example, a display method in which the frame color is changed for each case may also be used.

[0074] [Process flow for displaying focus state] Next, the processing flow in the depth-priority control means 215 in this embodiment will be described with reference to Fig. 10. Explanation of parts that overlap with the explanation of the processing flow using Fig. 7 will be omitted.

[0075] Steps S1001 to S1004 are the same as those in FIG. 7 referred to in the first embodiment, and therefore a description thereof will be omitted.

[0076] Next, in step S1005, it is determined whether the display control device is set to the both-eye depth priority mode. If it is set to the both-eye depth priority mode, the process proceeds to step S605, and if it is set to the one-eye priority mode, the process proceeds to step S1011.

[0077] Next, when the process proceeds to step S1006, the subsequent processing steps S1007 to S1010 are the same as steps S706 to S709 described with reference to FIG. 7, and therefore the description thereof will be omitted.

[0078] Next, the process proceeds to step S1011. In step S1011, the focus lens 121 and the aperture 122 are controlled to focus on the near pupil. Next, the process proceeds to step S1012.

[0079] In step S1012, similar to step S1006, it is determined whether both the left and right pupils have been detected and whether it is possible to control both pupils to be within a specific depth (depth of field). If the determination result shows that it is possible to control, the process proceeds to step S1013; if not, the process proceeds to step S1014.

[0080] Next, step S1013 will be described. A frame is displayed for the pupil that is not in focus, either the left or right pupil, indicating that focus control is possible to keep it within a specific depth (depth of field).

[0081] Next, step S1014 will be described. Here, a frame is displayed for the pupil that has been focused based on the control value calculated in step S1004, indicating that the pupil is within a specific depth (depth of field). With the above processing, the processing of the depth-priority control means 215 in this embodiment is completed.

[0082] As described above, in this embodiment, processing has been described assuming a display control device that has both pupil priority mode and single pupil priority mode. When both left and right pupils are detected for the same subject, if there is control that can focus on both pupils even when single pupil priority mode is set, a frame is displayed indicating that there is control that can focus on the pupil that is not in focus at that time. This frame display makes it possible for the user to fit both pupils within a specific depth (depth of field), and the user can select whether to prioritize focusing on one pupil or both pupils.

[0083] <Third embodiment> Next, a third embodiment of the present invention will be described with reference to FIG.

[0084] [Example of focus status display and focus control] FIG. 11 (11-1) shows a state in which both left and right pupils have been detected and are in focus, with 1101 and 1102 being frame displays indicating that the detected pupils are in focus. In this situation, in FIG. 11 (11-2), one pupil is not detected because hair is covering it, and the frame display for that pupil disappears. However, as shown in FIG. 11 (11-3), the focus control for both pupils that was previously in progress is maintained for a certain period of time. If both pupils become detectable again within a certain period of time, as shown in FIG. 11 (11-4), the focus control for both pupils continues. Furthermore, as shown in FIG. 11 (11-5), if the undetected pupil is not redetected after a certain period of time, the control values ​​of the focus lens 121 and aperture 122 are changed to control values ​​appropriate for the detected pupil.

[0085] As described above, in this embodiment, the control when one of the pupils becomes undetected after both the left and right pupils are in focus has been described. By performing the control described above, it is possible to suppress flickering of the control values ​​of the focus lens 121 and the aperture 122, and to achieve stable focusing control.

[0086] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described with reference to FIG.

[0087] [Example of depth of field adjustment] FIG. 12 shows a top view of the subject from the head, with FIG. 12 (12-1) showing a stationary subject and FIG. 12 (12-2) showing a moving subject. Both FIG. 12 (12-1) and FIG. 12 (12-2) show a situation in which both the left and right pupils are in focus. Therefore, frames 1201 and 1202 indicate focus. 1203 and 1206 indicate the difference in focus between the two pupils, 1204 and 1207 indicate the focus positions, and 1205 and 1208 indicate the specific depth (depth of field). When a subject is moving, as in FIG. 12 (12-2), the focus lens 121 may not be able to keep up with the subject's movement, potentially causing both pupils to deviate from the specific depth (depth of field). Therefore, the specific depth (depth of field) 1208 is set deeper than the specific depth (depth of field) 1205 in the case of Fig. 12 (12-1). One way to deepen the depth is to increase the aperture of the aperture 122.

[0088] As described above, in this embodiment, the specific depth (depth of field) when focusing on both the left and right pupils is changed according to the movement of the subject. As a setting policy for the specific depth (depth of field), it is desirable to set it deeper when the subject is moving than when the subject is stationary. By setting the specific depth (depth of field) in this way, it is possible to stably maintain the focused state of both the left and right pupils even when the subject moves back and forth.

[0089] (Other embodiments) The present invention can be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a computer-readable storage medium that stores the program.The present invention can also be realized by one or more processors in the computer of the system or device reading and executing the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0090] Furthermore, in the above-described embodiments, the present invention has been described with reference to a digital single-lens reflex camera as an example, but this is not limited to this example. That is, the present invention may be applied to any device equipped with an image sensor. That is, the present invention may be applied to any device capable of capturing images, such as a mobile phone terminal, a portable image viewer, a television equipped with a camera, a digital photo frame, a music player, a game console, or an e-book reader.

[0091] Furthermore, in the above embodiment, a lens-interchangeable camera has been described, but the present invention may also be applied to a camera with a lens barrel in which the lens barrel and the camera body are integrated. [Explanation of symbols]

[0092] 100 Display control device 101 Camera body 102 Arithmetic equipment 104 Image sensor 105 Display section 107 Image storage unit 120 Lens Unit

Claims

1. an acquisition means for acquiring pupil information from an image obtained by an imaging means that captures an object image obtained through an imaging optical system; a focus detection means for detecting a focus state of a subject; a display control means for controlling the display of an index on a display screen based on the pupil information obtained by the acquisition means; a determination means for determining whether or not it is possible to fit the plurality of pupils within a specific depth when pupil information of the plurality of pupils of the same subject is acquired by the acquisition means, The display control device is characterized in that, when the judgment means determines that it is not possible to fit multiple pupils of the same subject within a specific depth, the display control means controls the display of different indicators for pupils within the specific depth and pupils outside the specific depth.

2. a focus adjustment unit that controls the movement of a focus lens included in the photographing optical system to control focusing; The imaging device further includes an aperture control unit for controlling the driving of an aperture included in the imaging optical system, 2. The display control device according to claim 1, wherein the focus adjustment means and the aperture control means perform control based on the determination result of the determination means.

3. The display control device according to claim 2, characterized in that, when information about one of the pupils cannot be obtained while multiple pupils are within a specific depth, the focus adjustment means and the aperture control means maintain control to keep the multiple pupils within the specific depth for a certain period of time, and after the certain period of time has elapsed, perform control based on the pupil information that has been obtained.

4. 4. The display control device according to claim 2, wherein the focus adjustment means and the aperture control means change control values ​​in accordance with the movement of a subject that is the subject of control to fit within a specific depth.

5. 3. The display control device according to claim 2, wherein the display control device has a mode in which focusing is given priority to only one pupil and a mode in which focusing is given priority to both pupils.

6. In a mode in which focus is given priority to only one pupil, focus control is performed on the left pupil located in the foreground by the focus adjustment means, A display control device as described in claim 5, characterized in that in a mode that prioritizes focusing on both pupils, if the judgment means determines that it is possible to fit multiple pupils of the same subject within a specific depth, the focus adjustment means and the aperture control means perform control to fit the multiple pupils of the same subject within the specific depth.

7. The display control device according to claim 6, characterized in that in a mode in which priority is given to focusing on both pupils, if the determination means determines that it is not possible to fit multiple pupils of the same subject within a specific depth, the focus adjustment means performs focus control on the left pupil located in the foreground.

8. an acquisition step of acquiring pupil information from an image obtained by an imaging means that captures an object image obtained through an imaging optical system; a focus detection step of detecting a focus state of a subject; a display control step of controlling the display of an index on a display screen based on the pupil information obtained in the acquisition step; a determining step of determining whether or not it is possible to fit the plurality of pupils within a specific depth when pupil information of the plurality of pupils of the same subject is acquired by the acquiring step, A display control method characterized in that, in the display control process, if it is determined by the judgment process that it is not possible to fit multiple pupils of the same subject within a specific depth, different indicators are displayed for pupils within the specific depth and pupils outside the specific depth.

9. A program for causing a computer to execute each step of the display control method according to claim 8.

10. A computer-readable storage medium storing a program for causing a computer to execute each step of the display control method according to claim 8.

Images