Photography assistance device, photography assistance method, and program

Abstract

To clearly show the photographer which subject is in focus. [Solution] The shooting assistance device detects a subject from an image captured by imaging in accordance with at least the input of shooting settings related to focusing, and estimates a defocus range that represents the range of values ​​for the amount of defocus for the subject. Then, based on the subject detection result and the defocus range estimation result, the shooting assistance device selects at least one subject from among the detected subjects and outputs information regarding focusing for the selected subject.

Description

Technical Field

[0001] The present invention relates to a technique for assisting photography.

Background Art

[0002] Many imaging devices have a focus adjustment function by manual operation. However, in order to focus on a subject as intended by manual operation, a photographer needs a certain amount of experience and training. In order to reduce the difficulty of such manual focusing, Patent Document 1 discloses a method of superimposing and displaying the degree of focus deviation, that is, the degree of focus on the screen.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technique disclosed in the above-mentioned Patent Document 1 displays the degree of focus on a determined area within the screen, and there is a problem that it is difficult to know which subject within the screen is in focus.

[0005] Therefore, an object of the present invention is to be able to clearly present to a photographer which subject is in focus.

Means for Solving the Problems

[0006] The present invention provides a shooting assistance device that includes: a detection means for detecting a subject from an image captured by imaging in response to input shooting settings related to at least focusing; an estimation means for estimating a defocus range representing a range of values ​​for the amount of defocus for the subject; a selection means for selecting at least one subject from among the subjects detected by the detection means based on the subject detection result and the defocus range estimation result; and an output means for outputting information relating to focusing for the selected subject. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to clearly show the photographer which subject is in focus. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the functional configuration of the shooting assistance device according to the first embodiment. [Figure 2] This figure shows an example of the hardware configuration of an imaging device, including an imaging support device. [Figure 3] This is a flowchart outlining the shooting assistance process according to the first embodiment. [Figure 4] This is an explanatory diagram of the amount of defocusing. [Figure 5] This is an explanatory diagram of the defocus range. [Figure 6] This is a detailed flowchart of the shooting assistance process according to the first embodiment. [Figure 7] This is an explanatory diagram of an example of subject detection. [Figure 8] This is an explanatory diagram for estimating the defocus range. [Figure 9] This is an explanatory diagram illustrating an example of selecting a candidate subject for emphasis in the first embodiment. [Figure 10] This is an explanatory diagram illustrating an example of selecting a subject from multiple subjects. [Figure 11] This is an explanatory diagram of an example of highlighting in the first embodiment. [Figure 12]This is a detailed flowchart of the photographic support process related to the modified image. [Figure 13] This is an explanatory diagram illustrating an example of highlighting in a modified example. [Figure 14] This figure shows an example of a focus guide in a modified example. [Figure 15] This figure shows an example of focus guidance for multiple areas in a modified example. [Figure 16] This figure shows an example of the functional configuration of the shooting assistance device according to the second embodiment. [Figure 17] This is a detailed flowchart of the shooting assistance process according to the second embodiment. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments and modifications are not intended to limit the present invention, and not all combinations of features described in each embodiment are essential to the solution of the present invention. The configuration of each embodiment may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). Furthermore, some of the embodiments and modifications described later may be combined as appropriate. In the following embodiments, the same or similar components and processing steps are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] <First Embodiment> Figure 1 is a functional block diagram showing an example of the functional configuration of the imaging assistance device according to the first embodiment. Each functional unit of the imaging assistance device shown in Figure 1 is realized by the system control unit 212 of the imaging device having the hardware configuration shown in Figure 2, which will be described later, executing processes according to the flowcharts in Figures 3 and 6, which will be described later. Hereinafter, an overview of each functional unit of the imaging assistance device of this embodiment will be described with reference to Figure 1.

[0011] The setting unit 101 changes the shooting settings of the imaging device based on the shooting setting input from the user (hereinafter referred to as the photographer). Here, the shooting settings of the imaging device are, for example, settings such as the focus position setting, aperture, shutter speed, ISO sensitivity, etc. Although details will be described later, in this embodiment, as an example of the input of the shooting settings by the photographer, the case where the focus position (focus position) is set by manual operation is taken as an example.

[0012] The detection unit 102 detects one or more subjects from an image (hereinafter referred to as a live view image) displayed based on the captured image captured by the imaging device. In this embodiment, the "subject" is a way of referring to the whole of the subject (for example, a person) or parts (for example, in the case of a person, the torso, head, eyes, etc.). Hereinafter, in this embodiment, the "subject" including the whole and parts of the subject will be described, but when a distinction is particularly necessary, it will be clearly stated as the "whole of the subject" or the "part of the subject". Details of the subject detection process will be described later.

[0013] The range estimation unit 103 estimates the defocus range for the subject detected by the detection unit 102 from the live view image of the imaging device. Details of the defocus range and its estimation process will be described later. The selection unit 104 selects the subject for which information regarding the shooting settings should be output based on the defocus range estimated by the range estimation unit 103 from the subjects detected by the detection unit 102. Although details will be described later, in this embodiment, as an example of subject selection, when focusing manually, the in-focus (focused) subject and the subject with a high degree of focus are selected. The output unit 105 outputs information regarding the shooting settings for the subject selected by the selection unit 104. Although details will be described later, in this embodiment, as an example of the output of information regarding the shooting settings, a frame indicating the subject selected by the selection unit 104 is emphasized and superimposed on the live view image.

[0014] Figure 2 is a block diagram showing an example of the main hardware configuration of the imaging device 200 according to this embodiment. The imaging device 200 illustrated in Figure 2 is, for example, a lens-interchangeable digital camera, and is composed of a camera body 210 and a lens unit 230 of an imaging optical system that guides incident light to the image sensor 211. Although Figure 2 shows the configuration when the imaging device 200 is a lens-interchangeable digital camera, the imaging device is not limited to a digital camera, but may be, for example, a built-in camera of a smartphone.

[0015] First, let me explain the camera body 210. The image sensor 211 is composed of, for example, a CMOS type imaging sensor, and converts the optical image formed by the lens unit 230 into an electrical signal. The optical image is formed when light rays incident on the imaging lens 231 are imaged onto the image sensor 211 via the aperture 232 and shutter 213.

[0016] The system control unit 212 incorporates a well-known CPU and controls various parts within the camera body 210. The system control unit 212 further includes an image processing unit that performs various image processing on the imaging signal obtained by the image sensor 211. The system control unit 212 also further includes a phase-difference AF unit that performs focus detection processing using a phase-difference detection method based on focus detection image data (signals for phase-difference AF) acquired by the image sensor 211 and processed by the image processing unit. More specifically, the image processing unit generates a pair of image data formed by light beams passing through a pair of pupil regions of the imaging optical system by the lens unit 230 as focus detection image data. The phase-difference AF unit detects the amount of focus shift based on the amount of shift of the pair of image data. In this way, the phase-difference AF unit performs phase-difference AF (image plane phase-difference AF) based on the output information of the image sensor 211 without using a dedicated AF sensor. In particular, in this embodiment, the system control unit 212 realizes each functional part of the shooting assistance device of this embodiment shown in Figure 1 by executing processing according to the flowcharts in Figures 3 and 6, which will be described later.

[0017] Memory 214 stores programs, variables, constants, etc., for the operation of the system control unit 212. The programs include those that enable the CPU of the system control unit 212 to implement the processing of each flowchart according to this embodiment, which will be described later. Memory 214 also includes electrically erasable and restorable non-volatile memory, which stores various parameters, setting values ​​such as ISO sensitivity, shooting modes, and various correction data.

[0018] The power switch 215 switches the camera body 210 on or off. The mode switching section 216 is a switch for switching and setting various shooting modes such as live view shooting and video recording. The rear monitor 217 is configured to display operating status such as characters, images, and sounds, as well as captured information such as messages, in accordance with the execution of a program in the system control unit 212, and includes an LCD device and LEDs. The touch panel 218 is positioned in an area approximately the same size as the monitor surface of the rear monitor 217, and detects contact with a finger or pen, notifying the system control unit 212 of the contact position relative to the rear monitor 217. The system control unit 212 then executes the processing of operations or functions associated with those contact positions.

[0019] The viewfinder display unit 219, like the rear monitor 217, displays shooting information and the like in accordance with the execution of a program in the system control unit 212, and constitutes an electronic viewfinder (EVF) visible through the eyepiece lens 220. The eyepiece detection unit 221 detects the photographer's eyepiece position in the EVF. Based on the detection result of the eyepiece detection unit 221, the system control unit 212 selectively displays the aforementioned shooting information, etc., on the rear monitor 217 or the viewfinder display unit 219.

[0020] Next, I will explain the configuration of the lens unit 230. The camera body 210 and the lens unit 230 are mechanically and electrically connected via a lens mount mechanism 223. The lens mount mechanism 223 is configured to allow the lens unit 230 to be attached to and detached from the camera body 210. The lens unit 230 comprises an imaging lens 231, an aperture 232, a focus ring (not shown), a lens drive circuit 233, an aperture control circuit 234, and a lens control unit 235. Note that, for simplification, only one lens is shown for the imaging lens 231 in Figure 2; however, the imaging lens 231 is actually composed of multiple imaging lens groups.

[0021] The lens control unit 235 controls the entire lens unit 230. The lens control unit 235 includes a memory (not shown) that stores various constants, variables, and programs for lens operation. The lens control unit 235 also includes a non-volatile memory (not shown) that stores information specific to the lens unit, such as maximum and minimum aperture values ​​and focal length. The lens drive circuit 233 drives the focus lens and other components included in the imaging lens 231 of the lens unit 230 under the control of the lens control unit 235. The aperture control circuit 234 drives the aperture 232 under the control of the lens control unit 235.

[0022] Furthermore, the lens control unit 235 communicates with the system control unit 212 of the camera body 210 via the lens mount mechanism 223. For example, when the imaging device 200 is in autofocus mode, the system control unit 212 controls the lens drive circuit 233 via the lens control unit 235 to drive the focus lens included in the imaging lens 231, thereby performing autofocus operation. In autofocus mode, the system control unit 212 controls the lens drive circuit 233 to drive the focus lens based on the defocus amount calculated using the output information of the image sensor 211, thereby performing autofocus operation.

[0023] The imaging device 200 also offers a manual focus mode. Manual focus mode is a shooting mode in which the autofocus function is disabled. In manual focus mode, the photographer can adjust the focus position to focus on the desired subject by manually rotating a focus ring (not shown). For example, if the photographer rotates the focus ring clockwise, the focus lens is moved in accordance with the clockwise rotation of the focus ring, and the focus position is adjusted to focus on a subject that is further away. On the other hand, if the photographer rotates the focus ring counterclockwise, the focus lens is moved in accordance with the counterclockwise rotation of the focus ring, and the focus position is adjusted to focus on a subject that is closer. In the imaging device 200 of this embodiment, when the photographer operates the focus ring, the position information of the focus lens moved in accordance with the operation of the focus ring is sent to the system control unit 212 via the lens control unit 235. As a result, the system control unit 212 can recognize how the focus position has been adjusted based on the position information of the focus lens.

[0024] Figure 3 is a flowchart showing the general flow of the imaging assistance processing performed by the system control unit 212 of the imaging device 200 of this embodiment when the imaging device of this embodiment is in manual focus mode. Each processing step shown in the flowchart of Figure 3 is performed in each functional unit of Figure 1, which is formed when the system control unit 212 of Figure 2 executes the program according to this embodiment. Hereinafter, the outline of the imaging assistance processing performed by each functional unit of the imaging assistance device of this embodiment will be explained with reference to the flowchart of Figure 3. In the flowcharts thereafter, the symbol S represents each processing step.

[0025] In this embodiment, the system control unit 212 of the imaging device starts processing from S301 onwards in the flowchart of Figure 3 upon receiving a shooting start instruction from the photographer. Here, starting shooting means, for example, turning on the power of the imaging device and putting it into shooting mode, or making the imaging device ready to shoot from a sleep state by half-pressing the shutter button. If the imaging device is the built-in camera of a smartphone, starting shooting may also mean activating the smartphone's camera application. In S301, the setting unit 101 receives shooting setting input from the photographer and changes the shooting settings of the imaging device 200 according to the received shooting setting input. In this embodiment, the shooting setting of the imaging device is, as described above, the adjustment of the focus position (focus position adjustment) to focus on the subject by manual operation.

[0026] Next, in the S302 process, the detection unit 102 detects one or more subjects from the live view image as described later. Next, in S303, the range estimation unit 103 estimates the defocus range from the live view image as described later. Next, in the S304 process, the selection unit 104 selects a subject from among the one or more subjects detected by the detection unit 102 that should output information regarding the shooting settings, based on the defocus range estimated by the range estimation unit 103. In this embodiment, since the shooting setting is an example of setting the focus position by manual operation, the selection unit 104 selects a subject based on the focus position set by manual operation as the subject for which information regarding the shooting settings should be output. As will be described in detail later, the selection unit 104 selects subjects that are in focus and subjects with a high degree of focus.

[0027] Next, in the S305 process, the output unit 105 outputs information regarding the shooting settings for the subject selected in the selection unit 104. As will be described in detail later, the output unit 105, as outputting information regarding the shooting settings, highlights the frame indicating the subject selected in the selection unit 104 and overlays it on the live view image. Subsequently, in the S306 process, the system control unit 212 determines whether or not to continue shooting. The system control unit 212 determines to continue shooting unless the photographer gives an instruction to end shooting, and repeatedly executes the processes from S301 to S305. When the photographer gives an instruction to end shooting, the process of the flowchart in Figure 3 is terminated. Here, an instruction to end shooting includes, for example, turning off the power to the imaging device, switching the imaging device from shooting mode to photo viewing mode, or when the imaging device enters sleep mode after a predetermined period of inactivity without any input from the photographer. Note that if the imaging device is the built-in camera of a smartphone, ending shooting may also be defined as closing the smartphone's camera application, etc.

[0028] Next, we will explain the amount of defocus calculated by the system control unit 212 using the output information of the image sensor 211, and the defocus range estimated based on that amount of defocus. First, let's explain the amount of defocus. Figure 4 illustrates the relationship between the defocus amount d of the imaging optical system of the lens unit 230 and the phase difference (image shift amount) between the first focus detection signal and the second focus detection signal obtained from the output information of the image sensor 211.

[0029] In Figure 4, lens 401 represents the imaging lens 231 of the lens unit 230 described above. The imaging surface 400 of the image sensor 211 is positioned perpendicular to the optical axis of lens 401. The exit pupil of the imaging optical system formed by lens 401 is divided into two parts: a first pupil region 411 and a second pupil region 412. The defocus amount d is |d|, which is the distance (size) from the imaging position C of the light beam from subjects 421 and 422 to the imaging surface 400. When the imaging position C is on the imaging surface 400 and the image is in focus, d=0. The defocus amount d is defined such that a negative sign (d<0) represents a front-focus state where the imaging position C is on the subject side of the imaging surface 400, and a positive sign (d>0) represents a state where the imaging position C is on the opposite side of the imaging surface 400 from the subject. In the imaging optical system shown in Figure 4, the subject 421 is in focus (d=0), while the subject 422 is in front focus (d<0). In this embodiment, the front focus state (d<0) and the back focus state (d>0) are combined to form a defocused state (|d|>0).

[0030] In the front-focused state (d<0), the light beam from the subject 422 that passes through the first pupil region 411 is first focused, and then spreads out with a width Γ1 centered on the centroid position G1 of the light beam, forming a blurred image on the image sensor 400. This blurred image is received by each first focus detection pixel on the image sensor 211, and a first focus detection signal is generated. In other words, the first focus detection signal is a signal that represents a subject image on the image sensor 400 in which the subject 322 is blurred by a blur width Γ1 at the centroid position G1 of the light beam. The blur width Γ1 of the subject image increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|.

[0031] Similarly, in the front-focused state (d<0), the light beam from the subject 422 that passes through the second pupil region 412 is focused and then spreads out with a width Γ2 centered on the centroid position G2 of the light beam, forming a blurred image on the imaging surface 400. This blurred image is received by each second focus detection pixel on the image sensor 211, and a second focus detection signal is generated. In other words, the second focus detection signal is a signal that represents a subject image in which the subject 322 is blurred by a blur width Γ2 at the centroid position G2 of the light beam on the imaging surface 400. The blur width Γ2 of the subject image increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|.

[0032] Similarly, in the case of a front-focused state (d<0), the magnitude of the image shift amount p (=difference in the centroid position of the light beam G1-G2) between the first focus detection signal and the second focus detection signal, |p|, also increases roughly in proportion to the increase in the magnitude of the defocus amount d, |d|. On the other hand, in the case of back-focused conditions (d>0), the direction of image shift between the first focus detection signal and the second focus detection signal is the opposite of that in the case of front-focused conditions described above.

[0033] Thus, as the amount of defocus increases, the amount of image shift between the first focus detection signal and the second focus detection signal also increases. The system control unit 212 in this embodiment performs focus detection using an imaging plane phase difference detection method, which calculates the amount of defocus from the amount of image shift between the first focus detection signal and the second focus detection signal obtained from the output information of the image sensor 211. That is, the phase difference AF unit converts the amount of image shift into a defocus amount using a conversion coefficient calculated based on the baseline length, given the relationship that the amount of image shift between the first focus detection signal and the second focus detection signal increases as the amount of defocus of the imaging signal increases. In this embodiment, the product of the F value of the aperture in the imaging optical system during image acquisition and the allowable circle of confusion diameter δ [Fδ] is used as the unit of defocus.

[0034] Next, I will explain the defocus range. In this embodiment, the defocus range refers to the range of values ​​of the amount of defocus that the subject has. Figures 5(a) and 5(b) are used to explain the defocus range. Figure 5(a) shows a person 511 being photographed using the imaging device 200. Range 512 in Figure 5(a) represents the extent of the object (pupil) in the depth direction when viewed from the imaging device 200, and this range 512 corresponds to the range of defocus values ​​of the person's pupil, i.e., the defocus range for the pupil. Similarly, range 513 in Figure 5(a) represents the extent of the object (face) in the depth direction when viewed from the imaging device 200, and this range 513 corresponds to the range of defocus values ​​of the person's face, i.e., the defocus range for the face. Similarly, range 514 represents the extent of the object (torso) in the depth direction when viewed from the imaging device 200, and this range 514 corresponds to the range of defocus values ​​of the person's torso, i.e., the defocus range for the torso. In Figure 5(a), the focus position 515 indicates that the focus position of the imaging device 200 coincides with the range 512 of the person's pupil.

[0035] The system control unit 212 in Figure 2 calculates the amount of defocus using the output information of the image sensor 211, and the range estimation unit 103 further estimates the defocus range based on that amount of defocus. Figure 5(b) schematically shows the estimated defocus range for the range of the person's pupil 512, the range of the person's face 513, and the range of the person's torso 514, as exemplified in Figure 5(a). The horizontal axis in Figure 5(b) represents the magnitude of the amount of defocus, with the side closer to the imaging device 200 defined as "near" and the side further away as "far". The length of the line segment indicated by the arrow in Figure 5(b) represents the defocus range, which is the range of values ​​for the amount of defocus. Figure 5(b) shows the estimated defocus range 521 for the pupil range 512 in Figure 5(a), the estimated defocus range 522 for the face range 513 in Figure 5(a), and the estimated defocus range 523 for the torso range 514 in Figure 5(a).

[0036] In Figure 5(a), for example, the depth of the person's torso as seen from the imaging device 200 is such that the closest point is the tip of the person's nose and the furthest point is the tip of the person's shoulder. Therefore, the closest (maximum) defocus amount of the person's torso is the defocus amount that corresponds to the tip of the person's nose, and the furthest (minimum) defocus amount is the defocus amount that corresponds to the tip of the person's shoulder. In other words, the range of values ​​defined by these closest and furthest defocus amounts is the defocus range of the person's torso. In Figure 5(b), the defocus range 523 of the person's torso is estimated from these closest and furthest defocus amounts. Furthermore, if the product of the aperture F-number of the imaging optical system and the allowable circle of confusion diameter δ [Fδ] is used as the unit of defocus amount, for example, if the closest value is 0.2Fδ and the farthest value is -1.4Fδ, the defocus range of the person's torso will be in the range of 0.2Fδ to -1.4Fδ. The process of estimating the defocus range of the subject by the range estimation unit 103 will be explained in more detail later.

[0037] Figure 6 is a flowchart showing the detailed flow of the shooting assistance process in the system control unit 212, which overlays the in-focus subject frame onto the live view image in accordance with manual adjustment of the focus position. Note that the flowchart in Figure 6 also shows some processes that overlap with the flowchart in Figure 3 described above. The processes shown in the flowchart in Figure 6 are performed by the various functional units in Figure 1, which are formed when the system control unit 212 shown in Figure 2 executes the program according to this embodiment.

[0038] As mentioned above, when the system control unit 212 of the imaging device 200 receives a setting input for changing the focus position in accordance with the photographer's focus ring operation during shooting in manual focus mode, it starts processing from S601 onwards in Figure 6. In S601, the setting unit 101 of the system control unit 212 receives shooting setting input corresponding to the adjustment of the focus position in response to the photographer's operation of the focus ring.

[0039] Next, in the S602 process, the detection unit 102 detects a subject from the image captured by the image sensor 211 after the focus position has been adjusted by the photographer's manual operation. In this embodiment, the subject to be detected is, for example, a person, a dog, a cat, or a car. The detection unit 102 may detect one type of subject, multiple types, or a subject pre-set by the photographer. Furthermore, the detection unit 102 may detect the entire subject, a part of the subject, or both the entire subject and each of its parts. Examples of methods for detecting the entire subject or parts of a subject include detection methods using known machine learning techniques. For example, an object detector capable of detecting specific types of objects or their parts may be created in advance using a technique such as a convolutional neural network (CNN), and the detection unit 102 inputs the captured image from the image sensor 211 into the object detector to detect the subject.

[0040] Figure 7 is a diagram used to explain an example of subject detection processing. Figure 7(a) shows an example of the detection result in which the torso, face, right eye, and left eye of a person 701 were detected from an image 700 captured by the image sensor 211. The detection unit 102 detects a subject frame as a result of subject detection, with subject frame 702 being the frame showing the detection result of the torso, and similarly, subject frame 703 showing the detection result of the face, subject frame 704 showing the left eye, and subject frame 705 showing the detection result of the right eye.

[0041] Figure 7(b) is a table showing the subject information generated from the subject frame detected by the detection unit 102. In the table in Figure 7(b), column 711 shows the type of subject, column 712 shows the part of the subject, column 713 shows the coordinates of the upper left of the subject frame, column 714 shows the coordinates of the lower right of the subject frame, and column 715 shows the confidence level of the detection result. This subject type, part, coordinates of the subject frame, and confidence level of the detection result are information that can be obtained with the aforementioned known machine learning technique for object detectors. In this embodiment, the bounding box surrounding the area of ​​the detected subject is used as the subject frame, and an example is given in which the coordinates of the upper left and lower right points of that bounding box are obtained.

[0042] After the processing in S602 described above, the range estimation unit 103 estimates the defocus range of the subject in the next processing step, S603. In this embodiment, the range estimation unit 103 estimates the defocus range of each subject based on the image captured by the image sensor 211, the subject information detected by the detection unit 102, and a map of the defocus amount of the region encompassing each subject (referred to as the defocus map). In this embodiment, the defocus map is map information that shows a defocus amount distribution in which a defocus amount is assigned to each pixel of the whole or part of the image captured by the image sensor 211.

[0043] Figure 8 shows an example of the estimated defocus range. Figure 8(a) shows the detection results from the image 800 captured by the image sensor 211, showing the detection of a person 801 and a dog 811, and the various parts of the person 801 and dog 811. Subject frame 802 is the frame showing the detection result of the person's torso, and similarly, subject frame 803 shows the detection result of the person's face, subject frame 804 shows the detection result of the person's left eye, and subject frame 805 shows the detection result of the person's right eye. Subject frame 812 shows the detection result of the dog's torso, subject frame 813 shows the dog's face, subject frame 814 shows the dog's left eye, and subject frame 815 shows the detection result of the dog's right eye.

[0044] Figure 8(b) shows the estimated defocus range for the subject shown in Figure 8(a). The length of the line segments indicated by arrows in Figure 8(b) represents the defocus range, which is the range of values ​​for the amount of defocus, similar to the example in Figure 5(b) mentioned earlier. The horizontal axis in Figure 8(b) represents the magnitude of the amount of defocus, with the side closer to the imaging device 200 defined as "near" and the side further away as "far". In Figure 8(b), the estimated defocus range 821 is shown for the right eye of dog 811, and similarly, the estimated defocus ranges 822, 823, and 824 are shown for the left eye, face, and torso of dog 811, respectively. Similarly, in Figure 8(b), the estimated defocus ranges 831, 832, 33, and 834 are shown for the right eye, left eye, face, and torso of person 801, respectively. The ends of each of these line segments represent the nearest and farthest values ​​of the amount of defocus, and a longer line segment indicates a wider defocus range. For example, in both the person 801 and the dog 811, the torso has a wider depth range than the eyes as viewed from the imaging device 200, so the defocus range of the torso is wider than the defocus range of the eyes.

[0045] Here, we will explain in more detail how to estimate the defocus range. The defocus range can be obtained by using a machine learning model that has been trained using training data as input. A specific machine learning algorithm is deep learning, which uses a neural network to generate features and connection weights. This section will explain learning using neural networks.

[0046] The input data used for training includes training images, defocus maps, subject regions, and the correct defocus range. The training process also includes error detection and weight update. In the error detection process, the error between the output data output from the neural network's output layer and the target data is obtained, based on the input data input to the input layer. The target data used here is the defocus range of the correct answer. Alternatively, the error detection process may use a loss function to calculate the error between the output data from the neural network and the target data.

[0047] In the weight update process, the connection weight coefficients between nodes of the neural network are updated based on the error obtained in the error detection process, so as to reduce the error. In this weight update process, for example, backpropagation is used to update the connection weight coefficients. Backpropagation is a method that adjusts the connection weight coefficients between nodes of each neural network so as to reduce the above error. As a result of the learning described above, a machine learning model that estimates the defocus range is obtained. Using this machine learning model, the defocus range can be estimated. In this embodiment, an example of estimating the defocus range by including the subject area as input has been described, but the subject area is not a required input. The subject area may be detected after the defocus range has been estimated.

[0048] After the processing in S603 described above, in the next processing in S604, the selection unit 104 extracts candidate subjects to be highlighted from among the subjects detected by the detection unit 102 based on the defocus range estimated by the range estimation unit 103. In this embodiment, based on the estimated defocus range, the selection unit 104 extracts as candidate subjects to be highlighted a subject (A) that is estimated to be currently in focus and a subject (B) that is estimated to be close in focus from among the subjects detected by the detection unit 102. In other words, in this embodiment, the subjects to be highlighted in the end are the subject (A) that is estimated to be currently in focus and the subject (B) that is estimated to be close in focus.

[0049] Using Figure 9, we will explain the difference between a subject (A) that is estimated to be currently in focus and a subject (B) that is estimated to be close to the point of focus. FIG. 9 is a diagram showing the estimation results of the defocus ranges of the subjects of the dog 811 and the person 801 shown in FIG. 8(a), and is a diagram with annotations added for further explanation with respect to the example of FIG. 8(b).

[0050] Here, the subject (A) that can be estimated to be in focus is a subject whose current focus position is within the estimated defocus range. When the current focus position is expressed in terms of the defocus amount with the product [Fδ] of the F value of the aperture of the imaging optical system and the allowable confusion circle diameter δ described above as the unit, it becomes 0Fδ. Therefore, if the lower limit value of the defocus range is less than or equal to 0Fδ and the upper limit value is also greater than or equal to 0Fδ, it means that the subject within that defocus range is in focus. That is, when the lower limit value of the defocus range is Fmin and the upper limit value is Fmax, when Fmin ≤ 0 and 0 ≤ Fmax are satisfied, it can be regarded as being in focus. In the case of the example of FIG. 9, the defocus ranges 904, 905, 906, 907 represented by each line segment of the arrows in the figure are the defocus ranges of the subject (A) that can be estimated to be in focus. That is, in the case of the subjects illustrated in FIG. 9, the right eye, left eye, face, and torso of the person 801 are the subjects (A) that can be estimated to be in focus.

[0051] On the other hand, the subject (B) that can be estimated to have a focus position close is a subject whose current focus position is not within the estimated defocus range, but the difference between the lower limit value or the upper limit value of the defocus range and the focus position is within the threshold. In other words, the subject (B) that can be estimated to have a focus position close is a subject with a high degree of focus. Here, when the threshold is th, 0 < Fmin or Fmax < 0, and further when |Fmin| ≤ th or |Fmax| ≤ th is satisfied, it can be regarded as having a focus position close. In the case of the example of FIG. 9, the defocus ranges 900, 902, 903 represented by the line segments of the arrows in the figure are the defocus ranges of the subject (B) that can be estimated to have a focus position close. That is, in the case of the subjects illustrated in FIG. 9, the right eye, face, and torso of the dog 811 are the subjects (B) with a focus position close.

[0052] Following the processing in S604 described above, in the next process, S605, the selection unit 104 determines which subject to be highlighted from among the candidate subjects to be highlighted extracted in S604. In this embodiment, by selecting and narrowing down the subjects to be highlighted from among multiple candidate subjects, it is possible to display information that is more important to the photographer in priority and to display it without impairing visibility.

[0053] Referring to Figure 9 mentioned above, we will now explain the process for determining which subject to highlight. Here, since highlighting multiple parts of the same subject is redundant, the selection unit 104 narrows down the target of highlighting to the defocus ranges 904, 905, 906, and 907 of the subject (A) that is estimated to be currently in focus. For example, the selection unit 104 focuses on the subject type 910 and selects the part of the subject with the narrowest defocus range among subjects of the same type as the target of highlighting. However, if the difference is not greater than a predetermined threshold, it is not necessary to limit it to one, and multiple defocus ranges may be selected. In the example in Figure 9, the selection unit 104 selects the defocus ranges 904 and 905, which are the narrowest ranges among the defocus ranges 904 to 907 of person 801, and determines that the right eye and left eye of person 801, which correspond to these ranges, are the subjects to be highlighted.

[0054] Figure 10(a) shows an example where multiple subjects in focus include the torsos and faces of persons 1010, 1020, and 1030. In Figure 10(a), the image 1000 captured by the image sensor 211 contains multiple persons 1010, 1020, and 1030, and the figure shows the subject frames where each part of persons 1010, 1020, and 1030 has been detected. Subject frame 1011 shows the detection result of the torso of person 1010, and subject frame 1012 shows the detection result of the face of person 1010. Similarly, subject frame 1021 shows the detection result of the torso of person 1020, subject frame 1022 shows the detection result of the face of person 1020, subject frame 1031 shows the detection result of the torso of person 1030, and subject frame 1032 shows the detection result of the face of person 1030.

[0055] Similar to the example in Figure 8 mentioned above, Figure 10(b) shows the estimated defocus range of the subject shown in Figure 10(a), and the length of each line segment indicated by the arrow in Figure 10(b) represents the defocus range. Figure 10(b) shows the estimated defocus ranges of the faces and torsos of people 1010, 1020, and 1013, respectively: 1013 and 1014, 1023 and 1024, and 1033 and 1034.

[0056] Here, for example, if a narrower defocus range is selected within the same subject (e.g., a face), the subject frames of faces corresponding to the defocus ranges 1013, 1023, and 1033 will be selected and highlighted. However, if multiple subjects (subject frames of multiple faces) are highlighted, visibility from the photographer's perspective will decrease, so conditions may be set to further narrow down the selection. For example, the selection unit 104 may prioritize selecting subjects that are close to a specific area. In the case of Figure 10, the selection unit 104 may, for example, use the center 1050 of the image 1000 as a specific area and select only the subject frame 1022 of a face that is close to the center 1050. Alternatively, the selection unit 104 may use an area other than the center point of the image as a specific area, for example, an area previously selected by the photographer, and select subject frames based on the distance from that photographer's selected area. Note that the processing in S605 is a process to improve the visibility of subject display in specific situations, such as when there are multiple subject frames of the same subject, so it may be omitted to simplify the processing.

[0057] After the processing in S605 described above, in the next processing step, S606, the output unit 105 highlights the subject selected by the selection unit 104 in the previous processing steps on the live view image. In the example of this embodiment, the output unit 105 highlights the bounding box, such as the subject frame 702 in Figure 7, which was detected by the detection unit 102 in S602, and displays it superimposed on the subject on the live view image. Note that the method of highlighting the subject is not limited to displaying the bounding box; it may also be a method that extracts the contour of the subject using known image processing techniques and highlights that contour.

[0058] Furthermore, the output unit 105 may adjust the degree of emphasis based on the degree of focus of the subject when highlighting the subject frame. For example, the output unit 105 may strongly emphasize the subject frame of a subject that is in focus, or it may strongly emphasize the subject frame of a subject whose defocus range is close to the focus position, or it may weakly emphasize the subject frame of a subject whose defocus range is far from the focus position. In other words, the output unit 105h may adjust the degree of emphasis to be higher the higher the degree of focus of the subject. Other methods of highlighting may include, for example, displaying by gradually changing the line width, line type, line color and density, transparency, brightness, etc.

[0059] Figure 11 shows an example of highlighting for the subject examples of person 801 and dog 811 exemplified in Figure 8(a). Figure 11(a) shows an example where the subject frame is highlighted. Figure 11(b) shows the estimated defocus range for each subject exemplified in Figure 11(a). Subject frames 1104 and 1105 in Figure 11(a) are frames corresponding to the in-focus defocus ranges 1125 and 1124, respectively, and specifically represent the left and right eyes of person 801. Subject frames 1104 and 1105 show examples where the subject frame is highlighted by drawing it with the thickest possible solid line.

[0060] In addition, even if the focus is not exact, an emphasis display may be performed in which the subject frames 1112, 1113, and 1114 of subjects that are not exactly in focus but are close to the focus position to some extent are given different levels of emphasis. The defocus ranges corresponding to these subject frames 1112, 1113, and 1114 are the defocus ranges 1123, 1122, and 1120, respectively. Here, in these defocus ranges 1123, 1122, and 1120 corresponding to the subject frames 1112, 1113, and 1114, the distances (differences) from the focus position are the distances x1, x2, and x3, as shown in Fig. 11(b). Also, the magnitude relationship of these distances x1, x2, and x3 is x1 < x2 < x3. The output unit 105 may provide different levels of emphasis by increasing the line width in the order of the distances x1, x2, and x3 between the defocus ranges 1123, 1122, and 1120 corresponding to the subject frames 1112, 1113, and 1114 and the focus position. Thus, by changing the level of emphasis stepwise and linearly based on the distance between the focus position and the defocus range, it becomes easier for the photographer to intuitively understand the degree of focus adjustment.

[0061] As an example of the stepwise change in the thickness of the subject frame according to the distance between the focus position and the defocus range, an example is given in which the thickness of the subject frame linearly changes from the maximum thickness wmax with the highest degree of emphasis to the minimum thickness wmin with the lowest degree of emphasis. In this example, the thickness w of the subject frame corresponding to the defocus range with a distance x from the focus position can be obtained by the following formula (1). However, in formula (1), the threshold value when the distance x between the focus position and the defocus range is considered to be close is denoted as d, and the relationship between the threshold value d and the distance x is 0 < x < d.

[0062] w = wmax - (x / d) × (wmax - wmin) Formula (1)

[0063] In the example above, we showed how to change the line thickness, but it is also possible to express the degree of emphasis by changing the line type, for example. However, since the number of usable line types is somewhat limited, changing the line type may result in fewer levels of emphasis variation than changing the line thickness. In the example of changing the line type, for example, you could define the number of levels of emphasis variation by dividing it into sections, such as using a solid line when 0 < (x / d) ≤ 0.2 and a long dashed line when 0.2 < (x / d) ≤ 0.4.

[0064] After the processing in S606 described above, the system control unit 212 determines whether to continue shooting in the next processing step, S607. The system control unit 212 determines to continue shooting unless the photographer gives an instruction to end shooting, and repeatedly executes the processes from S601 to S605. When the photographer gives an instruction to end shooting, the system control unit 212 terminates the process shown in the flowchart in Figure 6.

[0065] According to the first embodiment described above, it is possible to clearly show the photographer which subject is in focus. Furthermore, in this embodiment, by also showing the photographer subjects that are slightly out of focus, it is possible to clearly show the photographer which subject to focus on when adjusting the focus manually. In other words, the imaging device of this embodiment improves the convenience of the photographer when adjusting the focus manually in manual focus mode.

[0066] <Modified form of the first embodiment> In the first embodiment described above, an example was explained in which the degree of emphasis is changed according to the focus position and degree of focus. However, the focus position and degree of focus may also be presented by a display or output other than the subject frame. In a modification of the first embodiment, an example is described in which information representing the focus position and degree of focus is presented to the photographer by a display or output other than the subject frame, focusing on the differences from the first embodiment.

[0067] Figure 12 is a flowchart showing the processing flow related to this modified example, in which the imaging device 200 presents information indicating the focus position and degree of focus, corresponding to manual adjustment of the focus position, to the photographer through displays and outputs other than the subject frame. The configuration of each functional unit that performs the shooting assistance processing in Figure 12 is the same as the block diagram shown in Figure 1, so its description and explanation are omitted. Each functional unit in Figure 1 related to this modified example is realized by the system control unit 212 in Figure 2 executing the processing according to the flowchart in Figure 12. Note that in the flowchart of Figure 12, S601 to S605 and S607 are the same as the processing in the flowchart of Figure 6 described above, so their explanations are omitted. In the flowchart of Figure 12 related to this modified example, after the processing in S605, the output unit 105 performs the processing in S1206.

[0068] Figure 13 is a diagram used to explain the photographic support process related to this modified example. Figure 13(a) shows an example in which an image 1300 captured by the image sensor 211 contains multiple subjects (people 1301 and 1321, and a dog 1311), and subject frames indicating the detection of each part of the multiple subjects are displayed. In the example in Figure 13(a), the subject frame 1302 of person 1301's face, the subject frame 1322 of person 1321's face, and the subject frame 1312 of dog 1311's face are detected.

[0069] Figure 13(b) shows the estimated defocus range for each subject shown in Figure 13(a), similar to Figures 9 and 10(b) mentioned above. The length of each arrow segment in Figure 13(b) represents the defocus range. Figure 13(b) shows the estimated defocus ranges 1331, 1332, and 1333 for the faces of dog 1311, person 1301, and person 1321, respectively. In the example in Figure 13(b), defocus range 1332 includes the in-focus position, indicating that the face of person 1301 is in focus. On the other hand, for example, defocus range 1331 indicates that the corresponding subject is in front of the in-focus position, and for example, defocus range 1333 indicates that the corresponding subject is behind the in-focus position.

[0070] In the flowchart of Figure 12, after processing in S605, when the process proceeds to S1206, the output unit 105 superimposes the subject frames 1302, 1312, and 1322, which were determined to be highlighted up to S605, onto the live view image. However, in this modified example, unlike the example of the first embodiment described above, no difference in the degree of highlighting is applied to the subject frames 1302, 1312, and 1322.

[0071] In this modified example, the output unit 105, as the next process in S1207, superimposes the images of the focus guides 1341, 1342, and 1343 shown in Figure 13(a) onto the live view image as information indicating the focus position and degree of focus. In the example in Figure 13(a), focus guide 1341 represents the focus position and degree of focus in the area indicated by the subject frame 1302 of the face of person 1301. Similarly, focus guide 1342 represents the focus position and degree of focus in the area indicated by the subject frame 1312 of the face of dog 1311, and focus guide 1343 represents the focus position and degree of focus in the area indicated by the subject frame 1322 of the face of person 1321.

[0072] Figure 14 is a diagram used to explain the focus guide 1400. Figures 14(a) to 14(e) show how the focus position and the difference in focus positions are represented on the focus guide 1400 for the same defocus range. In the focus guide 1400 shown in Figures 14(a) to 14(e), the band-shaped indicator 1402 indicates the defocus range. The output unit 105 determines the length of the band-shaped indicator 1402 on the circle 1403 of the focus guide 1400 (the length of the arc on the circle 1403) according to the magnitude of the defocus amount. In other words, the magnitude of the defocus amount is expressed by the arc length of the band-shaped indicator 1402 on the circle 1403 of the focus guide 1400.

[0073] In the focus guide 1400 shown in Figures 14(a) to 14(e), the triangular indicators 1401 indicate the focus position, and two of them are displayed symmetrically to improve visibility for the photographer. The output unit 105 determines the distance between the two triangular indicators 1401 on the circle 1403 of the focus guide 1400 (the length of the arc on the circle 1403) according to the magnitude of the defocus amount. That is, the magnitude of the defocus amount is expressed by the length of the arc between the two triangular indicators 1401 on the circle 1403 of the focus guide 1400. Furthermore, the output unit 105 changes the positions of the two triangular indicators 1401 along the circle 1403 of the focus guide 1400 in accordance with the photographer's manual adjustment of the focus position. For example, when the photographer changes the focus position by operating the focus ring, the output unit 105 changes the positions of the two triangular indicators 1401 along the circle 1403 in a direction that moves them closer together or further apart from each other. Furthermore, if the focus position is located beyond the center of the defocus range, the output unit 105 places two triangular indicators 1401 outside the circle 1403. In other words, the presence of two triangular indicators 1401 outside the circle 1403 indicates that the focus position is located beyond the center of the defocus range.

[0074] Figure 14(a) shows an example of the focus guide 1400 when the focus position is beyond the defocus range represented by the band-shaped indicator 1402. If the focus position is outside the defocus range, the output unit 105 displays the two triangular indicators 1401, for example, in white.

[0075] Figure 14(b) shows an example of the focus guide 1400 where the focus position has moved closer to the subject than in the example of the focus guide 1400 in Figure 14(a), approaching the defocus range, and the focus position is now within the defocus range. In this case, when the focus position is within the defocus range, the output unit 105 changes the color of the triangular indicator 1401, for example, from white to green, to indicate to the photographer that the subject is in focus.

[0076] Figure 14(c) shows an example of the focus guide 1400 when the focus position has moved even further towards the near side than in Figure 14(b), and the focus position is approximately in the center of the defocus range represented by the band-shaped indicator 1402. When the focus position is approximately in the center of the defocus range, the output unit 105 indicates to the photographer that the focus position is in the center of the defocus range by displaying two triangular indicators 1401 placed both outside and inside the circle 1403.

[0077] Figure 14(d) shows an example of the focus guide 1400 when the focus position has moved further towards the near side than in Figure 14(c), and the focus position is on the near side of the center of the defocus range represented by the band-shaped indicator 1402. However, in the example of Figure 14(d), the focus position is still within the defocus range. When the focus position is on the near side of the center of the defocus range and within the defocus range, the output unit 105 displays the two triangular indicators 1401 positioned inside the circle 1403.

[0078] Figure 14(e) shows an example of the focus guide 1400 when the focus position has moved even further towards the near side than in Figure 14(d), and the focus position has moved outside the defocus range. When the focus position has moved closer than the center of the defocus range and moved outside the defocus range, the output unit 105 changes the color of the two triangular indicators 1401 from green to white to indicate to the photographer that the focus is not in focus. In addition, since the two triangular indicators 1401 are positioned inside the circle 1403 at this time, the photographer can also know that the focus position is closer than the center of the defocus range and outside the defocus range.

[0079] In Figure 14, for the sake of simplicity, the defocus range represented by the band-shaped indicator 1402 is shown as constant and unchanging. However, in reality, the range estimation unit 103 constantly estimates the defocus range, and as a result, the defocus range is always changing. Furthermore, changes in the defocus range occur due to changes in the photographer's focus position, aperture, the position of the subject or imaging device, and the shape of the subject. For this reason, the output unit 105 actually adjusts the arc length of the band-shaped indicator 1402 in accordance with the changes in the defocus range. In this case, the output unit 105 changes the arc length of the band-shaped indicator 1402 representing the defocus range, without changing the position of the triangular indicator 1401 indicating the focus position. In this modified example, the magnitude of the defocus amount is expressed by the arc length of the band-shaped indicator 1402 on the circle 1403. Therefore, the length of the band-shaped indicator is determined by the ratio of the defocus amount at the boundary between the focus position and the defocus range, with the center of the focus position and the defocus range as the reference point. Furthermore, the position of the indicator representing the focus position remains unchanged if the photographer is not moving the focus ring, and changes according to the amount of rotation if the photographer moves the focus ring. In this way, the focus guide of this modified version provides consistency in the position of the indicator representing the focus position, allowing the photographer to intuitively understand that the position of the focus ring is the same as the position of the indicator representing the focus position.

[0080] In the example shown in Figure 13(a) above, a focus guide is displayed near all subject frames (for example, at the bottom), but this is not the only example. For example, a single common focus guide may be displayed for multiple different parts included in the whole of a single subject, and multiple defocus ranges corresponding to each of those parts may be displayed within that single focus guide. This prevents situations where, for example, multiple focus guides are displayed near the subject frame, reducing the photographer's visibility.

[0081] Figure 15 shows an example of displaying multiple defocus areas on a single focus guide. In Figure 15, for example, the band-shaped indicator 1501 represents the defocus area of ​​a person's torso, and the band-shaped indicator 1502 represents the defocus area of ​​the same person (for example, the face). In this case, the output unit 105 displays the band-shaped indicators 1501 and 1502, which represent these multiple defocus areas, in a way that makes them easily distinguishable, for example, by using different colors, transparency, or brightness.

[0082] Furthermore, the output unit 105 may adjust the degree of emphasis on the focus guide based on the distance between the defocus range and the in-focus position. By adjusting the degree of emphasis on the focus guide, it becomes possible to display a more prominent defocus range that is of greater interest to the photographer. For example, if there are subjects with narrow defocus ranges and subjects with wide defocus ranges, the subject with the narrower defocus range is more likely to be the subject that the photographer wants to focus on. For this reason, the output unit 105 may provide emphasis on the defocus range of the subject with the narrower defocus range in the focus guide.

[0083] Furthermore, the output unit 105 may adjust the length of the band-shaped indicator representing the defocus range to be displayed as needed, depending on the movement of the subject, whether the focus point is closer to or further away from the center of the defocus range, etc. Furthermore, as a measure to improve visibility, the output unit 105 may group defocus ranges of similar size into a single band-shaped indicator. For example, the defocus ranges of the right and left eyes of a person facing forward may be displayed together as a single band-shaped indicator.

[0084] After the processing in S1207 described above, in the next processing, S1208, the output unit 105 provides tactile feedback to the photographer that the in-focus position has moved beyond the boundary of the defocus range. For example, when the photographer rotates the focus ring to change the in-focus position and the in-focus position enters the defocus range, the output unit 105 slightly vibrates the focus ring to inform the photographer that the in-focus position has entered the defocus range. In this way, by using a method other than the highlighting method described in the first embodiment above to inform the photographer that the in-focus position has entered the defocus range, it becomes easier for the photographer to grasp the degree of focus even when a lot of information is displayed on the screen. The tactile feedback to the photographer may be provided by a change in the rotational resistance of the focus ring, the output of sound, or vibration of a connected device worn by the photographer. After processing in S1208, the system control unit 212 proceeds to processing in S607 as described above.

[0085] In this modified example, we have shown how the focus guide is communicated to the photographer through a combination of on-screen display and tactile feedback. However, other methods, such as using only tactile feedback, may also be used to communicate with the photographer. By suppressing the display of information on the screen in this way, the photographer can take pictures without impairing the visibility of the subject. As explained above, according to the modified version of the first embodiment, it is possible to more intuitively communicate to the photographer which subject is currently in focus.

[0086] <Second Embodiment> Next, as a second embodiment, we will describe an example in which autofocus is initiated based on the defocus range during focusing, focusing on the differences from the first embodiment. Figure 16 is a block diagram showing an example of the configuration of each functional unit in the imaging device of the second embodiment. As shown in Figure 16, the imaging device of the second embodiment has the same setting unit 101, detection unit 102, selection unit 104, range estimation unit 103, and output unit 105 as described in the first embodiment, and in addition to these, it also has an AF setting unit 1606. Figure 17 is a flowchart showing the processing flow in the shooting assistance device of the second embodiment. Each functional unit in Figure 16 is realized by the system control unit 212 of the hardware configuration in Figure 2 executing processing according to the flowchart in Figure 17. In the flowchart of Figure 17, S601 to S606 and S607 are the same processing as described in Figure 6, so their explanation is omitted. In the flowchart of Figure 17, after the processing of S606, the AF setting unit 1606 performs processing from S1707 onwards.

[0087] As part of the S1707 process, the AF setting unit 1606 determines whether the autofocus function has been enabled by the photographer. That is, the AF setting unit 1606 determines whether the photographer has changed from the manual focus mode to autofocus mode. If the autofocus function has been enabled by the photographer, the AF setting unit 1606 then executes the process in S1708. On the other hand, if the autofocus function has not been enabled by the photographer, the AF setting unit 1606 skips the process in S1708 and proceeds to S607. Note that the photographer's activation of the autofocus function may be done by, for example, a dedicated button on the imaging device 200 or by input using the touch panel 218, and is not limited to these methods.

[0088] Proceeding to S1708, the AF setting unit 1606 sets the subject for which focus tracking will be performed by the autofocus function. Known technologies can be used for the autofocus method, and their explanation is omitted here. The subject for focus tracking is the subject among those determined to be highlighted in S605 that has the smallest distance between the corresponding defocus range and the in-focus position. However, if there are multiple such subjects, the AF setting unit 1606 selects one from among them. In this embodiment, for example, the AF setting unit 1606 randomly selects one subject from among multiple subjects. In addition, the AF setting unit 1606 may, for example, take into account the autofocus method and prioritize selecting a subject that is easier to focus on based on the size of the subject frame, the variation in the amount of defocus within the subject area, brightness, etc.

[0089] For example, in the case shown in Figure 11(a) above, the defocus ranges with the smallest distance from the focus position are defocus ranges 1124 and 1125, which correspond to subject frames 1105 and 1104, respectively, which are the eyes of the person. Therefore, the AF setting unit 1606 sets one of these subject frames, for example subject frame 1105, as the target for autofocus tracking. The AF setting unit 1606 then starts autofocus tracking of the subject. After processing in S1708, the system control unit 212 proceeds to processing in S607 above. According to the second embodiment described above, the photographer can select a subject that they have intuitively focused on manually as the subject to be tracked by autofocus.

[0090] The present invention can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of concrete implementations of the present invention, and the technical scope of the invention should not be limited by them. That is, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0091] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Composition 1) A detection means for detecting a subject from an image captured by imaging in accordance with at least the input of shooting settings related to focusing, Estimation means for estimating a defocus range that represents the range of values ​​for the amount of defocus on the subject, A selection means that selects at least one subject from among the subjects detected by the detection means based on the subject detection result and the defocus range estimation result, An output means for outputting information regarding the focus on the selected subject, A photographic assistance device characterized by having the following features. (Configuration 2) The aforementioned shooting setting input is an input that changes the focus position through manual operation by the user. The shooting assistance device according to configuration 1, characterized in that the selection means selects the subject based on the subject detection result, the defocus range estimation result, and the focus position. (Composition 3) The estimation means estimates the defocus range for each of the multiple parts based on the amount of defocus for each of the multiple parts included in a single subject. The selection means selects from among the plurality of parts a part in which the estimated defocus range is narrower. The photographic assistance device according to configuration 1 or 2, characterized in that the output means outputs information regarding the focus on the selected part. (Composition 4) The shooting assistance device according to any one of configurations 1 to 3, characterized in that the selection means selects the subject based on the subject detection result and the defocus range estimation result, as well as the position of the subject in the captured image. (Composition 5) The shooting assistance device according to any one of configurations 1 to 4, characterized in that the selection means selects the subject based on the position of the subject in a specific region of the captured image. (Composition 6) The shooting assistance device according to configuration 5, characterized in that the selection means selects subjects by prioritizing subjects whose position is close to the specific region. (Composition 7) The shooting assistance device according to any one of configurations 1 to 6, characterized in that the output means highlights the subject selected by the selection means as output information regarding focus. (Composition 8) The shooting assistance device according to configuration 7, characterized in that the output means changes the highlighting of the subject selected by the selection means in stages based on the degree of focus on the subject. (Composition 9) The shooting assistance device according to configuration 8, characterized in that the output means increases the degree of emphasis of the highlighting display as the degree of focus on the subject increases. (Composition 10) The shooting assistance device according to any one of configurations 1 to 6, characterized in that the output means displays a first index representing the focus position for the selected subject and a second index representing the estimated result of the defocus range near the selected subject as output information regarding the focus. (Composition 11) The shooting assistance device according to configuration 10, characterized in that the output means displays an image in which the first indicator and the second indicator are arranged in line with the arc of a circle. (Composition 12) The shooting assistance device according to configuration 11, characterized in that the output means changes the position of a first indicator with respect to the arc of the circle in accordance with the change in the focus position, and changes the length of a second indicator with respect to the arc of the circle in accordance with the change in the defocus range. (Composition 13) The shooting assistance device according to configuration 10, characterized in that the output means highlights the first indicator in accordance with the change in the focus position and highlights the second indicator in accordance with the change in the defocus range. (Composition 14) The shooting assistance device according to any one of configurations 10 to 13, characterized in that the output means displays one first indicator and one second indicator for multiple parts of a single subject. (Composition 15) The shooting assistance device according to any one of configurations 1 to 6, characterized in that the output means outputs the information regarding focus as a tactile sensation to the user. (Composition 16) The shooting assistance device according to configuration 15, characterized in that the output means outputs information regarding the focus by touch based on the estimated focus position and defocus range for the selected subject. (Composition 17) The shooting assistance device according to configuration 16, characterized in that the output means provides tactile output when the focus position enters the defocus range. (Composition 18) The shooting assistance device according to any one of configurations 1 to 17, further comprising setting means for setting the autofocus function to focus on the selected subject. (Method 1) A detection step involves detecting a subject from an image captured by imaging in accordance with at least the input of shooting settings related to focusing, An estimation step for estimating the defocus range, which represents the range of values ​​for the amount of defocus on the subject, A selection step in which at least one subject is selected from among the subjects detected in the detection step, based on the subject detection result and the defocus range estimation result. An output step which outputs information regarding the focus on the selected subject, A method for assisting photography, characterized by having the following features. (Program 1) A program that causes a computer to function as a shooting assistance device described in any one of configurations 1 to 18. [Explanation of symbols]

[0092] 101: Setting unit, 102: Detection unit, 103: Range estimation unit, 104: Selection unit, 105: Output unit

Claims

1. A detection means for detecting a subject from an image captured by imaging in accordance with at least the input of shooting settings related to focusing, Estimation means for estimating a defocus range that represents the range of values ​​for the amount of defocus on the subject, A selection means that selects at least one subject from among the subjects detected by the detection means based on the subject detection result and the defocus range estimation result, An output means for outputting information regarding the focus on the selected subject, A photographic assistance device characterized by having the following features.

2. The aforementioned shooting setting input is an input that changes the focus position through manual operation by the user. The shooting assistance device according to claim 1, characterized in that the selection means selects the subject based on the subject detection result, the defocus range estimation result, and the focus position.

3. The estimation means estimates the defocus range for each of the multiple parts based on the amount of defocus for each of the multiple parts included in a single subject. The selection means selects from among the plurality of parts a part in which the estimated defocus range is narrower. The photographic assistance device according to claim 1, characterized in that the output means outputs information regarding the focus on the selected part.

4. The shooting assistance device according to claim 1, characterized in that the selection means selects the subject based on the subject detection result and the defocus range estimation result, as well as the position of the subject in the captured image.

5. The shooting assistance device according to claim 1, characterized in that the selection means selects the subject based on the position of the subject in a specific region of the captured image.

6. The shooting assistance device according to claim 5, characterized in that the selection means selects subjects by prioritizing subjects whose position is close to the specific region.

7. The shooting assistance device according to claim 1, characterized in that the output means highlights the subject selected by the selection means as output information regarding focus.

8. The shooting assistance device according to claim 7, characterized in that the output means changes the highlighting of the subject selected by the selection means in stages based on the degree of focus on the subject.

9. The shooting assistance device according to claim 8, characterized in that the output means increases the degree of emphasis of the highlighting display as the degree of focus on the subject increases.

10. The shooting assistance device according to claim 1, characterized in that the output means displays a first index representing the focus position for the selected subject and a second index representing the estimated result of the defocus range near the selected subject as output information regarding the focus.

11. The shooting assistance device according to claim 10, characterized in that the output means displays an image in which the first indicator and the second indicator are arranged in line with the arc of a circle.

12. The shooting assistance device according to claim 11, characterized in that the output means changes the position of a first indicator with respect to the arc of the circle in accordance with the change in the focus position, and changes the length of a second indicator with respect to the arc of the circle in accordance with the change in the defocus range.

13. The shooting assistance device according to claim 10, characterized in that the output means highlights the first indicator in accordance with the change in the focus position and highlights the second indicator in accordance with the change in the defocus range.

14. The shooting assistance device according to claim 10, characterized in that the output means displays one first indicator and one second indicator for multiple parts of a single subject.

15. The shooting assistance device according to claim 1, characterized in that the output means outputs the information regarding focus as a tactile sensation to the user.

16. The shooting assistance device according to claim 15, characterized in that the output means outputs information regarding the focus by touch based on the estimated focus position and defocus range for the selected subject.

17. The shooting assistance device according to claim 16, characterized in that the output means provides tactile output when the focus position enters the defocus range.

18. The shooting assistance device according to claim 1, further comprising setting means for setting the autofocus function to focus on the selected subject.

19. A detection step involves detecting a subject from an image captured by imaging in accordance with at least the input of shooting settings related to focusing, An estimation step for estimating the defocus range, which represents the range of values ​​for the amount of defocus on the subject, A selection step in which at least one subject is selected from among the subjects detected in the detection step, based on the subject detection result and the defocus range estimation result. An output step which outputs information regarding the focus on the selected subject, A method for assisting photography, characterized by having the following features.

20. Computers A detection means for detecting a subject from an image captured by imaging in accordance with at least the input of shooting settings related to focusing, Estimation means for estimating a defocus range that represents the range of values ​​for the amount of defocus on the subject, A selection means that selects at least one subject from among the subjects detected by the detection means based on the subject detection result and the defocus range estimation result, An output means for outputting information regarding the focus on the selected subject, A program that enables the device to function as a shooting assistance device.

Images