Imaging apparatus, program, and imaging method

Abstract

To provide an imaging apparatus that can automatically apply rating to a photographed image during photographing.SOLUTION: An imaging apparatus has: imaging means; information acquisition means that acquires photographing information during photographing carried out by the imaging means; defocus range estimation means that estimates a defocus range relative to one or more parts of a subject on the basis of the photographing information during the photographing; importance calculation means that calculates the importance of a photographed image for which a defocus range is estimated; and storage means that stores the importance and the photographed image in linkage with each other.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to image processing. [Background technology]

[0002] Conventionally, many digital cameras have been equipped with image display devices such as liquid crystal displays, which allow preview display or playback display of image data stored on a recording medium. Some digital cameras also have a means for assigning a priority to the displayed image after capture. Photographers using such digital cameras are able to store many images on a recording medium, and so they take a large number of images and assign a priority to each image.

[0003] Patent Document 1 discloses a technique in which the degree of focus of a captured image is used as a feature amount of the image after shooting, and importance is assigned to the image based on the feature amount. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-86521 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method described in Patent Document 1 determines whether the entire image is in focus, and is unable to quantify the degree of focus of a local region intended by the user. In this case, it is not possible to assign the user's intended importance to the captured image at a fine level of granularity. Furthermore, methods that assign importance after shooting involve a time lag from the time of shooting, and may not be applicable to scenes where the captured image is to be used immediately.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an imaging device that can automatically assign a rating to a captured image when the image is captured. [Means for solving the problem]

[0007] In order to achieve the above object, an imaging device according to one aspect of the present invention comprises an imaging means, an information acquisition means for acquiring imaging information at the time of imaging by the imaging means, a defocus range estimation means for estimating a defocus range for one or more parts of a subject based on the imaging information at the time of imaging, an importance calculation means for calculating an importance of the captured image for which the defocus range has been estimated based on the imaging information and the defocus range at the time of imaging, and a storage means for linking the importance to the captured image and storing them. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging device that can automatically assign a rating to a captured image when capturing the image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating a hardware configuration of an imaging apparatus according to an embodiment. [Figure 2] 1 is a diagram of an imaging optical system for explaining a defocus amount in an embodiment. [Figure 3] FIG. 1 is a diagram illustrating the configuration of an imaging device according to an embodiment. [Figure 4] FIG. 2 is a configuration diagram of an information acquisition unit in the embodiment. [Figure 5] FIG. 2 is a configuration diagram of a defocus range inference unit in the first and third embodiments. [Figure 6] FIG. 2 is a diagram for explaining the defocus range in the first to fourth embodiments. [Figure 7] 1 is a flowchart of processing executed by the imaging device according to the first and second embodiments. [Figure 8] 10 is an example of a defocus range estimation value in the first and second embodiments. [Figure 9] 11 is a flowchart of processing executed by an imaging device according to a third embodiment. [Figure 10]13 shows an example of an object detection result and a portion input by a user in the third embodiment. [Figure 11] 10 is a flowchart of processing executed by an imaging device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numeral, and duplicated explanations will be omitted.

[0011] <Embodiment 1> Prior to describing the embodiments of the present invention, the hardware configuration in which the imaging device 10 shown in each embodiment is implemented will be described with reference to Fig. 1. In this embodiment, a case will be described in which an interchangeable lens digital camera captures images in focus on multiple subjects, taking into consideration the depth direction spread of the subjects.

[0012] A preferred embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a hardware configuration diagram illustrating the main parts of an imaging device (digital camera) 10. The configuration of the imaging device 10 of the present invention will be described below with reference to Fig. 1. The imaging device 10 is, for example, a digital camera with interchangeable lenses, and is composed of a camera body 100 and a lens unit 200 that guides incident light to an imaging element 101.

[0013] First, we will explain the camera body 100. The image sensor 101 is made up of a CMOS imaging sensor and converts optical signals, which are optical images, into electrical signals. Light rays incident on the photographic lens 201 pass through the aperture 202 and shutter 103 and are focused as an optical image on the image sensor 101.

[0014] The system control unit 102 is configured with at least one computer incorporating a CPU or the like, and controls the entire camera body 100. The system control unit 102 further includes an image processing unit (not shown) for processing a video signal obtained by the image sensor 101. The system control unit 102 also includes a phase-difference AF unit that performs focus detection processing using a phase-difference detection method based on focus detection image data (a signal for phase-difference AF) obtained from the image sensor 101 and the image processing unit. More specifically, the image processing unit generates a pair of image data formed by light beams passing through a pair of pupil regions of the imaging optical system as focus detection image data. The phase-difference AF unit (not shown) detects the amount of focus deviation based on the amount of deviation between the pair of image data. As described above, the phase-difference AF unit of this embodiment performs phase-difference AF (image-surface phase-difference AF) based on the output of the image sensor 101 without using a dedicated AF sensor. The system control unit 102 may be configured and function as an image processing device (information processing device). In that case, the image sensor 10 will have an image processing device (information processing device) built in. The image sensor 10 may also function as an image processing device.

[0015] The memory 104 stores programs, variables, constants, etc. for the operation of the system control unit 102. The memory 104 also includes electrically erasable and rewritable non-volatile memory, which stores various parameters, settings such as ISO sensitivity, shooting modes, various correction data, etc. The power switch 105 switches the power on / off modes of the camera body 100. The mode switching unit 106 is a switch for switching between various shooting modes such as live view shooting and video shooting.

[0016] The rear monitor (display unit) 107 is composed of a liquid crystal device, LEDs, etc. that display operational status such as text, images, and audio, as well as shooting information such as messages, in accordance with the execution of a program in the system control unit 102. The touch panel 108 is arranged in an area roughly equivalent to the rear monitor 107, and detects contact with, for example, a finger or a pen, notifies the system control unit 102 of the contact position on the rear monitor 107, and executes an operation or function associated with the contact position.

[0017] The viewfinder display unit 109, like the rear monitor 107, is a display unit that displays shooting information in response to program execution by the system control unit 102, and together with an eyepiece 110, forms an electronic viewfinder (EVF). Reference numeral 111 denotes an eyepiece detection unit, and the system control unit 102 selectively displays the above-mentioned shooting information on the rear monitor 107 or the viewfinder display unit 109 depending on the photographer's eyepiece position. The shutter control unit 112 controls the operation of the shutter 103 based on the photometric results of the subject calculated by the system control unit 102. The shutter 103 can be controlled in conjunction with the aperture 202.

[0018] Next, the configuration of the lens unit 200 will be described. The camera body 100 and the lens unit 200 are mechanically and electrically connected via a lens mount mechanism (mount section) 113. Furthermore, the camera body 100 and the lens unit 200 are detachable via the lens mount mechanism 113. The lens unit 200 is made up of a photographing lens 201, an aperture 202, a lens drive circuit 203, an aperture control circuit 204, and a lens control section 205. Note that for the sake of simplicity, only one photographing lens 201 is shown in FIG. 1, but in reality it is made up of a group of many photographing lenses.

[0019] The lens control unit 205 is configured with at least one computer having a CPU, memory, etc., and controls the entire lens unit 200. The lens control unit 205 has a memory (not shown) that stores, for example, various constants, variables, programs, etc. for lens operation. It also has a non-volatile memory (not shown) that stores information specific to the lens unit, such as maximum and minimum aperture values ​​and focal length.

[0020] The system control unit 102 of the camera body 100 calculates the defocus amount using the output information of the image sensor 101. Then, the system control unit 102 communicates via the lens control unit 205 of the lens unit 200 based on the calculated defocus amount, and controls the lens drive circuit 203 to adjust the focus.

[0021] The above-mentioned defocus amount will now be described with reference to Fig. 2. Fig. 2 is a diagram illustrating the relationship between the defocus amount of the imaging optical system and the phase difference (image shift amount) between the first focus detection signal and the second focus detection signal acquired from the imaging element.

[0022] An image sensor (not shown) is disposed on the imaging plane 300 in FIG. 2, and the exit pupil of the imaging optical system is divided into a first pupil region 311 and a second pupil region 312. The defocus amount d is defined such that |d| is the distance (size) from the imaging position C of the light beams from the subject 321 and the subject 322 to the imaging plane 300, and a front-focus state in which the imaging position C is located on the subject side of the imaging plane 300 is represented by a negative sign (d<0). Furthermore, a back-focus state in which the imaging position C is located on the opposite side of the subject from the imaging plane 300 is represented by a positive sign (d>0). In the in-focus state in which the imaging position C is located on the imaging plane 300, d=0. The imaging optical system is in-focus (d=0) with respect to the subject 321, and in a front-focus state (d<0) with respect to the subject 322. The front-focus state (d<0) and the back-focus state (d>0) are collectively referred to as a defocus state (|d|>0).

[0023] In a front-focus state (d<0), the light beam from the subject 322 that passes through the first pupil region 311 (second pupil region 312) is first focused and then spreads to a width Γ1 (Γ2) centered at the center of gravity G1 (G2) of the light beam, forming a blurred image on the imaging surface 300. This blurred image is received by each first focus detection pixel (each second focus detection pixel) on the image sensor, and a first focus detection signal (second focus detection signal) is generated. In other words, the first focus detection signal (second focus detection signal) is a signal that represents an image of the subject 322 at the center of gravity G1 (G2) of the light beam on the imaging surface 300, where the subject 322 is blurred by the width Γ1 (Γ2).

[0024] The width Γ1 (Γ2), which is the width of blur of the subject image, increases roughly in proportion to the increase in the magnitude |d| of the defocus amount d. Similarly, the magnitude |p| of the image shift amount p (= the difference G1-G2 in the center of gravity positions of the light beams) between the first focus detection signal and the second focus detection signal also increases roughly in proportion to the increase in the magnitude |d| of the defocus amount d. In the back-focus state (d>0), the direction of the image shift between the first focus detection signal and the second focus detection signal is opposite to that in the front-focus state, but the same is true.

[0025] Thus, the amount of image shift between the first and second focus detection signals increases as the defocus amount increases. In this embodiment, image-surface phase-difference detection focus detection is performed, which calculates the defocus amount from the image shift between the first and second focus detection signals obtained using the image sensor 101. Therefore, the phase-difference AF unit of the system control unit 102 converts the image shift amount into a detected defocus amount as the defocus amount of the image-capturing signal increases. Specifically, because the image shift amount between the first and second focus detection signals increases, the image shift amount is converted into a detected defocus amount using a conversion coefficient calculated based on the base length. Note that the unit of defocus amount in this embodiment is the product [Fδ] of the aperture F-number and the permissible circle of confusion diameter δ in the optical system of the image capture device when capturing an image.

[0026] In this embodiment, a method for assigning a level of importance to a captured image based on the "part of the body to be used as a reference for the degree of focus when assigning a level of importance to a captured image" selected by the user during shooting will be described. In this embodiment, an example will be described in which the user selects a person's right eye mode (i.e., the user wants to assign a level of importance based on how focused the right eye is). However, the above mode is merely an example and does not limit the present invention. For example, categories can also include people, animals, vehicles, etc. Furthermore, there are other parts of the human body besides the right eye, such as the left eye, face, torso, feet, ankles, hands, and wrists, and modes that combine these parts (combination modes) may be selected.

[0027] 3 is a block diagram showing an example of functions of the imaging device 10 according to embodiment 1. The imaging device 10 according to this embodiment has, as its functional units, a photographing unit 400, an information acquisition unit 401, a defocus range inference unit 402, an importance calculation unit 403, and a storage unit 404. The operation (processing) of each of these functional units is controlled by the system control unit 102.

[0028] The image capturing unit 400 captures still images and videos. When capturing an image, the image capturing unit 400 receives input of a predetermined mode from a live view screen or a dial (not shown) that the image capturing apparatus 10 has.

[0029] The information acquisition unit 401 acquires information (imaging information) at the time of imaging by the imaging unit 400. Fig. 4 is a configuration diagram of the information acquisition unit in the first and third embodiments. Fig. 4(A) is a configuration block diagram of the information acquisition unit 401 in the first and second embodiments. Fig. 4(B) is a configuration block diagram of the information acquisition unit 401 in the third and fourth embodiments. The functional units in Fig. 4(B) will be described later.

[0030] The information acquisition unit 401 of the first embodiment has, as functional units, a captured image acquisition unit 500, an AF point acquisition unit 501, and a mode information acquisition unit 502. The captured image acquisition unit 500 acquires images and videos captured by the imaging unit 400. In the case of videos, the videos are divided into frames and acquired, and processed one by one in the same way as images. In this embodiment, it is assumed that an image is acquired. The AF point acquisition unit 501 acquires a focal point in the depth direction (hereinafter referred to as the AF point) when capturing an image with the imaging unit 400. The mode information acquisition unit 502 acquires mode information, which is information on a mode of a part that is to be used as a reference for the degree of focus when assigning importance to a subject, set when capturing an image with the imaging unit 400. In other words, the mode information acquisition unit 502 acquires mode information, which is information on a mode for setting a target subject to be captured in a captured image selected during capture.

[0031] The information acquisition unit 401 outputs the acquired shooting information, that is, the captured image and mode information, to the defocus range inference unit 402. Furthermore, the information acquisition unit 401 outputs the acquired shooting information, that is, the mode information and AF point (focus position), to the importance calculation unit 403. That is, the shooting information in this embodiment includes information on the captured image, mode information, and information on the AF point.

[0032] The defocus range inference unit (estimation unit) 402 identifies a part of the subject for which the defocus range is to be inferred (estimated) based on the photographing information output from the information acquisition unit 401, and infers the defocus range of the identified part. The defocus range is a range of values ​​of the defocus amount of the part of the subject. The parameters of the defocus range are the defocus amount values ​​of the two end points of the range (the nearest defocus amount and the farthest defocus amount).

[0033] Fig. 5 is a configuration diagram of a defocus range inference unit in embodiments 1 to 4. Fig. 5(A) is a configuration block diagram of the defocus range inference unit 402 in embodiment 1. Fig. 5(B) is a configuration block diagram of the defocus range inference unit 402 in embodiment 2. Fig. 5(C) is a configuration block diagram of the defocus range inference unit 402 in embodiment 3. Fig. 5(D) is a configuration block diagram of the defocus range inference unit 402 in embodiment 4. Functional units and the like in Figs. 5(B) to 5(D) will be described later.

[0034] The defocus range inference unit 402 is composed of a target identification unit 600 and a defocus range output unit 601. The target identification unit 600 identifies the part of the subject for which defocus range inference is to be performed (hereinafter referred to as the target part) from the mode information acquired by the mode information acquisition unit 502. The defocus range output unit 601 performs defocus range inference for the part of the subject identified by the target identification unit 600 for the captured image. The defocus range inference unit 402 outputs the inferred defocus range, mode information, and captured image to the importance calculation unit 403.

[0035] FIG. 6 is a diagram illustrating the defocus range. FIG. 6(A) illustrates the manner in which a person 700 is photographed using the imaging device 10. Reference numeral 701 denotes the person's pupil, 702 denotes the person's face, and 703 denotes the person's torso, each visualized as an object in the depth direction as viewed from the imaging device 10 (the range of existence). Reference numeral 704 indicates that the in-focus position of the imaging device 10 is the position of the person's pupil 701. FIG. 6(B) is a schematic diagram illustrating the estimated defocus ranges of the person's pupil 701, face 702, and torso 703. The horizontal axis indicates the amount of defocus, which is the deviation from the in-focus position, based on the in-focus position, which is the focal plane. That is, the magnitude (absolute value) of the defocus amount increases with increasing distance from the in-focus position. The side closer to the imaging device 10 is defined as the near side, and the far side is defined as the far side. The length of the line segments indicates the range in which each part of the subject (person) exists (in Figure 6(B) this is the person's pupils, face, and torso), and the distribution of the defocus amount of the subject parts corresponding to that range is shown.

[0036] In FIG. 6A, for example, the extent (existence range) of a person's torso 703 as an object in the depth direction as viewed from the camera is such that the closest point is, for example, the person's nose, and the farthest point is, for example, the person's shoulder. Therefore, the maximum defocus amount (nearest value) of the person's torso 703 is the defocus amount representing the person's nose, and the minimum defocus amount (farthest value) is the defocus amount representing the person's shoulder. The range defined by these values ​​is the defocus range of the person's torso 703. The person's torso in FIG. 6B represents this relationship, with the farthest value being, for example, 1.4Fδ and the nearest value being, for example, -0.2Fδ. The farthest value and the nearest value are acquired for each part of the subject as parameters indicating the defocus range.

[0037] In this way, the defocus range output unit 601 estimates a defocus range, which is a range of defocus amounts, taking into account the perspective relationship in the depth direction of the estimation target, such as a person's pupils, face, or torso. In this embodiment, the defocus range output unit 601 receives a captured image and a defocus map as input, and outputs the defocus range of the subject. The defocus map is information on the distribution of defocus amounts in which defocus amounts are assigned to a certain number of pixels on the imaging surface. As an estimation result, the defocus range output unit 601 distinguishes between subjects and outputs defocus ranges for the entire subject and for each part, such as the pupils, face, or torso.

[0038] The defocus range output unit 601 can acquire the defocus range by machine learning using training data as input. A specific example of a machine learning algorithm is deep learning, which uses a neural network to generate features and connection weighting coefficients for learning. Here, learning using a neural network will be described. Learning is performed using training data including a training image, a defocus map, and a correct defocus range as input data. The learning includes an error detection process and a weight update process. The error detection process obtains the error between the output data output from the output layer of the neural network according to the input data input to the input layer and the training data. At this time, the correct defocus range is used as the training data. The error detection process may use a loss function to calculate the error between the output data from the neural network and the training data.

[0039] In the weight update process, the connection weighting coefficients between the nodes of the neural network are updated based on the error obtained in the error detection process so as to reduce the error. This weight update process updates the connection weighting coefficients, for example, using the backpropagation method. The backpropagation method is a technique for adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the error.

[0040] The output data as a result of the learning is a machine learning model that estimates the defocus range. The defocus range is estimated using the machine learning model that has been trained using the above-mentioned learning method.

[0041] The importance calculation unit 403 calculates the importance of the captured image based on the defocus range output from the defocus range inference unit 402 and the positional relationship of the AF point output from the information acquisition unit 401. The importance calculation unit 403 outputs the calculated importance and the captured image to the storage unit 404.

[0042] The storage unit 404 associates the importance calculated by the importance calculation unit 403 with the captured image and stores the image in a storage medium such as an external storage device. Examples of the external storage device include an SD card, a flexible disk (FD), a CD-ROM, a DVD, a USB memory, and an MO. Alternatively, the external storage device may be a server device connected via a network.

[0043] Next, a processing procedure performed by the imaging device 10 in this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart of processing executed by the imaging device 10 in embodiment 1. Note that each operation (process) shown in the flowchart in Fig. 7 is realized by the system control unit 102 executing a program stored in the memory 104 or the like. In the following description, each process (step) will be represented by adding an S to the beginning, and the notation of the process (step) will be omitted.

[0044] In S800, the photographing unit 400 determines whether a desired mode was selected by the user when photographing. That is, it determines whether a mode for a part of the subject that the user wants to use as a reference when assigning importance to the image being photographed is selected when photographing. If a mode for a part of the subject is selected, proceed to S801. On the other hand, if a mode for a part of the subject is not selected, the process waits until a mode for that part is selected. In this embodiment, it is assumed that a mode specifying "a person's right eye" (right eye mode) is selected.

[0045] In S801, the photographing unit 400 photographs an image. Note that the photographing unit 400 photographs an image, for example, in response to a user input. Note that in S801, the image is photographed in the mode selected in S800 ("person's right eye" mode).

[0046] In S802, the information acquisition unit 401 acquires shooting information, which is information at the time of shooting in S801. Specifically, the captured image acquisition unit 500 included in the information acquisition unit 401 acquires an image (captured image) or video captured by the shooting unit 400. Furthermore, the AF point acquisition unit 501 included in the information acquisition unit 401 acquires the AF point. Furthermore, the mode information acquisition unit 502 included in the information acquisition unit 401 acquires information that shooting was performed in right-eye mode as mode information. The information acquisition unit 401 outputs the acquired captured image and mode information to the defocus range inference unit 402. Furthermore, the information acquisition unit 401 outputs the acquired mode information and AF point information to the importance calculation unit 403. In this way, the information acquisition unit 401 acquires the captured image (or video), the AF point, and mode information as shooting information.

[0047] In S803, the defocus range inference unit 402 infers (estimates) and outputs the defocus range. In this processing, the target identification unit 600 first identifies the target part from the mode information output in S802. In the present embodiment, the target part is identified as "the person's right eye." Next, the defocus range output unit 601 performs defocus range inference for the part of the subject identified by the target identification unit 600 for the captured image output in S802. In this processing, the defocus range output unit 601 infers the defocus range of "the person's right eye" in the captured image. The defocus range inference unit 402 outputs the inferred defocus range, mode information, and captured image to the importance calculation unit 403.

[0048] In S804, the defocus range inference unit 402 determines whether or not an AF point is included within the defocus range output in S803. Note that the AF point refers to the in-focus position (focus plane) when the imaging device performs autofocus control. If the AF point is included within the defocus range, the process proceeds to S805. On the other hand, if the AF point is not included within the defocus range, the process proceeds to S806.

[0049] 8A and 8B are examples of defocus range estimated values ​​in the first and second embodiments. Fig. 8A shows an example of the output of the defocus range in this embodiment. Fig. 8B is a diagram showing an example of the output of the defocus range in the second embodiment. Fig. 8B will be explained later.

[0050] In FIG. 8A, the defocus range of "the person's right eye" is "-2.0Fδ to 1.0Fδ," and the AF point is included in the defocus range.

[0051] In S805, the importance calculation unit 403 calculates the focus value V based on the AF point and the closest value of the defocus range. The focus value V is calculated, for example, as in the following equation (1), but is not limited to this.

number

[0052] In S806, the importance calculation unit 403 calculates a focus value V based on the difference between the AF point and the closest value or the farthest value of the defocus range. The focus value V is calculated, for example, using the following equation (2), but is not limited to this.

number

[0053] In S807, the importance calculation unit 403 calculates the range width P of the defocus range based on the difference between the closest value and the farthest value. The range width P is calculated, for example, according to the following equation (3).

number

[0054] In S808, the importance calculation unit 403 calculates the importance R based on the focus value V calculated in S805 or S806 and the range width P calculated in S807. Thereafter, the importance calculation unit 403 outputs the calculated importance R and the captured image to the storage unit 404. In this embodiment, the importance R is calculated, for example, according to the following equation (4), but is not limited to this.

number

[0055] In S809, the storage unit 404 associates the captured image for which the defocus range has been inferred with the importance R calculated in S808, and stores the image as Exchangeable Image File Format (Exif) information in a storage medium such as an external storage device.

[0056] As described above, the image capturing apparatus 10 of the first embodiment makes it possible to automatically assign a level of importance to a captured image at the time of capturing, depending on the degree of focus of a reference part of the subject set by the user. In other words, by correctly inferring the degree of focus of a local area intended by the user for each captured image based on the area selected by the user at the time of capturing and the defocus range inference result, it becomes possible to automatically assign a rating to the captured image at the time of capturing.

[0057] <Embodiment 2> In each of the following embodiments (Embodiments 3 and 4) including Embodiment 2, differences from Embodiment 1 will be explained, and unless otherwise specified below, they will be considered to be the same as Embodiment 1. Therefore, explanations of the same points and configurations as those of Embodiment 1 will be omitted.

[0058] In this embodiment, a method for assigning importance to a captured image based on the "priority of the part to be used as the reference for the degree of focus when assigning importance to the captured image" selected by the user during shooting will be described. In this embodiment, an example will be described in which the user selects a mode that gives first priority to "a person's right eye" and second priority to "a person's face." In other words, it is assumed that the user selects a mode with the intention of assigning importance based on whether the right eye is in focus first, and whether the face is in focus second. However, this mode is merely an example and does not limit the present invention. Furthermore, the number of priorities to be set can be changed within the range of parts for which defocus inference is possible.

[0059] The defocus range inference unit (estimation unit) 402 identifies all parts of the subject for which a defocus range is to be inferred, and infers the defocus range of each of the identified parts, based on the shooting information output from the information acquisition unit 401. As shown in Fig. 5(B) , the defocus range inference unit 402 in the second embodiment has a part priority specification unit 602 and a defocus range output unit 603.

[0060] The part priority specification unit 602 specifies a target part from the mode information acquired by the mode information acquisition unit 502. That is, the part priority specification unit 602 specifies multiple parts of the subject for which a defocus range is to be output and the priority of each part from the mode information acquired by the mode information acquisition unit 502. The part priority specification unit 602 specifies target parts that include all parts for which priorities have been set.

[0061] The defocus range output unit 603 performs defocus range inference for each target region identified by the region priority identification unit 602 for the captured image. That is, the defocus range output unit 603 estimates a defocus range for each region for which a priority is set. The defocus range inference unit 402 outputs the inferred defocus range, mode information, and captured image to the importance calculation unit 403.

[0062] The importance calculation unit 403 calculates the importance of the captured image based on the inferred positional relationship between the defocus range and the AF point and the priority of the parts in the mode information. The importance calculation unit 403 outputs the calculated importance and the captured image to the storage unit 404.

[0063] Next, a processing procedure performed by the imaging device 10 in this embodiment will be described with reference to Fig. 7. Note that each operation (process) shown in the flowchart in Fig. 7 is realized by the system control unit 102 executing a program stored in the memory 104 or the like. In the following description, each process (step) will be represented by adding an S to the beginning, and the notation of the process (step) will be omitted.

[0064] Note that S801 to S802 in the second embodiment are similar to the processing of S801 to S802 in the first embodiment, and therefore a description thereof will be omitted. Also, S804 to S807 in the second embodiment are similar to the processing of S804 to S807 in the first embodiment, and therefore a description thereof will be omitted. Also, S809 in the second embodiment is similar to the processing of S809 in the first embodiment, and therefore a description thereof will be omitted.

[0065] In S800, the photographing unit 400 determines whether a desired mode was selected by the user when photographing. That is, it determines whether the user has set (specified) a priority mode for a part of the subject that is to be used as a reference for the degree of focus when assigning importance to the image to be photographed. If a priority mode for a part of the subject has been set, the process proceeds to S801. On the other hand, if a priority mode for a part of the subject has not been set, the process waits until the priority mode is set. In this embodiment, it is assumed that a priority mode has been set in which "a person's right eye" is specified as the first priority and "a person's face" is specified as the second priority.

[0066] In S803, the defocus range inference unit 402 infers (estimates) and outputs the defocus range. In this process, the part priority identification unit 602 first identifies all target parts from the mode information output in S802. In the present embodiment, "person's right eye" and "person's face" are identified. Next, the defocus range output unit 603 performs defocus range inference for the target parts of the subject identified by the target identification unit 600 for the captured image output in S802. In this process, the defocus range output unit 601 infers the defocus ranges of "person's right eye" and "person's face" in the captured image. The defocus range inference unit 402 outputs the inferred defocus range, mode information, and captured image to the importance calculation unit 403.

[0067] As shown in Figure 8(B), the AF point for the "right eye of the person" is not included in the defocus range, so the focus value V riht_eye = 99, and from the above formula (3), the range width P riht_eye = 3.0. Since the AF point for the "person's face" is included within the defocus range, the maximum width P max_face = 40, the focus value V face ≒120, and from the above formula (3), the range width P face =6.0.

[0068] In S808, the importance calculation unit 403 calculates the importance R based on the focus value V of each part calculated in S805 or S806, the range width P calculated in S807, and the priority of the part in the mode information (priority for each part). The importance calculation unit 403 outputs the calculated importance R and the captured image to the storage unit 404. The calculation of the importance R in the second embodiment is performed, for example, according to the following equation (5), but is not limited to this.

[0069]

number

[0070] In the second embodiment, Parts_1=right_eye and Parts_2=face. W is a weight that is set in advance for each priority, and the weight changes according to the order of priority. In this embodiment, if W1=1.0 and W2=0.5, the importance R is calculated as 43.

[0071] As described above, according to the imaging device 10 of the second embodiment, it is possible to set a priority for the part of the subject that is used as the basis for the degree of focus set by the user, and it is possible to automatically assign importance to the captured image at the time of shooting while taking into account the degree of focus that takes the priority into account.

[0072] <Embodiment 3> In the third embodiment, when a user takes a photograph, the user inputs "a position to be used as a reference for the degree of focus when assigning a priority to the photographed image" on the live view screen during the photographing. Then, a method of assigning a priority to the photographed image based on the degree of focus and the distance of the part of the subject that is closest to the input position will be described.

[0073] The information acquisition unit 401 acquires information (shooting information) at the time of shooting by the shooting unit 400. As shown in Fig. 4(B) , the information acquisition unit 401 of the third embodiment has a captured image acquisition unit 503, an AF point acquisition unit 504, and a touch information acquisition unit 505. Note that the captured image acquisition unit 503 and the AF point acquisition unit 504 perform the same processes as the captured image acquisition unit 500 and the AF point acquisition unit 501 of the first embodiment, and therefore a description thereof will be omitted.

[0074] The touch information acquisition unit 505 acquires information about a position (hereinafter referred to as an input position) that is to be used as a reference for the degree of focus when assigning importance to a captured image input to the live view screen during shooting by the shooting unit 400 (touch position information acquisition). The touch information acquisition unit 505 outputs the captured image and the input position, which are the acquired shooting information, to the defocus range inference unit 402. Furthermore, the touch information acquisition unit 505 outputs the AF point, which is the acquired shooting information, to the importance calculation unit 403. Note that input may be by any means, such as a touch by the user, eye-gaze input, or voice input, and is not limited to these examples. Note that in the third embodiment, it is assumed that a touch input is made.

[0075] The defocus range inference unit 402 identifies a target region from the position of the input location and infers (estimates) the defocus range of the identified region based on the imaging information output from the information acquisition unit 401. As shown in Fig. 5(C), the defocus range inference unit 402 in the third embodiment has a region position output unit 604, a target identification unit 605, and a defocus range output unit 606.

[0076] The part position output unit 604 performs object detection inference on the captured image output from the information acquisition unit 401, and outputs all inferable part positions. The object detection model held by the part position output unit 604 is configured with DNN, and has been fully trained to determine the positions of parts of a subject. The subject categories include people, animals, and vehicles, and parts include, but are not limited to, eyes, torso, and face. The categories and parts must be the same as those that can be output by the defocus range output unit 606. The output result of object detection is the center position, category, and part type (label) of each part of the subject that can be inferred. In the third embodiment, it is assumed that the center positions of multiple parts of the subject that can be output are estimated (part position estimation).

[0077] The target identification unit 605 compares the input location (input position on the live view screen) in the captured image output from the information acquisition unit 401 with the output result of the object detection performed by the part position output unit 604 (center position of multiple parts) to identify the target part. Specifically, it calculates the distance (distance information) on a two-dimensional plane between the input location in the captured image and the center position of each of the inferred parts, and identifies the part of the subject that is closest to the input location (proximate part) as the target part. In other words, based on the distance information between the input position on the live view screen and the center position of multiple parts, the target identification unit 605 identifies the part that is closest to the input position on the live view screen, and identifies that part as the target part. Note that the distance calculation method may be, for example, Euclidean distance on a plane, but is not limited to this.

[0078] The defocus range output unit 606 performs defocus range inference on the target part (nearby part) identified by the target identification unit 605 for the captured image output from the information acquisition unit 401. The defocus range inference unit 402 outputs the inferred defocus range to the importance calculation unit 403.

[0079] Next, a processing procedure performed by the imaging device 10 in the third embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart of processing executed by the imaging device 10 in the third embodiment. Note that each operation (process) shown in the flowchart in Fig. 9 is realized by the system control unit 102 executing a program stored in the memory 104 or the like. In the following description, each process (step) will be represented by adding an S to the beginning, and the notation of the process (step) will be omitted.

[0080] Note that S1005 to S1008 in the third embodiment are similar to the processes of S804 to S807 in the first embodiment, and therefore a description thereof will be omitted. Also, S1010 in the third embodiment is similar to the process of S809 in the first embodiment, and therefore a description thereof will be omitted.

[0081] In S1000, the image capturing unit 400 determines whether a desired position on the live view screen has been input by the user during image capturing. That is, it determines whether a "position to be used as a reference for the degree of focus when assigning importance to a captured image" has been input by the user during image capturing. If a desired position on the live view screen has been input, the process proceeds to S1001. On the other hand, if a desired position on the live view screen has not been input, the process waits until the desired position is input. In this embodiment, the position to be used as a reference for the degree of focus when assigning importance to a captured image is input by touch input.

[0082] In S1001, the photographing unit 400 photographs an image. Note that the photographing unit 400 photographs an image, for example, in response to a user input.

[0083] In S1002, the information acquisition unit 401 acquires shooting information, which is information at the time of shooting in S1001. Specifically, the captured image acquisition unit 503 included in the information acquisition unit 401 acquires the image (captured image) and video captured by the shooting unit 400. Furthermore, the AF point acquisition unit 504 included in the information acquisition unit 401 acquires the AF point. Furthermore, the touch information acquisition unit 505 included in the information acquisition unit 401 acquires information on the input location that was touch-input in S1000. The information acquisition unit 401 outputs the acquired captured image and information on the input location to the defocus range inference unit 402. Furthermore, the information acquisition unit 401 outputs the acquired information on the AF point to the importance calculation unit 403. In this way, the information acquisition unit 401 acquires information on the captured image (or video), the AF point, and the input location as shooting information.

[0084] In S1003, the defocus range inference unit 402 identifies a part of the subject for which the defocus range is to be inferred (estimated) based on the input position on the live view screen, and infers and outputs the defocus range of the identified part. During this processing, the defocus range inference unit 402 first performs object detection inference on the captured image output by the part position output unit 604 in S1002, and outputs all positions of parts that can be inferred. Next, the target identification unit 605 compares the input position in the captured image with the output of the object detection to identify the target part. When identifying the target part, the target identification unit 605 calculates the distance between the input position and the position output by each object detection, and identifies the part of the subject that is closest to the input position as the target part.

[0085] 10 is a diagram showing an example of input locations in a captured image and output object detection in embodiment 3. 1101 is the input location for captured image 1100. 1102 to 1105 are the output object detection results. As detection results, 1102 is the position of the "person's right eye," 1103 is the position of the "person's left eye," 1104 is the position of the "person's face," and 1105 is the position of the "person's torso." In this case, "person's right eye" 1102, which is closest to input location 1101, is identified as the target part.

[0086] In S1004, the defocus range inference unit 402 infers (estimates) and outputs the defocus range of the target part identified by the target identification unit 605 for the captured image output from the information acquisition unit 401. In this process, the defocus range output unit 606 infers the defocus range of the "right eye of the person" in the captured image identified as the target part by the target identification unit 605. The defocus range inference unit 402 outputs the captured image and the inferred defocus range to the importance calculation unit 403.

[0087] In S1009, the importance calculation unit 403 calculates the importance R based on the focus value V calculated in S1006 or S1007, the range width P calculated in S1008, and the target distance D calculated in S1003. Thereafter, the importance calculation unit 403 outputs the calculated importance R and the captured image to the storage unit 404. Note that the calculation of the importance R in the third embodiment is performed, for example, according to the above formula (4), but is not limited to this.

[0088] Furthermore, after performing the processes of S1000 to S1010 for the first captured image, for the second and subsequent captured images, the processes of S801 to S809 in Figure 7 may be performed assuming that the same target region mode as for the first image is selected until S1000 is performed again.

[0089] As described above, the imaging device 10 of the third embodiment allows the user to automatically assign importance to a captured image at the time of shooting depending on the degree of focus of a desired point in the scene being photographed.

[0090] <Embodiment 4> In the fourth embodiment, when a user takes a photograph, he or she inputs "a position to be used as a reference for the degree of focus when assigning a priority to the photographed image" on the live view screen during shooting. A method for assigning a priority to a photographed image based on the input position, the degree of focus of multiple parts of the subject, and the distance will be described.

[0091] The information acquisition unit 401 performs the same processing as in embodiment 3, and therefore description thereof will be omitted. The defocus range inference unit 402 detects the positions of parts for inferring the defocus range of the subject, and infers the defocus range of each of the identified parts.

[0092] 5(D), the defocus range inference unit 402 in the fourth embodiment has a part position output unit 607, a distance identification unit 608, and a defocus range output unit 609. The part position output unit 607 performs the same processing as the part position output unit 604 in the third embodiment, and therefore a description thereof will be omitted.

[0093] The distance determination unit 608 calculates each distance on a two-dimensional plane of the captured image based on the input location in the captured image and the output of object detection. The distance determination unit 608 calculates each distance on a two-dimensional plane between the input location in the captured image and the center position of each inferred part, in the same way as in the third embodiment. In other words, the distance determination unit 608 acquires distance information between the input position on the live view screen and the center positions of the multiple parts (distance acquisition). The distance between the input location and the center position of each inferred part is referred to as the target distance.

[0094] The defocus range output unit 609 performs defocus range inference for all parts of the subject output by the part position output unit 607 for the captured image output from the information acquisition unit 401. The defocus range inference unit 402 outputs all the defocus ranges inferred by the defocus range output unit 609 and the target distances calculated by the distance identification unit 608 to the importance calculation unit 403.

[0095] The importance calculation unit 403 calculates the importance of the captured image based on the estimated positional relationship between each defocus range and AF point and / or the target distance. The importance calculation unit 403 also calculates the importance by vectorizing each importance and taking the inner product of each vector. The importance calculation unit 403 outputs the calculated importance and the captured image to the storage unit 404.

[0096] Next, a processing procedure performed by the imaging device 10 in the fourth embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart of processing executed by the imaging device 10 in the fourth embodiment. Note that each operation (process) shown in the flowchart in Fig. 11 is realized by the system control unit 102 executing a program stored in the memory 104 or the like. In the following description, each process (step) will be represented by adding an S to the beginning, and the notation of the process (step) will be omitted.

[0097] In the fourth embodiment, processing starts when the user inputs something on the live view screen during shooting in S1200 (when the system control unit 102 receives a signal indicating that an input has been made), and the processing of S1200 to S1211 is performed on the first captured image. Then, the second and subsequent captured images undergo the processing of S1212 to S1213, and then the processing of S1204 to S1211.

[0098] First, the processing for the first captured image (the captured image to be processed first) will be described. Here, S1200 to S1202 in the third embodiment are the same as the processing of S1000 to S1002 in the third embodiment, and therefore a description thereof will be omitted. Also, S1205 to S1208 are the same as the processing of S804 to S807 in the first embodiment, and therefore a description thereof will be omitted.

[0099] In S1203, the defocus range inference unit 402 performs object detection on all positions of the subject's body parts that can be inferred, and outputs the defocus range. First, the body part position output unit 607 performs object detection inference on the captured image and outputs all positions of the body parts that can be inferred. In this process, it is assumed that the center positions of the "dog's right eye," the "dog's left eye," the "dog's face," and the "dog's body" are output. Next, the distance identification unit 608 compares the input location in the captured image with the object detection output to calculate the target distance D.

[0100] In S1204, the defocus range inference unit 402 infers and outputs the defocus ranges of all parts of the subject inferred in S1203 for the captured image output from the information acquisition unit 401. In this process, the defocus range output unit 609 infers the defocus ranges of the above four parts (the dog's right eye, the dog's left eye, the dog's face, and the dog's body) that the defocus range inference unit 402 has determined to be inferable parts. The defocus range inference unit 402 outputs the captured image, the inferred defocus range, and the target distance D to the importance calculation unit 403.

[0101] In S1209, the importance calculation unit 403 calculates the region importance R for each region (multiple regions) based on the focus value V calculated in S1206 or S1207, the range width P calculated in S1208, and the target distance D calculated in S1203. Thereafter, the importance calculation unit 403 outputs the calculated importance R and the captured image to the storage unit 404. The region importance R is calculated, for example, according to the following equation (6), but is not limited to this.

number

[0102] Hereinafter, this will be referred to as the reference vector B. In the present embodiment, if "the dog's right eye" = right_eye, "the dog's right eye" = left_eye, "the dog's face" = face, and "the dog's body" = body, then the following equation (7) is obtained.

number

[0103] In S1210, the storage unit 404 associates the captured image for which the defocus range has been inferred with the importance calculated in S1209, and stores the associated image as Exif information in a storage medium such as an external storage device. When processing the first captured image, the importance may not be assigned, or the maximum value of the composition importance S at the time the processing in Fig. 11 is completed may be used as the importance of the first captured image. The composition importance S will be described later.

[0104] In S1211, the system control unit 102 determines whether to end the process. If the process is to be ended, the process shown in Fig. 11 ends. On the other hand, if the process is not to be ended, the process proceeds to S1212. In this process, the process proceeds to S1212 to process the second and subsequent captured images.

[0105] Next, processing for the second captured image and subsequent images will be described. In S1212, the imaging unit 400 captures an image. Note that the imaging unit 400 captures an image, for example, in response to a user input. In S1213, the information acquisition unit 401 acquires imaging information, which is information at the time of imaging in S1211. Specifically, the captured image acquisition unit 503 included in the information acquisition unit 401 acquires the captured image (captured image) or video. Furthermore, the AF point acquisition unit 504 included in the information acquisition unit 401 acquires the AF point. The information acquisition unit 401 outputs information about the acquired captured image to the defocus range inference unit 402. Furthermore, the information acquisition unit 401 outputs the acquired information about the AF point to the importance calculation unit 403. Thereafter, the process proceeds to S1204.

[0106] In S1204, the defocus range inference unit 402 infers and outputs the defocus ranges of all the subject parts inferred when processing the first captured image. In the present embodiment, the defocus range is output for each of the four parts (the dog's right eye, the dog's left eye, the dog's face, and the dog's body) described above. The captured image and the inferred defocus ranges are output to the importance calculation unit 403.

[0107] Next, the processing of S1205 to S1208 is performed on the output defocus range of each part. Note that S1205 to S1208 are the same as the processing performed on the first captured image, and therefore a description thereof will be omitted.

[0108] In S1209, first, the importance calculation unit 403 calculates the region importance R' for each region based on the focus value V calculated in S1206 or S1207 and the range width P calculated in S1208.

[0109] The calculation of the region importance R' is performed, for example, as shown in the above formula (4), but is not limited to this. The calculated region importance R' is collected into one vector (hereinafter referred to as composition vector B'). In the present embodiment, this is expressed by the following formula (8).

number

[0110] Next, the composition importance S is calculated using the following equation (9).

number

[0111] The importance calculation unit 403 then calculates the dot product of the reference vector B held when processing the first captured image and the composition vector B' to determine the composition importance S. When determining the composition importance S, the degree of focus of each part in the first image is used as a reference, and the similarity with the degree of focus of each part in the second and subsequent images is calculated and determined. If the focus composition is similar to that of the first captured image (for example, the right eye and face are in focus, but the torso is not), the composition importance S is calculated to be higher. The importance calculation unit 403 outputs the calculated composition importance S and the captured image to the storage unit 404.

[0112] In S1210, the storage unit 404 associates the captured image for which the defocus range has been inferred with the composition importance S calculated in S1209, and stores the linked image as Exif information in a storage medium such as an external storage device. Then, the process proceeds to S1211, where the same processing as described above is performed.

[0113] As described above, the imaging device 10 of the fourth embodiment enables the user to automatically assign importance to a captured image based on the composition of the captured image, which is determined based on the degree of focus of a desired point in the scene when the image is captured.

[0114] The embodiment described above is merely a typical example, and various modifications and changes can be made to the embodiment when implementing the present invention.

[0115] 7, 9, and 11 performed by the imaging device 10 are merely examples, and the imaging device 10 of this embodiment may change the processing or the details of the processing depending on user settings and the situation just before and after the processing starts. In other words, the imaging device 10 does not necessarily have to perform all of the processes (steps) described in the flowcharts shown in FIGS. 7, 9, and 11.

[0116] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0117] The disclosure of this embodiment includes the following configuration, method, and program.

[0118] (Configuration 1) An imaging means; information acquisition means for acquiring photographing information at the time of photographing by the imaging means; a defocus range estimation means for estimating a defocus range for one or more parts of a subject based on the photographing information at the time of photographing; an importance calculation means for calculating an importance of a captured image in which the defocus range is estimated based on the photographing information at the time of photographing and the defocus range; a storage means for storing the importance level and the captured image in association with each other, An imaging device characterized by:

[0119] (Configuration 2) The information acquisition means a photographed image acquisition means for acquiring the photographed image; and an AF point acquisition means for acquiring a focus position in the depth direction during shooting by the imaging means. 2. The imaging device according to claim 1,

[0120] (Configuration 3) 3. The imaging device according to configuration 2, wherein the importance calculation means calculates the importance based on a positional relationship between the defocus range and the in-focus position.

[0121] (Configuration 4) The imaging device according to any one of configurations 1 to 3, characterized in that the information acquisition means includes a mode information acquisition means for acquiring mode information, which is information on a mode for setting a target subject to be photographed in a photographed image selected during photography.

[0122] (Configuration 5) The defocus range estimation means a target specifying means for specifying a part of a subject that outputs the defocus range based on the mode information; and a defocus range output means for estimating and outputting the defocus range of the part. 5. The imaging device according to configuration 4.

[0123] (Configuration 6) The defocus range estimation means a region specifying means for specifying one or more regions of a subject for which a priority is set to output the defocus range from the mode information; a defocus range output means for estimating and outputting a defocus range for each of the parts to which the priority is set, 5. The imaging device according to configuration 4.

[0124] (Configuration 7) 7. The imaging device according to configuration 6, wherein the importance calculation means calculates the importance based on the positional relationship between the defocus range and the in-focus position, and the priority of each part.

[0125] (Configuration 8) 8. The imaging device according to any one of configurations 1 to 7, wherein the information acquisition means includes position information acquisition means for acquiring information about a position on a live view screen that is input during shooting.

[0126] (Configuration 9) The defocus range estimation means a part position estimation means for estimating the center positions of a plurality of parts of a subject; a target specifying means for specifying a region closest to the input position on the live view screen based on distance information between the input position on the live view screen and a center position of the plurality of regions; and a defocus range output means for estimating and outputting the defocus range of the portion closest to the center position. 2. The imaging device according to claim 1,

[0127] (Configuration 10) the information acquisition means includes an AF point acquisition means for acquiring a focus position in the depth direction when photographing by the imaging means, 10. The imaging device according to configuration 9, wherein the importance calculation means calculates the importance based on the positional relationship between the defocus range and the in-focus position and the distance information.

[0128] (Configuration 11) The defocus range estimation means a part position estimation means for estimating the center positions of a plurality of parts of a subject; a distance acquisition unit that acquires distance information between the input position on the live view screen and a center position of each of the plurality of parts; and a defocus range output means for estimating and outputting the defocus ranges of the plurality of portions. 2. The imaging device according to claim 1,

[0129] (Configuration 12) the information acquisition means includes an AF point acquisition means for acquiring a focus position in the depth direction when photographing by the imaging means, 12. The imaging device according to configuration 11, wherein the importance calculation means calculates the importance based on the positional relationship between the defocus ranges of the plurality of parts and the in-focus position or the distance information.

[0130] (Configuration 13) 13. A program for causing a computer to function as each of the means of the imaging device according to any one of configurations 1 to 12.

[0131] (Configuration 14) an information acquisition step of acquiring photographing information at the time of photographing by the imaging means; a defocus range estimating step of estimating a defocus range for one or more parts of a subject based on the photographing information at the time of photographing; an importance calculation step of calculating an importance of a captured image in which the defocus range is estimated based on the shooting information at the time of shooting and the defocus range; a storing step of storing the importance level and the captured image in association with each other, An imaging method comprising: [Explanation of symbols]

[0132] 10. Imaging device 400 Photography Department 401 Information Acquisition Department 402 Defocus range inference unit 403 Importance calculation part 404 Preservation Department

Claims

1. An imaging means; information acquisition means for acquiring photographing information at the time of photographing by the imaging means; a defocus range estimation means for estimating a defocus range for one or more parts of a subject based on the photographing information at the time of photographing; an importance calculation means for calculating an importance of a captured image in which the defocus range is estimated based on the photographing information at the time of photographing and the defocus range; a storage means for storing the importance level and the captured image in association with each other, An imaging device characterized by:

2. The information acquisition means a photographed image acquisition means for acquiring the photographed image; an AF point acquisition means for acquiring a focus position in the depth direction during shooting by the imaging means, 2. The imaging device according to claim 1.

3. 3. The imaging apparatus according to claim 2, wherein the importance calculation means calculates the importance based on a positional relationship between the defocus range and the in-focus position.

4. 2. The imaging device according to claim 1, wherein the information acquisition means includes mode information acquisition means for acquiring mode information that is information on a mode for setting a subject of interest in a photographed image selected during photography.

5. The defocus range estimation means a target specifying means for specifying a part of a subject that outputs the defocus range based on the mode information; and a defocus range output means for estimating and outputting the defocus range of the part.

5. The imaging device according to claim 4.

6. The defocus range estimation means a region specifying means for specifying one or more regions of a subject for which a priority is set to output the defocus range from the mode information; a defocus range output means for estimating and outputting a defocus range for each of the parts to which the priority is set, 5. The imaging device according to claim 4.

7. 7. The imaging apparatus according to claim 6, wherein the importance calculation means calculates the importance based on the positional relationship between the defocus range and the in-focus position, and the priority of each of the parts.

8. 2. The imaging apparatus according to claim 1, wherein the information acquisition means includes position information acquisition means for acquiring information about a position on a live view screen that is input during shooting.

9. The defocus range estimation means a part position estimation means for estimating the center positions of a plurality of parts of a subject; a target specifying means for specifying a region closest to the input position on the live view screen based on distance information between the input position on the live view screen and a center position of the plurality of regions; and a defocus range output means for estimating and outputting the defocus range of the portion closest to the center position.

2. The imaging device according to claim 1.

10. the information acquisition means includes an AF point acquisition means for acquiring a focus position in a depth direction when photographing by the imaging means, 10. The imaging apparatus according to claim 9, wherein the importance calculation means calculates the importance based on the positional relationship between the defocus range and the in-focus position and the distance information.

11. The defocus range estimation means a part position estimation means for estimating the center positions of a plurality of parts of a subject; a distance acquisition unit that acquires distance information between the input position on the live view screen and a center position of each of the plurality of parts; and a defocus range output means for estimating and outputting the defocus ranges of the plurality of portions.

2. The imaging device according to claim 1.

12. the information acquisition means includes an AF point acquisition means for acquiring a focus position in a depth direction when photographing by the imaging means, 12. The imaging apparatus according to claim 11, wherein the importance calculation means calculates the importance based on the positional relationship between the defocus ranges of the plurality of portions and the in-focus position or the distance information.

13. A program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 12.

14. an information acquisition step of acquiring photographing information at the time of photographing by the imaging means; a defocus range estimating step of estimating a defocus range for one or more parts of a subject based on the photographing information at the time of photographing; an importance calculation step of calculating an importance of a captured image in which the defocus range is estimated based on the shooting information at the time of shooting and the defocus range; a storing step of storing the importance level and the captured image in association with each other, An imaging method comprising:

Images