Image processing apparatus, image processing method, and program

Abstract

To provide an image processing apparatus that supports a user's selection of a picked-up image.SOLUTION: An image processing apparatus has: calculation means that acquires a defocus range of a photographed image from storage means storing information including the defocus range as incidental information of the photographed image, and calculates the focusing rate of the photographed image on the basis of the information including the defocus range; and display control means that performs display control of the photographed image on the basis of the focusing rate.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to image display processing. [Background technology]

[0002] When checking captured images after taking pictures with a camera, it is sometimes necessary to select the captured images by checking each image to see if they are focused at the desired position. In this case, when checking a large number of captured images, it can take a lot of time and effort to select the captured images that are focused at the desired position.

[0003] To solve such problems, Patent Document 1 proposes a method for selecting captured images in which the face area is in focus by obtaining the face area and the focus position from the captured image and displaying captured images in which the area is determined to match. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-15742 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology disclosed in the above-mentioned Patent Document 1 selects a captured image based only on the face area and the focus position, which poses a problem in that it is difficult to select a captured image that is accurately focused on the position desired by the user.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device that supports a user in selecting a captured image. [Means for solving the problem]

[0007] In order to achieve the above object, an image processing device according to one aspect of the present invention is characterized in that it has: a calculation means for acquiring information including the defocus range of a captured image from a storage means in which information including the defocus range of the captured image is stored as ancillary information of the captured image; a calculation means for calculating a focus ratio of the captured image based on the information including the defocus range; and a display control means for controlling the display of the captured image based on the focus ratio. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device that supports a user in selecting a captured image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a hardware configuration diagram of an image processing apparatus according to an embodiment. [Figure 2] 5A and 5B are diagrams illustrating a schematic relationship between a defocus amount and an image shift amount according to an embodiment. [Figure 3] FIG. 4 is a schematic explanatory diagram of a defocus range according to the embodiment. [Figure 4] 1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment. [Figure 5] 4 is a flowchart showing processing performed by the image processing device according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of a result of estimating the defocus range of the entire subject in a captured image according to the first embodiment. [Figure 7] 10 is a diagram showing an example of the configuration of a captured image in which captured image defocus range information according to the first embodiment is saved as supplementary information. FIG. [Figure 8] FIG. 10 is a diagram showing an example of a display in which captured images and focusing rates are linked based on the display order according to the first embodiment. [Figure 9] FIG. 10 is a block diagram showing an example of the configuration of an image processing device according to a second embodiment. [Figure 10] 10 is a flowchart showing processing performed by an image processing device according to a second embodiment. [Figure 11]10A and 10B are diagrams showing an example of the configuration of a captured image in which defocus range information during AF control and a lens driving amount are saved as supplementary information according to the second embodiment. [Figure 12] FIG. 10 is a diagram showing a method for calculating captured image defocus range information according to the second embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a display in which a captured image and a focusing rate are linked based on a partial region image according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0011] Before describing the embodiments of the present invention, the hardware configuration for implementing each of the embodiments will be described below with reference to Fig. 1. Fig. 1 is a diagram showing the hardware configuration of an image processing device 10 in each of the embodiments. That is, the image processing devices 10 in the first and second embodiments have the same configuration.

[0012] The image processing device 10 is an image processing device with an imaging function, such as an imaging device (imaging apparatus). The image processing device 10 is made up of a camera body 1000 and a lens unit 1100 that guides incident light to an imaging element 1001. First, the camera body 1000 will be described below.

[0013] The image sensor 1001 is made up of a CMOS imaging sensor and converts optical signals, which are optical images, into electrical signals. Light rays incident on the photographing lens 1101 pass through the aperture 1102 and shutter 1003 and are focused as an optical image on the image sensor 1001.

[0014] The system control unit 1002 is configured with at least one computer incorporating a CPU or the like, and controls the entire camera body 1000. The system control unit 1002 further includes an image processing unit (not shown) for processing video signals obtained by the image sensor 1001. The system control unit 1002 also includes a phase-difference AF unit (not shown) that performs focus detection processing using a phase-difference detection method based on focus detection image data (signals for phase-difference AF) obtained from the image sensor 1001 and the image processing unit. More specifically, the image processing unit generates, as focus detection image data, a pair of image data formed by light beams passing through a pair of pupil regions of the imaging optical system. The phase-difference AF unit detects the amount of focus deviation based on the amount of deviation between the pair of image data. In this way, the phase-difference AF unit performs phase-difference AF (image-plane phase-difference AF) based on the output of the image sensor 1001 without using a dedicated AF sensor. The system control unit 1002 may be configured and function as an image processing device. In that case, the camera body 1000, which is an imaging device (imaging apparatus) equipped with an imaging function, may have an image processing device built in.

[0015] The ROM 1004 is a non-volatile memory that records programs that control the operation of the camera body 1000, trained machine learning models, etc. The RAM 1005 is a volatile memory that records variables and various parameters necessary for the operation of the camera body 1000, setting values ​​such as ISO sensitivity, thresholds set in advance for image selection, live view images acquired during AF control, shooting modes, various correction data, etc.

[0016] The memory 1006 is a removable flash memory, and is a recording medium for recording captured images (picked-up images) and additional information associated with the captured images. The power switch 1007 switches between power on and off modes for the camera body 1000. The mode switching unit 1008 is a switch for switching between various shooting modes such as live view shooting and video shooting.

[0017] The rear monitor 1009 is a display unit composed of an LCD device, LEDs, etc. that displays shooting information such as text, captured images, audio, and other operating status and messages in response to program execution in the system control unit 1002. The touch panel 1010 is arranged in an area roughly equivalent to the rear monitor 1009, detects contact with a finger or pen, notifies the system control unit 1002 of the contact position on the rear monitor 1009, and executes an operation or function associated with the contact position.

[0018] The viewfinder display unit 1011, like the rear monitor 1009, is a display unit that displays shooting information in response to program execution by the system control unit 1002, and together with the eyepiece 1012, forms an electronic viewfinder (EVF). An eyepiece detection unit 1013 causes the system control unit 1002 to selectively display the above-mentioned shooting information on the rear monitor 1009 or the viewfinder display unit 1011 depending on the photographer's eye position. The shutter control unit 1014 controls the operation of the shutter 1003 based on the photometric results of the subject calculated by the system control unit 1002. The shutter 1003 can be controlled in conjunction with the aperture 1102.

[0019] Next, the configuration of the lens unit 1100 will be described. The camera body 1000 and the lens unit 1100 are mechanically and electrically connected via a lens mount mechanism (mount section) 1015. Furthermore, the camera body 1000 and the lens unit 1100 are detachable via the lens mount mechanism 1015. The lens unit 1100 is made up of a photographing lens 1101, an aperture 1102, a lens drive circuit 1103, an aperture control circuit 1104, and a lens control section 1105. Note that for the sake of simplicity, only one photographing lens 1101 is shown in FIG. 1, but in reality it is made up of a group of multiple photographing lenses.

[0020] The lens control unit 1105 is configured with at least one computer having a CPU, memory, etc., and controls the entire lens unit 1100. The lens control unit 1105 has a memory (not shown) that stores, for example, various constants, variables, programs, etc. for lens operation. It also has a non-volatile memory (not shown) that stores information specific to the lens unit, such as maximum and minimum aperture values ​​and focal length.

[0021] The system control unit 1002 of the camera body 1000 calculates the defocus amount using the output information of the image sensor 1001. Then, based on the calculated defocus amount, the system control unit 1002 communicates via the lens control unit 1105 of the lens unit 1100 and controls the lens drive circuit 1103 to adjust the focus.

[0022] The above-mentioned defocus amount will now be described in detail with reference to Fig. 2. Fig. 2 is a diagram illustrating the relationship between the defocus amount of the imaging optical system and the phase difference (image shift amount) between the first focus detection signal and the second focus detection signal acquired from the imaging element in this embodiment.

[0023] An image sensor (not shown) is disposed on an image pickup surface 200 in FIG. 2, and the exit pupil of the image pickup optical system is divided into a first pupil region 201 and a second pupil region 202. The defocus amount d is defined such that |d| is the distance (size) from an imaging position C of light beams from the subject 203 and the subject 204 to the image pickup surface 200, and a front-focus state in which the imaging position C is located on the subject side of the image pickup surface 200 is represented by a negative sign (d<0). Furthermore, a back-focus state in which the imaging position C is located on the opposite side of the image pickup surface 200 from the subject is represented by a positive sign (d>0). In a focused state in which the imaging position C is located on the image pickup surface 200, d=0. The image pickup optical system is in a focused state (d=0) with respect to the subject 203, and in a front-focus state (d<0) with respect to the subject 204. The front-focus state (d<0) and the back-focus state (d>0) are collectively referred to as a defocused state (|d|>0).

[0024] In a front-focus state (d<0), a light beam from the subject 204 that passes through the first pupil region 201 (second pupil region 202) is first focused and then spreads to a width Γ1 (Γ2) centered at the center of gravity G1 (G2) of the light beam, forming a blurred image on the imaging surface 200. This blurred image is received by each first focus detection pixel (each second focus detection pixel) on the image sensor, and a first focus detection signal (second focus detection signal) is generated. In other words, the first focus detection signal (second focus detection signal) is a signal that represents an image of the subject 204 at the center of gravity G1 (G2) of the light beam on the imaging surface 200, blurred by the width Γ1 (Γ2).

[0025] The width Γ1 (Γ2), which is the width of blur of the subject image, increases roughly in proportion to the increase in the magnitude |d| of the defocus amount d. Similarly, the magnitude |p| of the image shift amount p (= the difference G1-G2 in the center of gravity positions of the light beams) between the first focus detection signal and the second focus detection signal also increases roughly in proportion to the increase in the magnitude |d| of the defocus amount d. In the back-focus state (d>0), the direction of the image shift between the first focus detection signal and the second focus detection signal is opposite to that in the front-focus state, but the same is true.

[0026] Thus, the amount of image shift between the first and second focus detection signals increases as the defocus amount increases. In this embodiment, image-surface phase-difference detection focus detection is performed, which calculates the defocus amount from the image shift between the first and second focus detection signals obtained using the image sensor 1001. Therefore, the phase-difference AF unit of the system control unit 1002 converts the image shift amount into a detected defocus amount as the defocus amount of the image-capturing signal increases. Specifically, because the image shift amount between the first and second focus detection signals increases, the image shift amount is converted into a detected defocus amount using a conversion coefficient calculated based on the base length. Note that the unit of defocus amount in this embodiment is the product [Fδ] of the aperture F-number and the permissible circle of confusion diameter δ in the optical system of the image capture device when capturing an image.

[0027] The defocus range, which will be described later, will be described in detail with reference to Fig. 3. Fig. 3 is a diagram illustrating the defocus range in this embodiment. Fig. 3(A) illustrates how a person 301 is photographed using the image processing device 10.

[0028] 3, 302 indicates the person's pupil, 303 indicates the person's face, and 304 indicates the person's torso, which are visualized as objects in the depth direction as viewed from the image processing device 10. Also, 305 indicates that the focus position in the image processing device 10 is the position of the person's pupil 302.

[0029] FIG. 3B is a schematic diagram showing the defocus ranges of a person's pupils 302, face 303, and torso 304. The horizontal axis indicates the defocus amount, which is the deviation from the in-focus position, based on the in-focus position, which is the focal plane. That is, the magnitude (absolute value) of the defocus amount increases with increasing distance from the in-focus position. In this embodiment, the horizontal axis is defined as the side closer to the image processing device 10 as the near side and the side farther away as the far side, with the defocus amount taking negative values ​​on the near side and positive values ​​on the far side. The line segments indicate the ranges within which each part of the subject (the pupils, face, and torso in FIG. 3B) is located, and the distribution of defocus amount values ​​for the subject parts corresponding to that range is shown. This range is hereinafter referred to as the defocus range. The parameters of the defocus range are the defocus amount values ​​at the two end points of the range (the closest defocus amount and the farthest defocus amount shown in FIG. 7). For example, the defocus range for each part is indicated as a person's pupil (-0.05 to 0.10 [Fδ]), a person's face (-0.20 to 0.30 [Fδ]), a person's torso (-0.20 to 0.80 [Fδ]), etc.

[0030] In FIG. 3A, for example, the extent of a person's torso 304 as an object in the depth direction as viewed from the image processing device (imaging device) 10 is such that the closest side is, for example, the person's nose, and the farthest side is, for example, the person's shoulder. Therefore, the maximum defocus amount (nearest value) of the person's torso 304 is the defocus amount representing the person's nose, and the minimum defocus amount (farthest value) is the defocus amount representing the person's shoulder. The range defined by these values ​​is the defocus range of the person's torso 304. The person's torso in FIG. 3B represents these relationships, and the person's eyes and face are also represented based on the above relationships. Note that, although the defocus ranges of the person's eyes, face, and torso have been described as examples here, there are no restrictions on the target subject or body part, and the present embodiment is not limited to this example.

[0031] <Embodiment 1> The image processing device 10 of the first embodiment calculates a focus ratio based on defocus range information of the entire subject that is stored as supplementary information of the captured image, and then determines the selection and display order of the captured images based on the calculated focus ratio, and supports the task of selecting captured images by displaying them in association with the focus ratio.

[0032] Fig. 4 is a block diagram showing an example of the configuration of the image processing device 10 according to the first embodiment. The image processing device 10 includes a captured image acquisition unit 401, a defocus range estimation unit 402, an auxiliary information storage unit 403, and a defocus range acquisition unit 404. The image processing device 10 further includes a depth of field calculation unit 405, a focus ratio calculation unit 406, a captured image selection unit 407, a display order determination unit 408, and a captured image display unit 409. Note that Fig. 4 is an example of a functional configuration example, and does not limit the scope of application of the present invention.

[0033] The captured image acquisition unit 401 acquires a captured image captured by the imaging device (image processing device 10) and recorded in the memory 1006. Thereafter, the captured image acquisition unit 401 outputs the acquired captured image to the defocus range estimation unit 402.

[0034] The defocus range estimation unit 402 estimates the defocus range of the entire subject in the received captured image. Details of the captured image defocus range estimation by the defocus range estimation unit 402 will be described later using the flowchart in Fig. 5. The defocus range estimation unit 402 acquires captured image defocus range information (first range information) that is the estimation result, and outputs the captured image defocus range information to the incidental information storage unit 403.

[0035] The incidental information storage unit 403 stores the received captured image defocus range information as incidental information of the captured image recorded in the memory 1006. Details of the process of storing the incidental information of the captured image by the incidental information storage unit 403 will be described later using the flowchart in Fig. 5. The incidental information storage unit 403 outputs the captured image defocus range information stored as incidental information of the captured image to the defocus range acquisition unit 404.

[0036] The defocus range acquisition unit 404 acquires the defocus range of the captured image based on the received captured image defocus range information, and then outputs the acquired defocus range to the depth of field calculation unit 405.

[0037] The depth of field calculation unit 405 calculates the depth of field of the captured image recorded in the memory 1006, which is associated with the received defocus range. Details of the calculation of the depth of field by the depth of field calculation unit 405 will be described later using the flowchart in Fig. 5. The depth of field calculation unit 405 outputs the defocus range and the calculated depth of field to the focus ratio calculation unit 406.

[0038] The focusing ratio calculation unit 406 calculates the focusing ratio based on the received defocus range and depth of field. Details of the focusing ratio calculation by the focusing ratio calculation unit 406 will be explained later using the flowchart in Fig. 5. The focusing ratio calculation unit 406 stores the calculated focusing ratio in the auxiliary information of the captured image recorded in the memory 1006 linked to the defocus range.

[0039] The captured image selection unit 407 acquires one or more captured images recorded in the memory 1006 and selects a captured image based on the focus rate stored in the accompanying information of the captured image. Specifically, the captured image selection unit 407 selects a captured image whose focus rate value is equal to or greater than a predetermined (default) threshold. The captured image selection unit 407 outputs information on the display order of the one or more selected captured images to the display order determination unit 408.

[0040] The display order determination unit 408 determines the display order of the received one or more selected captured images when they are displayed on an imaging unit such as the rear monitor 1009, based on the focus rates stored in the supplementary information of the captured images. The display order determination unit 408 outputs the selected one or more captured images and the display order to the captured image display unit 409.

[0041] The captured image display unit 409 displays the received captured images on the rear monitor 1009 based on the selected one or more captured images and the display order. When displaying the received captured images on the rear monitor 1009, the captured image display unit 409 displays them on the rear monitor 1009 in association with the focusing rate. Note that, while it is preferable that the captured images be displayed in association with the focusing rate when displayed on the rear monitor 1009 as described above, the captured image display unit 409 may display only the captured images without linking the focusing rate. In other words, only the captured images may be displayed based on the display order.

[0042] The captured image display unit 409 may also display the captured image and the focusing ratio on a display device such as an external monitor or display. The display device may be configured as an integrated part of a client device (information processing device) such as a PC, or may be configured as a separate device. The captured image display unit 409 may also display on a plurality of display means such as the external device or the rear monitor 1009.

[0043] Next, the processing procedure performed by the image processing device 10 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the processing of the image processing device 10 in embodiment 1. Specifically, this flowchart shows the processing procedure for calculating the focus rate of the entire subject based on defocus range information stored in the supplementary information of the captured image, and displaying the captured images by selecting images and determining the display order.

[0044] 5 is realized by the system control unit 1002 executing a program stored in the ROM 1004 or the like. In the following description, each process (step) is represented by adding an S to the beginning of the process (step) to omit the notation of the process (step).

[0045] In S501, the captured image acquisition unit 401 acquires a captured image captured by an imaging device (image processing device 10) and recorded in the memory 1006. Thereafter, the captured image acquisition unit 401 outputs the acquired captured image to the defocus range estimation unit 402. Note that, as an example, the processing proceeds assuming that one (one) captured image has been acquired. However, the number of captured images is not limited to one, and there is no restriction on the number of captured images to be acquired; for example, multiple captured images may be acquired and processed, and the present embodiment is not limited to the example described below.

[0046] In S502, the defocus range estimation unit 402 inputs the captured image received from the captured image acquisition unit 401 into a defocus range estimation model recorded in the ROM 1004. Then, the defocus range estimation unit 402 estimates captured image defocus range information that indicates the defocus range of the entire subject that encompasses each part of the human body of the subject in the captured image, such as the pupils, face, and torso. Thereafter, the defocus range estimation unit 402 acquires captured image defocus range information (first range information) that is the estimation result, and outputs the acquired captured image defocus range information to the incidental information storage unit 403.

[0047] The defocus range estimation model is a model obtained by machine learning. A specific example of a machine learning algorithm is deep learning, which uses a neural network to generate features and connection weighting coefficients for learning. Here, learning using a neural network is described. Learning is performed using training data including training images and a correct defocus range as input data. The learning process includes an error detection process and a weight update process. The error detection process obtains the error between the output data output from the output layer of the neural network based on the input data input to the input layer and the training data. The correct defocus range is used as the training data. The error detection process may use a loss function to calculate the error between the output data from the neural network and the training data. The weight update process updates the connection weighting coefficients between the nodes of the neural network based on the error obtained in the error detection process to reduce the error. This weight update process updates the connection weighting coefficients, for example, using backpropagation. Backpropagation is a technique for adjusting the connection weighting coefficients between the nodes of each neural network to reduce the error.

[0048] The defocus range can be estimated using a machine learning model trained by the above-described learning method. Here, as an example, the defocus range is estimated using a defocus range estimation model trained using a training image and a correct defocus range as input data, but the present embodiment is not limited to this example. A machine learning model trained using different input data or a program that calculates the defocus range according to a predetermined algorithm may also be used.

[0049] Fig. 6 shows an example of the defocus range estimation result of the entire subject in a captured image, estimated using the defocus range estimation model. Hereinafter, the defocus range estimation and the result of the entire subject in a captured image will be described with reference to Fig. 6. Fig. 6(A) illustrates how a subject 601 is photographed using the image processing device 10. Fig. 6(B) shows an example of the defocus range estimation result of the entire subject.

[0050] Reference numeral 602 in FIG. 6A visualizes the extent of the entire subject as an object in the depth direction as viewed from the image processing device 10. Reference numeral 603 displays the in-focus position in the image processing device 10. In the example shown in FIG. 6B, a defocus range is estimated, which is the range of defocus amounts from the nose tip of the subject, which is the closest side, to the shoulder tip of the subject, which is the farthest side. Note that, as an example, the entire subject has been described here as a defocus range estimated from the range of defocus amounts from the nose tip of the subject to the shoulder tip of the subject, but this is not limited to this example in this embodiment. For example, the entire subject may be defined as a range that includes the defocus amounts of any part.

[0051] In S503, the incidental information storage unit 403 stores the captured image defocus range information received from the defocus range estimation unit 402 as incidental information of the captured image recorded in the memory 1006. Thereafter, the incidental information storage unit 403 extracts the captured image defocus range information stored as incidental information of the captured image, and outputs it to the defocus range acquisition unit 404.

[0052] Fig. 7 shows an example of the configuration of a captured image in which captured image defocus range information is stored as incidental information. Hereinafter, with reference to Fig. 7, an example of the configuration of a captured image in which captured image defocus range information is stored as incidental information will be described.

[0053] Data 701 is data of a captured image in a state in which captured image defocus range information is stored as incidental information in the captured image recorded in memory 1006. Information 702 is captured image defocus range information stored as incidental information, and is composed of a defocus range estimation target, a nearest defocus amount, and a farthest defocus amount. Note that, as an example, the captured image defocus range information composed of the defocus range estimation target, the nearest defocus amount, and the farthest defocus amount is stored as incidental information linked to the captured image. However, this is not limiting, and there are no restrictions on the configuration of the captured image defocus range information stored as incidental information, and this embodiment is not limited to this example. Note that, if a defocus range is obtained for each part of the subject, the information on those parts and information 702 are stored in memory 1006 as incidental information in a format in which they are associated with each other.

[0054] In S504, the defocus range acquisition unit 404 receives captured image defocus range information from the incidental information storage unit 403. Then, the defocus range acquisition unit 404 acquires the defocus range of the entire subject in the captured image based on the defocus range estimation target, the closest defocus amount, and the farthest defocus amount recorded in the captured image defocus range information. The defocus range acquisition unit 404 outputs the acquired defocus range of the entire subject to the depth of field calculation unit 405.

[0055] In S505, the depth of field calculation unit 405 calculates the depth of field of the captured image recorded in the memory 1006, which is associated with the defocus range received from the defocus range acquisition unit 404. The depth of field calculation unit 405 outputs the calculated depth of field and the defocus range received from the defocus range acquisition unit 404 to the focus ratio calculation unit 406.

[0056] In the process of S505, the depth of field calculation unit 405 acquires the camera parameters (focal length S, aperture value F, subject distance D, and permissible circle of confusion diameter δ) at the time of shooting recorded in the supplementary information of the captured image in order to calculate the depth of field. Then, it calculates the depth of field X of the captured image using the following formula (1).

number

[0057] Here, as an example, the depth of field is calculated using the above formula (1). However, there are no restrictions on the method of calculating the depth of field, and the depth of field may be calculated by different means, such as calculation using a different calculation formula or estimation using a machine learning model. Furthermore, to simplify the processing, the calculated value of the depth of field may be stored as additional information of the image.

[0058] In S506, the focus ratio calculation unit 406 calculates a focus ratio according to the defocus range received from the depth of field calculation unit 405. Specifically, the focus ratio calculation unit 406 calculates a focus ratio indicating the ratio of the defocus range included in the depth of field based on the defocus range and depth of field received from the depth of field calculation unit 405. In this processing, the received defocus range is the defocus range of the entire subject in the captured image, so the focus ratio calculation unit 406 calculates the focus ratio based on the defocus range of the entire subject in the captured image and the depth of field. The focus ratio calculation unit 406 then stores the calculated focus ratio in the memory 1006 as auxiliary information of the captured image associated with the defocus range received from the defocus range acquisition unit 404. That is, by performing the processing of S506, in addition to the above-mentioned information recorded in the memory 1006 associated with the defocus range received from the defocus range acquisition unit 404, information on the focusing rate is also saved as auxiliary information.

[0059] In explaining the method for calculating the focus rate in S506, let DN be the closest defocus amount of the defocus range, DF be the farthest defocus amount DF, and XN be the closest defocus amount of the depth of field calculated from the depth of field X, and XF be the farthest defocus amount X. In order to calculate the defocus amount as a positive numerical value, when the numerical values ​​of DN, DF, XN, and XF are corrected so that their minimum values ​​are 0, and dn, df, xn, and xf are respectively used, the focus rate Z (%) of the captured image is calculated using the following formula (2):

number

[0060] Here, as an example, the focus ratio is calculated using the above formula (2). However, there are no restrictions on the method for calculating the focus ratio of a captured image, and the focus ratio may be calculated by a different means, such as by calculation using a different formula.

[0061] In S507, the captured image selection unit 407 acquires one or more captured images recorded in the memory 1006 and selects the captured images based on the focus ratios stored in the supplementary information of the captured images. Specifically, the captured images to be displayed on the rear monitor 1009 or the like are selected based on the focus ratios stored in the supplementary information of the acquired captured images and a predetermined threshold. The captured image selection unit 407 then outputs the selected one or more captured images to the display order determination unit 408. Note that the predetermined threshold refers to a threshold used for selection that is previously set in the RAM 1005, such as a focus ratio of 80% or more. In other words, when selecting captured images to be displayed on the rear monitor 1009 or the like, the captured image selection unit 407 determines whether the focus ratio stored in the supplementary information of each captured image is equal to or greater than a predetermined threshold. If the focus ratio is equal to or greater than the predetermined threshold, the captured image is selected as the captured image to be displayed on the rear monitor 1009 or the like, and if the focus ratio is less than the predetermined threshold, the captured image is not selected as the captured image to be displayed on the rear monitor 1009 or the like.

[0062] In S508, the display order determination unit 408 acquires one or more selected captured images from the captured image selection unit 407, and determines the display order to display them on the rear monitor 1009 etc. based on the focus rates stored in the supplementary information of the captured images. Thereafter, the display order determination unit 408 outputs the one or more selected captured images received from the captured image selection unit 407 and the information on the display order determined above to the captured image display unit 409, and proceeds to S509.

[0063] When determining the display order in S508, specifically, the display order determination unit 408 sorts the images in descending order of focus rate and creates a list describing that order. Note that, although a method of determining the display order by sorting the images in descending order of focus rate and creating a list has been shown as an example here, there are no restrictions on the determination method or storage method, such as sorting the images in descending order of focus rate and storing the display order in the additional information of the captured image instead of creating a list.

[0064] In S509, the captured image display unit 409 associates the captured image with a focus rate based on the information about the selected one or more captured images and display order received from the display order determination unit 408, and displays the associated captured image and focus rate on the rear monitor 1009. Thereafter, the processing ends. FIG. 8 is an example of a display in which the captured image and focus rate are associated based on the display order. Below, the display of the captured image associated with the focus rate will be described with reference to FIG. 8.

[0065] Photographed image 801, photographed image 802, photographed image 803, and photographed image 804 are examples of photographed images selected by the photographed image selection unit 407. Focusing rate 805 is information stored as incidental information of photographed image 801. Focusing rate 806 is information stored as incidental information of photographed image 802. Focusing rate 807 is information stored as incidental information of photographed image 803. Focusing rate 808 is information stored as incidental information of photographed image 804.

[0066] The captured image display unit 409 displays these selected captured images on the rear monitor 1009 or the like in the order of captured image 801 to captured image 804 based on the information on the display order determined by the display order determination unit 408. Furthermore, the captured image display unit 409 also displays the focus rates stored in the captured images as incidental information associated with the captured images. In the case of FIG. 8 , for example, when captured image 801 is displayed, focus rate 805 is also displayed together with the captured image on the rear monitor 1009 or the like, as shown in FIG. 8 . In other words, the captured image display unit 409 displays information on focus rate 805 superimposed on captured image 801 on the rear monitor 1009 or the like. Note that, while an example has been shown here in which captured images and focus rates are associated and displayed based on the display order, there are no restrictions on display control, such as the number of captured images to be displayed or the display position of the focus rates, and the present embodiment is not limited to this example. Furthermore, the focus rate displayed in association with the captured image may be displayed not as a numerical value indicating a percentage as in FIG. 8, but as a graph showing the focus rate as a percentage.

[0067] Through the above processing, the focus rate of the entire subject is calculated based on the defocus range information stored in the supplementary information of the captured image, and the captured images can be displayed by selecting images and determining the display order.

[0068] In this way, according to the image processing device 10 of this embodiment, the focus ratio is calculated based on the defocus range information of the entire subject that is saved as supplementary information of the captured image. Then, the captured image selection and display order are determined based on the calculated focus ratio, and the captured images are displayed in association with the focus ratio, thereby supporting the user (operator, worker) in selecting captured images.

[0069] In this embodiment, the defocus range may be estimated from a live view image acquired during AF control, and the result of the estimation may be used to perform AF control and acquire a captured image.The defocus range may then be estimated again from a captured image captured and saved after AF control, and the result may be saved in the supplementary information of the captured image.

[0070] In this embodiment, an imaging device (imaging apparatus) is shown as an example of the image processing apparatus 10, but the focus ratio can also be calculated using a general-purpose PC as the image processing apparatus. In this case, the focus ratio is calculated from the defocus range information and the information at the time of shooting (focal length S, aperture value F, subject distance D, and permissible circle of confusion diameter δ) as supplementary information of the captured image, making it possible to perform the processes of S508 and S509 in Fig. 5. In addition, at this time, the captured image display unit 913 may display the image on an external monitor.

[0071] <Embodiment 2> The image processing device 90 of the second embodiment will be described below. In the second embodiment, the defocus range is estimated using a live view image acquired during AF control, and the drive amount (lens drive amount) of the photographing lens 1101 during AF control is calculated based on the estimated result. AF control is then performed using the result to acquire a photographed image. The photographed image, the estimated result, and the lens drive amount are then saved as supplementary information for the photographed image. This is because there is a time lag between the live view image acquired during AF control and the photographed image, and the estimated result of the defocus range may not match. The defocus range of the photographed image is then calculated using the estimated result of the defocus range of the saved live view image and the lens drive amount. Furthermore, the defocus range of each part of the subject is acquired from the photographed image defocus range information, and the focus ratio for each part is calculated. A photographed image is then selected based on the calculated focus ratio and the focus ratio corresponding to the selected part input by the user, and a partial region image is displayed linked to the focus ratio. By performing such processing, it is possible to assist the user in selecting a photographed image.

[0072] 9 is a block diagram showing an example of the configuration of an image processing device 90 according to embodiment 2. The image processing device 90 includes a live view image acquisition unit 901, a defocus range estimation unit 902, a lens drive amount acquisition unit 903, a captured image acquisition unit 904, an auxiliary information storage unit 905, and a defocus range calculation unit 906. The image processing device 90 further includes a defocus range acquisition unit 907, a depth of field calculation unit 908, a focus ratio calculation unit 909, a selected portion information acquisition unit 910, a captured image selection unit 911, a partial region information acquisition unit 912, and a captured image display unit 913.

[0073] The configuration of the image processing device 90 according to the second embodiment is substantially the same as the configuration of the image processing device 10 according to the first embodiment shown in Fig. 4. Therefore, descriptions of the same functional parts will be omitted, and only functional parts that differ will be described. Specifically, the following describes a live view image acquisition unit 901, a defocus range estimation unit 902, a lens driving amount acquisition unit 903, an auxiliary information storage unit 905, and a defocus range calculation unit 906. Furthermore, the following also describes a selected portion information acquisition unit 910, a captured image selection unit 911, a partial region information acquisition unit 912, and a captured image display unit 913.

[0074] The live view image acquisition unit 901 acquires a live view image temporarily stored in the RAM 1005 for use during autofocus control (AF control). The live view image acquisition unit 901 then outputs the acquired live view image to a defocus range estimation unit 902 and a lens drive amount acquisition unit 903.

[0075] A defocus range estimation unit (AF control defocus range estimation unit) 902 estimates the defocus range of each part of the subject in the received live view image. Details of the defocus range estimation during AF control by the defocus range estimation unit 902 will be explained later using the flowchart in Fig. 10. The defocus range estimation unit 902 acquires AF control defocus range information (second range information) that is the estimation result, and outputs the AF control defocus range information to a lens drive amount acquisition unit 903 and an additional information storage unit 905.

[0076] The lens drive amount acquisition unit 903 calculates the drive amount of the photographing lens 1101 in the lens control unit 1105 based on the received live view image and the defocus range information during AF control, and acquires the lens drive amount to be used for AF control. Thereafter, the lens drive amount acquisition unit 903 outputs the acquired lens drive amount to the photographed image acquisition unit 904.

[0077] The incidental information storage unit 905 performs AF control based on the received lens drive amount and acquires a captured image. It also receives defocus range information during AF control output from the defocus range estimation unit 902. The incidental information storage unit 905 then saves the received lens drive amount and defocus range information during AF control as incidental information for the acquired captured image, and records this in the memory 1006. Details of the process by the incidental information storage unit 905 to save the information as incidental information for the captured image will be explained later using the flowchart in FIG. 10. The incidental information storage unit 905 outputs the defocus range information during AF control and the lens drive amount to the defocus range calculation unit 906.

[0078] The defocus range calculation unit 906 reflects the amount of change in the defocus range based on the received AF control defocus range information and lens drive amount, and calculates captured image defocus range information (first range information) which is the defocus range estimation result of the captured image. Note that details of the captured image defocus range calculation will be described later with reference to the flowchart in Fig. 10. The defocus range calculation unit 906 outputs the calculated captured image defocus range information to the defocus range acquisition unit 907.

[0079] The selected body part information acquisition unit 910 acquires information on the body part of the subject selected from an input device such as the touch panel 1010 (selected body part information). Then, the selected body part information acquisition unit 910 outputs the acquired selected body part information to the photographed image selection unit 911.

[0080] The captured image selection unit 911 acquires one or more captured images recorded in the memory 1006, and selects a captured image in which the focus rate of a specific part is equal to or greater than a predetermined threshold, based on the focus rate and selected part information stored in the supplementary information of the captured image. The captured image selection unit 911 outputs the one or more selected captured images and the selected part information to the partial region information acquisition unit 912.

[0081] The partial region information acquisition unit 912 acquires partial region information indicating the region of the specific part within the photographed image, based on the region information of each part detected during photography and the selected part information stored in the supplementary information of the one or more selected photographed images received from the photographed image selection unit 911. Thereafter, the partial region information acquisition unit 912 outputs the one or more selected photographed images and the acquired partial region information to the photographed image display unit 913.

[0082] The captured image display unit 913 associates the captured image with a focusing rate based on the received one or more selected captured images and partial region information, and displays them on the rear monitor 1009. When displaying the captured image on the rear monitor 1009, the captured image display unit 913 associates the focusing rate with the captured image and displays it on the rear monitor 1009. As in the first embodiment, the captured image display unit 913 may display the captured image and focusing rate on a display device such as an external monitor or display. Note that the display device may be configured as an integrated unit with a client device (information processing device) such as a PC, or may be configured as a separate device. The captured image display unit 913 may display the captured image on multiple display means such as the external device or the rear monitor 1009.

[0083] Next, the processing procedure performed by the image processing device 90 in this embodiment will be described with reference to Fig. 10. In the following description, each process (step) will be denoted with an S at the beginning. Fig. 10 is a flowchart showing the processing of the image processing device 10 in the second embodiment. Specifically, the flowchart shows the processing procedure in which the image processing device 90 calculates the focus rate of each part of the subject based on the defocus range information during AF control stored in the supplementary information of the captured image, and selects an image and displays a partial region image based on the input selected part information.

[0084] 10 is realized by the system control unit 1002 executing a program stored in the ROM 1004 or the like. In the following description, each process (step) is represented by adding an S to the beginning of the process (step), and the process (step) is not represented in detail. In order to clarify the differences from the process shown in FIG. 5 described in the first embodiment, the description of the same process as that described in FIG. 5 will be omitted. Specifically, S1004 and S1008 in FIG. 10 are similar to S501 and S505 shown in FIG. 5 in the first embodiment, and therefore the description will be omitted.

[0085] In S1001, the live view image acquisition unit 901 acquires a live view image that is temporarily stored in the RAM 1005 used during AF control. Thereafter, the captured image acquisition unit 904 outputs the acquired live view image to the defocus range estimation unit 902 and the lens drive amount acquisition unit 903. Note that, as an example, the processing proceeds assuming that one (one) live view image has been acquired. However, the acquired live view image is not limited to one, and there is no restriction on the number of captured images that can be acquired; for example, multiple live view images can be acquired and processed, and the present embodiment is not limited to the example described below.

[0086] In S1002, the defocus range estimation unit 902 inputs the live view image received from the live view image acquisition unit 901 into a defocus range estimation model recorded in ROM 1004. Then, it estimates defocus range information during AF control that indicates the defocus range of each part of the human body, such as the pupils, face, and torso of the subject in the live view image. The defocus range estimation unit 902 then acquires the defocus range information during AF control (first range information), which is the estimation result, and outputs it to the acquired lens drive amount acquisition unit 903 and the incidental information storage unit 905.

[0087] Note that the defocus range estimation model is the same as the machine learning model described in S502, and therefore a detailed description thereof will be omitted. Here, as an example, the defocus range is estimated using a defocus range estimation model trained using a training image and a correct defocus range as input data, but this embodiment is not limited to this example. It is also possible to use a machine learning model trained using different input data, or a program that calculates the defocus range according to a predetermined algorithm.

[0088] In S1003, the lens drive amount acquisition unit 903 calculates the lens drive amount based on the live view image received from the live view image acquisition unit 901 and the AF control defocus range information received from the defocus range estimation unit 902. This acquires the lens drive amount used by the lens control unit 1105 when performing control during AF control. Thereafter, the lens drive amount acquisition unit 903 outputs the acquired lens drive amount to the captured image acquisition unit 904.

[0089] In S1005, the incidental information storage unit 905 performs AF control based on the lens drive amount received from the captured image acquisition unit 904, and acquires a captured image. The incidental information storage unit 905 also receives defocus range information during AF control from the defocus range estimation unit 902. The incidental information storage unit 905 then saves the received lens drive amount and defocus range information during AF control as incidental information for the captured image, and records these in the memory 1006. The incidental information storage unit 905 then outputs the defocus range information during AF control and the lens drive amount to the defocus range calculation unit 906. FIG. 11 shows an example of the configuration of a captured image in which defocus range information during AF control is saved as incidental information. A captured image in which defocus range information during AF control is saved as incidental information will be described below with reference to FIG. 11.

[0090] In FIG. 11, 1111 denotes a captured image in which defocus range information during AF control is saved as incidental information to the captured image recorded in memory 1006. 1112 denotes defocus range information during AF control saved as incidental information. The defocus range information during AF control is composed of a defocus range estimation target, a closest defocus amount, and a furthest defocus amount. 1113 denotes the lens drive amount used for AF control when capturing the captured image.

[0091] Note that, here, as an example, defocus range information during AF control, which is configured from a defocus range estimation target, a closest defocus amount, a furthest defocus amount, and a lens drive amount during AF control, is saved as incidental information. However, this is not limiting, and there are no restrictions on the configuration of the defocus range information during AF control saved as incidental information, and the present embodiment is not limited to this example.

[0092] In S1006, the defocus range calculation unit 906 calculates captured image defocus range information, which is the defocus range estimation result of the captured image. Specifically, the defocus range calculation unit 906 calculates the captured image defocus range information by reflecting the amount of change in the defocus range based on the AF control defocus range information and the lens drive amount received. The defocus range calculation unit 906 outputs the calculated captured image defocus range information to the defocus range acquisition unit 907.

[0093] To calculate the captured image defocus range information, the lens drive coefficient B, aperture value F, permissible circle of confusion diameter δ, and lens drive amount L at the time of shooting, which are recorded in the incidental information of the captured image, are acquired. Then, the amount of change C in the defocus range is calculated using the following equation (3), and the amount of change C is used to calculate the captured image defocus range information.

number

[0094] Here, as an example, the amount of change in the defocus range is calculated using the above formula (3). However, there are no restrictions on the method for calculating the amount of change in the defocus range, and the amount of change in the defocus range may be calculated by a different means, such as by calculation using a different calculation formula. Fig. 12 shows a method for calculating captured image defocus range information. Hereinafter, the method for calculating captured image defocus range information will be described with reference to Fig. 12.

[0095] Reference numeral 1201 denotes defocus range information during AF control. Reference numeral 1202 denotes a change in the defocus range calculated from the lens drive amount. Reference numeral 1203 denotes calculated captured image defocus range information. Note that, as an example, the captured image defocus range information during AF control, including the defocus range of each part of the subject's pupil, face, and torso, is calculated based on a change in the defocus range calculated from the lens drive amount. However, there are no restrictions on the configuration or calculation method of the defocus range information, and the present embodiment is not limited to this example.

[0096] In S1007, the defocus range acquisition unit 907 receives captured image defocus range information from the defocus range calculation unit 906. Then, the defocus range of each part of the subject in the captured image is acquired based on the defocus range estimation target, the closest defocus amount, and the farthest defocus amount recorded in the captured image defocus range information. The defocus range acquisition unit 907 outputs the acquired defocus range of each part of the subject to the depth of field calculation unit 908.

[0097] In S1009, the focus ratio calculation unit 909 calculates a focus ratio according to the defocus range received from the depth of field calculation unit 908. Specifically, the focus ratio calculation unit 909 calculates a focus ratio indicating the ratio of the defocus range included in the depth of field of each part, based on the defocus range and depth of field of each part of the subject received from the depth of field calculation unit 908. The focus ratio calculation unit 909 stores the calculated focus ratio of each part in the auxiliary information of the captured image recorded in the memory 1006, which is associated with the defocus range received from the defocus range acquisition unit 907. Note that while the focus ratio of each part is calculated in S1009, the method of calculating the focus ratio is the same as that described in S506, and therefore detailed description thereof will be omitted.

[0098] In S1010, the selected body part information acquisition unit 910 acquires selected body part information, which is information about the subject's body parts such as the pupils, face, and torso used when selecting an image, from an input device such as a touch panel 1010, and outputs the information to the captured image selection unit 911. Note that, while an example in which selected body part information such as the subject's pupils, face, and torso is acquired from an input device such as a touch panel is shown here as an example, there are no restrictions on the input method of the selected body part information or the configuration of the selected body part information, and the present embodiment is not limited to this example.

[0099] In S1011, the captured image selection unit 911 acquires one or more captured images recorded in the memory 1006. Then, based on the focus rate stored in the incidental information of the captured image and the selected part information received from the selected part information acquisition unit 910, the captured image selection unit 911 selects captured images in which the focus rate of a specific part is equal to or greater than a predetermined threshold set in advance. The one or more selected captured images and the selected part information are output to the partial region information acquisition unit 912. Note that the predetermined threshold refers to a threshold used for selection that is set in advance in the RAM 1005, such as a focus rate of 80% or more. This refers to a threshold used for selection that is set in advance in the RAM 1005.

[0100] In S1012, the partial region information acquisition unit 912 acquires partial region information based on the region information of each part of the subject detected during shooting and the selected part information stored in the supplementary information of the one or more selected photographed images received from the photographed image selection unit 911. The partial region information is information that indicates the area of ​​a specific part within the photographed image. Then, the partial region information acquisition unit 912 outputs the one or more selected photographed images and the acquired partial region information to the photographed image display unit 913.

[0101] In S1013, the captured image display unit 913 associates the captured image with the focusing rate based on the one or more selected captured images and partial region information received from the partial region information acquisition unit 912, and displays them on the rear monitor 1009 or the like. Thereafter, the processing ends. Fig. 13 is an example of a display in which the captured image and the focusing rate are associated based on the partial region information. Below, a display in which the captured image and the focusing rate are associated based on the partial region information will be described with reference to Fig. 13.

[0102] Photographed images 1301, 1302, 1303, and 1304 are examples of photographed images in which a face is input by the selected body part information acquisition unit 910, and a partial facial region of the photographed image selected based on the focusing rate of the face is enlarged and displayed. Focusing rate 1305 is information stored as incidental information of photographed image 1301. Focusing rate 1306 is information stored as incidental information of photographed image 1302. Focusing rate 1307 is information stored as incidental information of photographed image 1303. Focusing rate 1308 is information stored as incidental information of photographed image 1304.

[0103] The captured image display unit 913 also displays the focus rate, which is stored as incidental information for the captured image, in association with the captured image. In the case of Fig. 13, for example, when a captured image 1301 is displayed, a focus rate 1305 is also displayed together with the captured image on the rear monitor 1009 or the like, as shown in Fig. 11. In other words, the captured image display unit 913 displays the focus rate 1305 information superimposed on the captured image 1301. Note that, while an example has been shown here in which the captured image and the focus rate are displayed in association with each other based on partial region information, there are no restrictions on display control, such as the number of captured images to be displayed or the display position of the focus rate, and the present embodiment is not limited to this example.

[0104] As described above, the image processing device 90 of this embodiment calculates the focus rate for each part of the subject based on the defocus range information during AF control stored in the supplementary information of the captured image. Then, by selecting an image and displaying a partial region image based on the selected part information input by the user, it is possible to support the user (operator, worker) in the task of selecting a captured image, as in the first embodiment.

[0105] The embodiment described above is merely a typical example, and various modifications and changes can be made to the embodiment when implementing the present invention.

[0106] For example, in the first and second embodiments, an image processing apparatus has been described in which defocus range information is stored in the incidental information of a captured image within one image processing apparatus, the defocus range is acquired from the defocus range information in the stored incidental information, and the focus ratio is calculated. However, the present invention is not limited to this example. For example, a captured image with incidental information stored therein may be recorded in a removable flash memory, and the flash memory may be read by another image processing apparatus. Then, the defocus range may be acquired from the defocus range information stored in the incidental information of the captured image within the other image processing apparatus, and the focus ratio may be calculated.

[0107] 5 performed by the image processing device 10 and the process of Fig. 10 performed by the image processing device 90 are just examples, and the image processing device of this embodiment may change the process or the details of the process depending on user settings and the situation before and after the start of the process. In other words, the image processing device does not necessarily have to perform all the processes (steps) described in the flowcharts shown in Fig. 5 and Fig. 10.

[0108] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0109] The disclosure of this embodiment includes the following configuration, method, and program.

[0110] (Configuration 1) a calculation means for acquiring information including a defocus range of a photographed image from a storage means in which information including the defocus range of the photographed image is stored as supplementary information of the photographed image, and for calculating a focus ratio of the photographed image based on the information including the defocus range; a display control means for controlling the display of the captured image based on the focusing rate; 1. An image processing device comprising:

[0111] (Configuration 2) 2. The image processing device according to configuration 1, wherein the defocus range of the captured image is the defocus range of the entire subject in the captured image, which is estimated using a defocus range estimation model.

[0112] (Configuration 3) 3. The image processing device according to configuration 1 or 2, wherein the calculation means calculates a focus ratio according to a defocus range of the entire subject in the captured image.

[0113] (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the calculation means calculates the focus ratio based on the defocus range and the depth of field of the captured image.

[0114] (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the display control means causes the display means to display the focusing rate in association with the captured image.

[0115] (Configuration 6) a display order determination means for determining the display order of the photographed images based on the focusing rate; 6. The image processing device according to any one of configurations 1 to 5, wherein the display control means causes the captured images to be displayed on the display means based on the determined display order.

[0116] (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, further comprising a selection unit that selects a captured image to be displayed on a display unit based on the focusing rate.

[0117] (Configuration 8) 7. The image processing device according to any one of configurations 1 to 6, further comprising a selection unit that selects a captured image to be displayed on a display unit based on the focusing rate and a predetermined threshold value.

[0118] (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the defocus range of the captured image is estimated from the defocus range estimated from a live view image and a lens driving amount for AF control.

[0119] (Configuration 10) 10. The image processing device according to any one of configurations 1 to 9, wherein the defocus range of the captured image is estimated for each part of the subject.

[0120] (Configuration 11) The image processing device according to configuration 9, wherein the storage means stores the defocus range estimated from the live view image and the lens driving amount as information including the defocus range of the captured image as auxiliary information of the captured image.

[0121] (Configuration 12) 7. The image processing device according to any one of configurations 1 to 6, further comprising a selection unit that selects a captured image to be displayed on the display unit based on a focus rate corresponding to a part of a subject selected from an input device.

[0122] (Configuration 13) acquiring information including the defocus range of a photographed image from a storage means in which the information including the defocus range of the photographed image is stored as supplementary information of the photographed image; a calculation step of calculating a focus ratio of the captured image based on information including the defocus range; a display control step of controlling the display of the captured image based on the calculated focusing rate. An image processing method comprising:

[0123] (Configuration 14) 13. A program for causing a computer to function as each means of the image processing device according to any one of configurations 1 to 12.

[0124] (Configuration 15) an acquisition means for estimating a defocus range of a captured image and acquiring the estimation result as defocus range information; a focus ratio calculation unit that calculates a focus ratio of the captured image based on the defocus range information acquired by the acquisition unit, 1. An image processing device comprising:

[0125] (Configuration 16) an acquisition step of estimating a defocus range of a captured image and acquiring the estimation result as defocus range information; a focus ratio calculation step of calculating a focus ratio of the captured image based on the defocus range information acquired in the acquisition step. An image processing method comprising: [Explanation of symbols]

[0126] 10 Image processing device 402 Defocus range estimation unit 403 Additional Information Storage Unit 404 Defocus range acquisition unit 406 Focus rate calculation section 407 Photographed image display section

Claims

1. a calculation means for acquiring information including a defocus range of a photographed image from a storage means in which information including the defocus range of the photographed image is stored as supplementary information of the photographed image, and for calculating a focus ratio of the photographed image based on the information including the defocus range; a display control means for controlling the display of the captured image based on the focusing rate; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the defocus range of the captured image is the defocus range of the entire subject in the captured image, which is estimated using a defocus range estimation model.

3. 2. The image processing apparatus according to claim 1, wherein the calculation means calculates a focus ratio according to a defocus range of the entire subject in the captured image.

4. 2. The image processing apparatus according to claim 1, wherein the calculation means calculates the focus ratio based on the defocus range and the depth of field of the captured image.

5. The image processing apparatus according to claim 1 , wherein the display control means causes the display means to display the focusing rate in association with the captured image.

6. a display order determination means for determining the display order of the photographed images based on the focusing rate; 2. The image processing apparatus according to claim 1, wherein the display control means causes the display means to display the captured images based on the determined display order.

7. 2. The image processing apparatus according to claim 1, further comprising a selection unit for selecting a photographed image to be displayed on the display unit based on the focusing rate.

8. 2. The image processing apparatus according to claim 1, further comprising a selection unit that selects a captured image to be displayed on the display unit based on the focusing rate and a predetermined threshold value.

9. 2. The image processing device according to claim 1, wherein the defocus range of the captured image is estimated from a defocus range estimated from a live view image and a lens driving amount for AF control.

10. 2. The image processing apparatus according to claim 1, wherein the defocus range of the photographed image is estimated for each part of the subject.

11. 10. The image processing device according to claim 9, wherein the storage means stores the defocus range estimated from the live view image and the lens driving amount as information including the defocus range of the captured image as supplementary information of the captured image.

12. 2. The image processing apparatus according to claim 1, further comprising a selection unit for selecting a photographed image to be displayed on the display unit based on a focus ratio corresponding to a part of a subject selected from an input device.

13. acquiring information including the defocus range of a photographed image from a storage means in which the information including the defocus range of the photographed image is stored as supplementary information of the photographed image; a calculation step of calculating a focus ratio of the captured image based on information including the defocus range; a display control step of controlling the display of the captured image based on the calculated focusing rate. An image processing method comprising:

14. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 12.

15. an acquisition means for estimating a defocus range of a captured image and acquiring the estimation result as defocus range information; a focus ratio calculation unit that calculates a focus ratio of the captured image based on the defocus range information acquired by the acquisition unit, 1. An image processing device comprising:

16. an acquisition step of estimating a defocus range of a captured image and acquiring the estimation result as defocus range information; a focus ratio calculation step of calculating a focus ratio of the captured image based on the defocus range information acquired in the acquisition step. An image processing method comprising:

Images