Image processing device, image processing method, and program

Abstract

The degree of focus on a specific object within an image is evaluated using the same index, even when photographed under different conditions. [Solution] Image data of a captured image is input to the data input unit 11, and the focus calculation unit 13 calculates a value indicating the degree of focus of the objects contained in the image based on the image data input to the data input unit 11 and the shooting information at the time the image was taken. Then, the display unit 15 displays the value indicating the degree of focus calculated by the focus calculation unit 13 on the display device 3.

Description

Technical Field

[0001] The present invention relates to a technique for quantifying the focus degree of a subject and making a determination based on a photographed image and photographing information.

Background Art

[0002] In recent years, with the improvement of the continuous shooting performance of digital cameras, the total number of photographed images in one shooting has increased dramatically. Along with this, the work of extracting the most focused image during image selection after shooting takes a huge amount of time, which has become a heavy burden on professional photographers and general users.

[0003] In Patent Document 1, the original image is converted into a spectrum in the spatial frequency domain, the converted image is approximated by a sigmoid function with respect to the data integrated in the angular direction, and an evaluation value indicating the quality of the image is obtained from the parameter indicating the inflection point, thereby evaluating the degree of blur.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, Patent Document 1 has a problem that it can be evaluated if it is the same composition such as during continuous shooting or when the geometric conditions do not change, but it cannot be evaluated with the same index when the composition is different or the shooting lens (also referred to as a lens unit) used for shooting is different. In addition, since the evaluation is performed on the entire image, there is a problem that accurate evaluation cannot be performed on an image with a composition that focuses on a part.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a technique capable of evaluating the focus degree with the same index even when photographing a specific object in an image under different conditions. [Means for solving the problem]

[0007] To solve the above problems, the image processing apparatus of the present invention is characterized by comprising: an input means for receiving image data of a captured image; a focus calculation means for calculating a value indicating the degree of focus of an object included in the image based on the image data input to the input means and the image data for which the image was captured; and an output means for outputting the value indicating the degree of focus calculated by the focus calculation means. [Effects of the Invention]

[0008] According to the present invention, the degree of focus can be quantified by frequency analysis that takes visual characteristics into account, and by making a determination using the shooting information, the degree of focus can be evaluated using the same index even under various conditions. [Brief explanation of the drawing]

[0009] [Figure 1] This is an example of the configuration of a system including an image processing device according to the first embodiment. [Figure 2] Figure 1 is a flowchart showing the processing flow of the image processing device. [Figure 3] (A) is a flowchart showing the processing flow of the object detection unit, and (B) is a flowchart showing the processing flow of the focus calculation unit. [Figure 4] This diagram shows an example of cutting out an object. [Figure 5] This diagram illustrates the characteristics of human vision (contrast response). [Figure 6] This is an example of a difference image between when the image is in focus and when it is out of focus. [Figure 7] This is an example of a user interface where images and evaluation values ​​are displayed. [Figure 8] This is an example of the configuration of a system including an image processing device according to the second embodiment. [Figure 9](A) is a flowchart showing the processing flow of the image processing apparatus shown in FIG. 1, and (B) is a flowchart showing the processing flow of the focus determination unit. [Figure 10] It is a diagram showing an example of a threshold surface. [Figure 11] It is an example of a user interface on which an image, an evaluation value, and a determination result are displayed. [Figure 12] It is an example of a determination mode displayed on the display device according to the third embodiment. [Figure 13] It is an example of a determination mode displayed on the display device according to the third embodiment. [Figure 14] It is an example of a determination mode displayed on the display device according to the third embodiment.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the situations described below, and can be applied to any embodiment that conforms to the gist of the present invention.

[0011] [First Embodiment] FIG. 1 shows a configuration example of a system including an image processing apparatus according to the first embodiment. The system shown in FIG. 1 includes an image processing apparatus 1, a camera 2, a display device 3, and an external storage 4.

[0012] The image processing apparatus 1 includes a data input unit 11, an object detection unit 12, a focus degree calculation unit 13, a data holding unit 14, and a display unit 15.

[0013] The data input unit 11 receives image data from the camera 2. That is, the data input unit 11 functions as an input means for receiving the image data of the captured image. The object detection unit 12 detects and extracts an object from the image data input to the data input unit 11. The focus degree calculation unit 13 calculates a value indicating the degree of focus of the detected object.

[0014] The data holding unit 14 holds the input image data, shooting information, and a value indicating the degree of focus calculated by the focus calculation unit 13. The data holding unit 14 also functions as a work memory for the processes described later. Note that the shooting information in the present embodiment refers to information regarding the conditions, environment, and settings at the time of shooting, such as lens characteristics described later, and information obtained as a result of shooting, such as pixel size. The display unit 15 generates screen data constituting the image data, applications, etc., and causes the display device 3 to display it. That is, the display unit 15 functions as an output unit that outputs a value indicating the degree of focus calculated by the focus calculation means.

[0015] The above-described data input unit 11, object detection unit 12, focus calculation unit 13, data holding unit 14, and display unit 15 realize their respective functions by the CPU executing a program stored in a memory included in, for example, the image processing apparatus 1.

[0016] The camera 2 includes an interchangeable lens unit 2a (hereinafter referred to as the lens unit 2a). The lens unit 2a can be exchanged according to the application, etc. The camera 2 shoots a subject and transfers the image data to the image processing apparatus 1 or the external storage 4. Note that the transfer of the image data may be performed by wired communication or wireless communication.

[0017] The display device 3 displays the screen data output by the display unit 15, and for example, a liquid crystal display or the like can be used.

[0018] The external storage 4 holds image data and the like, such as a hard disk drive (HDD). Note that the external storage 4 may be configured as a cloud storage accessed via a network. Further, the external storage 4 may be connectable to the image processing apparatus 1.

[0019] (System processing flow) Figure 2 is a flowchart showing the processing flow (image processing method) of the image processing device 1 shown in Figure 1. In this embodiment, the size of the region containing the object for which the degree of focus value is to be calculated on the image (number of pixels in the vertical and horizontal directions) is used as the shooting information, and frequency analysis is used to calculate the value indicating the degree of focus. Note that the flowchart shown in Figure 2, etc., can be realized, for example, by a processing device such as the CPU of the image processing device 1 reading and executing a program stored in a storage device such as memory.

[0020] First, in step S1, image data of an image captured by camera 2, or image data held in external storage 4, is input from the data input unit 11.

[0021] Next, in step S2, the object detection unit 12 detects an object from the input image data. Details of the processing by the object detection unit 12 will be described later.

[0022] Next, in step S3, the focus calculation unit 13 calculates a value indicating the degree of focus of the detected object. Details of the processing by the focus calculation unit 13 will be described later.

[0023] Then, in step S4, the display unit 15 displays the image data and the calculated value indicating the degree of focus on the display device 3. An example of the display will be described later.

[0024] In other words, step S1 functions as an input process, steps S2 and S3 as focus calculation processes, and step S4 as an output process.

[0025] (Processing of the object detection unit 12 in step S2) Figure 3(A) is a flowchart showing the processing flow of the object detection unit 12 in step S2 described above.

[0026] First, in step S21, an object is detected from the input image data. When detecting an object, if it is, for example, a person's pupil, a known person detection algorithm can be used to detect the pupil of the person closest to the focus point.

[0027] Next, in step S22, an object such as a person's pupil is detected from the input image data, and the region containing the detected object is cut out (extracted). When cutting out, as shown in Figure 4, the pupil closest to the focus point is cut out. In this embodiment, the size of the cut-out region is a rectangle containing the pupil, and the vertical and horizontal pixel sizes are powers of 2, for example, 128 pixels × 128 pixels. Note that the vertical and horizontal pixel sizes (hereinafter referred to as pixel size) of the region from which an object such as a pupil is cut out are not limited to powers of 2. That is, the object detection unit 12 functions as an extraction means that detects an object from an image and extracts the region on the image that contains the object.

[0028] Furthermore, the object is not limited to a person's eyes; it may also be a person's face, head, or entire body, or the eyes, face, head, or entire body of an animal other than a person. In other words, the object may be a living organism or a specific part of a living organism.

[0029] Furthermore, although this flowchart describes an example in which the object detection unit 12 is used to determine the position and pixel size of an object, the user may pre-specify information such as the position and pixel size of the object.

[0030] (Processing of the focus calculation unit 13 in step S3) Figure 3(B) is a flowchart showing the processing flow of the focus calculation unit 13 in step S3 described above. Figure 3(B) explains an example of calculating a value indicating the degree of focus based on frequency analysis.

[0031] First, in step S31, image data of the region containing the object cut out in step S2 is input.

[0032] Next, in step S32, a Fourier transform is performed on the input image data to convert it into frequency domain data.

[0033] Next, in step S33, the frequency components are corrected using visual characteristics. For example, human visual characteristics (contrast response) are shown in Figure 5. As shown in Figure 5, it is known that human visual characteristics (contrast response) have the highest sensitivity at 5 cycles / deg. Here, if the observation distance is 400 mm, the distance per 1 deg on a display 400 mm away can be calculated as shown in equation (1) below. 400 × tan(1) = 6.98 (1)

[0034] From equation (1), the value is 6.98 mm / deg. If the above-mentioned display is a 24-inch, Full HD display (518 mm wide, 1920 pixels), then we get equation (2). 1920 × 6.98 ÷ 518 = 25.9 ... (2)

[0035] Therefore, on a 24-inch, Full HD display with an observation distance of 400mm, the distance per degree corresponds to 25.9 pixels / deg.

[0036] From equation (2) above and Figure 5, if 25.9 pixels equals 5 cycles, then the line with 5.2 pixels / cycles has the highest pixel count. In other words, finer lines than this result in lower sensitivity.

[0037] Based on the above, the cutoff frequency is set to 5.2 pixels / cycle, and frequencies higher than 5.2 pixels / cycle are cut off. Specifically, filtering is performed to reduce the power of frequencies higher than 5.2 pixels / cycle to zero. In other words, the focus calculation unit 13 converts the region extracted by the extraction means into a frequency domain and performs filtering on the converted frequency domain according to the visual characteristics.

[0038] Returning to Figure 3(B), in step S34, an inverse Fourier transform (inverse FFT) is performed on the data after frequency processing.

[0039] Then, in step S35, the absolute difference between the image input in step S31 and the image obtained by performing the inverse Fourier transform in step S34 is calculated for each pixel. The sum of the absolute differences of each pixel in the entire image (extracted image) is then calculated as a value indicating the degree of focus (hereinafter referred to as the degree of focus).

[0040] When calculating the degree of focus, for example, it is good to calculate the average value of the G channel pixel among the RGB pixels. Alternatively, the luminance value L can be obtained from the RGB pixel values ​​using the following equation (3) and the average value can be calculated. L = (3R + 6G + B) / 10 ... (3)

[0041] Figure 6 shows an example of a difference image. A difference image is an image obtained by calculating the absolute difference for each pixel between the image input in step S31 and the image obtained by performing the inverse Fourier transform in step S34.

[0042] Figure 6(A) is the difference image when the image is in focus, and (B) is when it is not in focus. When the image is in focus, the input image contains many high-frequency components, so many of these high-frequency components are cut out by the frequency component correction in step S33. When the inverse Fourier transform is applied to the result in step S34, the output image is a significantly blurred image of the input image. Therefore, the difference (absolute value of the difference) becomes large. On the other hand, when the image is not in focus, there are not many high-frequency components, so the difference becomes small. If we calculate the integral of the pixel values ​​of the difference images in this state (the sum of the difference values ​​of each pixel in the entire image), the value will be larger for the in-focus Figure 6(A), so we judge that the image is in focus when the degree of focus is greater.

[0043] In the example above, the cutoff frequency was determined according to the observation conditions and visual characteristics, but the cutoff frequency may also be calculated according to the MTF (Modulation Transfer Function) characteristics of the lens unit 2a. For example, if the maximum value of the MTF characteristics of the lens unit 2a is 0.8, the resolution of the lens will decrease, so the cutoff frequency also needs to be lowered. Therefore, in this case, it is appropriate to multiply it by 0.8. In other words, a value indicating the degree of focus may be calculated by performing a filter process according to the visual characteristics and shooting information (for example, lens characteristics such as MTF characteristics).

[0044] Thus, the focus calculation unit 13 functions as a focus calculation means that calculates a value indicating the degree of focus of objects included in an image based on the image data input to the input means and the shooting information at the time of image capture.

[0045] (Example of display of focus calculation results) Figure 7 shows an example of the display of the focus degree calculation result shown on the display device 3 in step S4. The display device 3 functions as a display means capable of displaying at least a value indicating the degree of focus. In Figure 7, the camera model name, lens type of lens unit 2a, TV (Time Value, shutter speed value), AV (Aperture Value, aperture value), ISO sensitivity, focal length (FL), and degree of focus are displayed in association with the image. By displaying the degree of focus, the state of focus can be quantified and easily determined. For example, among the six images shown in Figure 7, the image with the highest value for the degree of focus can be determined to be the most in focus.

[0046] In the example shown in Figure 7, the degree of focus is displayed along with the image, but the image does not need to be displayed. For example, the degree of focus could be displayed in conjunction with the file name of the image data. In other words, any information that can identify the image is acceptable, not limited to displaying the image itself; file names or other similar information may be used.

[0047] Furthermore, in the configuration shown in Figure 1, the image processing device 1 uses the display unit 15 as an output means to output image data and values ​​indicating the degree of focus to the display device 3. However, the output destination of the output means is not limited to the display. For example, the display unit 15 may be changed to an external interface such as a USB interface, and output may be sent to a storage device such as an HDD. Alternatively, the display unit 15 may be changed to a network interface, and output may be sent to a network such as the Internet.

[0048] (modified version) In the embodiment described above, the degree of focus calculation unit 13 calculated the degree of focus by applying a filter that takes visual characteristics into account from the frequency analysis results, but a machine learning model may be used instead. Examples of machine learning models that can be used include CNN (Convolutional Neural Network) and ViT (Vision Transformer).

[0049] When calculating the degree of focus using a machine learning model, either a regression problem or a classification problem will be used.

[0050] When using regression problems, for example, the parameters of the machine learning model are adjusted so that the output of an image that is in focus on the object is 1.0, and the output of an image that is out of focus on the object is 0.0. In this case, the correct values ​​can be divided into multiple stages based on the degree of focus, such as 1.0, 0.8, 0.6, 0.4, 0.2, and 0.0, and regression learning can be performed to achieve each of these values.

[0051] When using a classification problem, the model is trained to classify images that are in focus on the object as Class 1, and images that are out of focus as Class 0. While the example uses two classes, 1 and 0, for focusing, it is also possible to divide the degree of focus into multiple classes and train the model to classify images into each class.

[0052] Alternatively, different machine learning models can be trained for each pixel size, and the trained model used can be switched according to the image size during inference. Or, different machine learning models can be trained for each lens type of lens unit 2a, and the trained model used can be switched according to the lens type during inference. By doing so, the degree of focus can be calculated with greater accuracy.

[0053] According to this embodiment, by performing frequency analysis on image data and applying a filter that takes visual characteristics into consideration to calculate the degree of focus, it is possible to evaluate the degree of focus even if conditions such as pixel size and the type of lens of the lens unit 2a are different.

[0054] [Second Embodiment] Next, an image processing apparatus according to the second embodiment will be described. Note that the same configuration as the first embodiment will not be described, and the following description will focus on the differences from the first embodiment.

[0055] In the first embodiment, image data was frequency-analyzed, and a filter considering visual characteristics was applied to calculate the degree of focus. In this embodiment, a threshold surface considering pixel size and MTF characteristics is set for the calculation result, and the quality of the focus state is determined by where the calculated degree of focus is located on the surface. An example of determining the state of focus is to use a four-stage system: "◎: Very in focus," "〇: In focus," "△: Slightly out of focus," and "×: Out of focus." However, it is not limited to these four stages; it may also be determined using a two-stage system, such as "〇: In focus" and "×: Out of focus," or it may be used with three or five or more stages. In short, it is sufficient if the state of focus can be evaluated in multiple stages. Also, instead of symbols such as "〇," numbers or letters may be used.

[0056] Figure 8 shows an example of the configuration of a system including the image processing apparatus according to this embodiment. In the system shown in Figure 8, the image processing apparatus 1 is changed to image processing apparatus 1A compared to Figure 1. Image processing apparatus 1A has a focus determination unit 16 added to image processing apparatus 1.

[0057] The focus determination unit 16 determines the focus state based on the degree of focus calculated by the degree of focus calculation unit 13, as well as shooting information such as pixel size and lens characteristics. In other words, the focus determination unit 16 functions as a determination means that determines the focus state of the object based on a value indicating the degree of focus.

[0058] (System processing flow) Figure 9(A) is a flowchart showing the processing flow of the image processing device 1A shown in Figure 8.

[0059] Steps S1 to S3 are the same as in Figure 2, so their explanation is omitted. In step S5, which follows step S3, the focus determination unit 16 determines the focus status of the object detected in step S2. Details of the processing of the focus determination unit 16 will be described later.

[0060] Next, in step S6, the display unit 15 displays the image data, the calculation result of the degree of focus, and the judgment result performed in step S16 on the display device 3. An example of the display will be described later.

[0061] (Processing by the focus determination unit 16) Next, the processing of the focus determination unit 16 described above will be explained. The focus state is determined from the degree of focus calculated in step S3 of Figure 9(A), the lens characteristics of the lens unit 2a, and the pixel size. First, a threshold surface as shown in Figure 10 is created in advance. When creating it, it is good to use several representative images and visually determine four levels, for example, "◎ Very in focus," "〇 In focus," "△ Slightly out of focus," and "× Out of focus," and find the correspondence between the degree of focus calculated by the focus calculation unit 13 and the visual results. At this time, multiple pixel sizes and lens characteristics are used when creating the threshold surface. Then, surfaces are created that form the boundary points between ◎ and 〇, 〇 and △, and △ and ×, and these are the respective threshold surfaces. The created threshold surface data is stored in the data holding unit 14. As for lens characteristics, for example, MTF characteristics can be considered, and although they differ depending on the position in the image, for example, the value at the center is the average value. At this time, information such as MTF characteristics can be used from catalogs, etc., or they can be measured.

[0062] The process of the focus determination unit 16 (step S5) will be explained using the flowchart in Figure 9(B).

[0063] First, in step S41, the lens type is obtained from the Exif (Exchangeable image file format) information of the captured image input in step S1, and the lens characteristics are obtained based on the obtained lens type. The lens characteristics can be stored in the data storage unit 14 beforehand, for example.

[0064] Next, in step S42, the focus state is determined from the degree of focus, pixel size, and lens characteristics. For the determination, the threshold surface shown in Figure 10 is read from the data holding unit 14, and the degree of focus calculated in step S3, the pixel size extracted in step S2, and the lens characteristics acquired in step S41 are plotted in the space where the read threshold surface is set. Then, the judgment of ◎ or ○ described above is made based on which surfaces the plot position lies between. If it lies on a surface, the higher evaluation is selected.

[0065] (Example of display of focus determination result in this embodiment) Figure 11 shows an example of the display of the focus determination result in step S6. The display example shown in Figure 11 is generated by the display unit 15 and displayed on the display device 3. In Figure 11, in addition to the degree of focus shown in Figure 7, the determination result from the focus determination unit 16 is displayed. By displaying the determination result, it becomes easy to distinguish whether the image is in focus or not for each lens type and pixel size of the lens unit 2a, even if the degree of focus is the same. Alternatively, only the determination result may be displayed without showing the degree of focus.

[0066] Figure 11 shows images taken with lens X (three images in the top row and one in the bottom left) and images taken with lens Y (two images in the bottom row). For example, suppose the degree of focus (##.#) of the image taken with lens Y is lower than the degree of focus (##.#) of the image taken with lens X. However, by determining the focus state using a threshold surface that includes lens characteristics, as shown in Figure 11, even lens Y, which does not have a high degree of focus, may be judged as having a focus state of ◎.

[0067] Furthermore, although the lens characteristics were described as MTF characteristics in this embodiment, other characteristics, such as AV (aperture value) or focal length, may also be used. In addition, other shooting information such as TV (shutter speed value) and ISO sensitivity may be used.

[0068] Furthermore, in this embodiment, the threshold surface is set in a two-dimensional space of pixel size and lens characteristics, but it is not limited to this. The threshold may be set in a one-dimensional space of only pixel size or only lens characteristics. Alternatively, the threshold surface may be set in a three-dimensional space by adding information such as lens characteristics (AV, focal length) or shooting information (TV, ISO sensitivity) in addition to pixel size and lens characteristics (MTF characteristics), or the threshold surface may be set in a space of four or more dimensions. In other words, the multiple-stage determination threshold for determining the focus state may be changed according to the shooting information.

[0069] Alternatively, multiple focus calculation units may be provided, one of which calculates the focus based on frequency analysis, and another of which calculates the focus based on machine learning as described in the modified example of Embodiment 1. Each focus value may be handled in two-dimensional space, and a threshold surface may be set.

[0070] By using multiple pieces of information to determine the focus state in this way, it becomes possible to make a more stable determination than by using only a single piece of information.

[0071] According to this embodiment, the degree of focus is quantified by frequency analysis that takes visual characteristics into account, and the focus state is determined using the quantified degree of focus, the pixel size, and the MTF characteristics of the lens unit 2a. Therefore, it is possible to extract the image that is in the best focus regardless of the pixel size or the lens unit 2a.

[0072] [Third Embodiment] Next, an image processing apparatus according to the third embodiment will be described. Note that the same configuration as the first embodiment will not be described, and the following description will focus on the differences from the first and second embodiments.

[0073] In the first embodiment, quantification was performed using the degree of focus, and in the second embodiment, focus determination was performed for each pixel size and lens characteristic using a threshold surface. In this embodiment, we will describe an example of determining the degree of focus for the entire lens rather than for each lens, and an example of displaying only images taken with a specific lens.

[0074] (Example of display of focus determination result in this embodiment) Figure 12 shows an example of a judgment mode selection. Figure 12 is an example of a judgment mode menu screen displayed on the display device 3. As shown in Figure 12, it is desirable to be able to select judgment modes such as judging the entire image by absolute value (absolute judgment), displaying all images taken with all lenses based on judgment for each lens unit 2a, or displaying only images taken with specific lens types (lens X, lens Y). In other words, the display device 3 can switch the displayed content based on the shooting information.

[0075] Figure 13 shows an example of the display when the focus state is determined by an absolute value. In the example display in Figure 13, the degree of focus and the determination result are displayed on the display device 3 in correspondence with the image. When determining by an absolute value, the focus state is determined by fixing the MTF characteristic value to 1.0 on the threshold surface shown in Figure 10, regardless of the type of lens unit 2a used for shooting (to avoid lens type dependence). In other words, the determination conditions are changed when determining the focus state independently of the lens type of lens unit 2a and when determining the focus state for each lens unit 2a.

[0076] In this way, the focus can be determined using an absolute value that is independent of the type of lens unit 2a used for shooting. That is, the focus determination unit 16 outputs a focus state determination result that does not depend on the lens unit 2a. When determining using an absolute value, as shown in Figure 13, in the case of lens Y, for example, which has poor MTF characteristics, the degree of focus is not high, so even when it is in the best focus, the focus state determination may not be marked as "◎" when compared with lens X.

[0077] Figure 14 shows an example where only images taken with a specific lens unit 2a, for example, lens Y, are displayed. In this case, the focus state is determined according to the MTF characteristics of lens Y, so the evaluation and display can be performed within images taken with the same type of lens.

[0078] In other words, in this embodiment, it is possible to switch between displaying the focus state determination result, which is independent of the lens unit 2a, as shown in Figure 13, and the focus state determination result for each lens unit 2a, as shown in Figure 14.

[0079] Furthermore, "AllLens" in Figure 12 displays all judgment results simultaneously when there are judgment results for each lens type, as shown in Figure 11. In other words, "AllLens" adds judgment results to a display like that in Figure 11. In this case, as explained in Figure 11, for example, an image taken with lens Y, which has a lower degree of focus than lens X, may be displayed as ◎ or ○.

[0080] According to this embodiment, when multiple types of lens units 2a are used and multiple images are captured for each lens unit 2a, it is possible to switch between displaying the overall in-focus image and the images in focus for each lens unit 2a. Therefore, the desired image can be easily identified.

[0081] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0082] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. Embodiments of this disclosure include the following configurations, methods, and programs.

[0083] [Configuration 1] An input means into which the image data of the captured image is input, A focus calculation means calculates a value indicating the degree of focus of an object included in an image based on the image data input to the input means, based on the shooting information at the time the image was taken. An output means that outputs a value indicating the degree of focus calculated by the degree of focus calculation means, An image processing apparatus characterized by having [Configuration 2] The system further includes determination means for determining the focus state of the object based on the value indicating the degree of focus, The image processing apparatus according to Configuration 1, characterized in that the output means outputs the determination result of the determination means along with the value indicating the degree of focus, or outputs the determination result of the determination means instead of the value indicating the degree of focus. [Configuration 3] The image processing apparatus according to configuration 2, characterized in that the focus determination means determines the focus state in multiple stages, and the determination thresholds for the multiple stages are changed according to the shooting information. [Structure 4] An image processing apparatus according to any one of configurations 1 to 3, characterized in that it has an extraction means for detecting the object from the image and extracting a region on the image that includes the object. [Composition 5] The image processing apparatus according to configuration 4, characterized in that the focus degree calculation means converts the region extracted by the extraction means into a frequency domain, and performs a filter process on the converted frequency domain according to the visual characteristics to calculate a value indicating the degree of focus. [Composition 6] The image processing apparatus according to configuration 5, characterized in that the focus degree calculation means performs a filter process according to the visual characteristics and the shooting information to calculate a value indicating the degree of focus. [Composition 7] The image processing apparatus according to configuration 6, characterized in that the degree of focus calculation means calculates the degree of focus based on the difference between the image data showing the region extracted by the extraction means and the image data after the filtering process has been performed. [Structure 8] The image processing apparatus according to any one of configurations 1 to 7, characterized in that the focus degree calculation means calculates a value indicating the degree of focus using a machine learning model learned under learning conditions corresponding to the shooting information. [Composition 9] The image processing apparatus according to any one of configurations 1 to 8, characterized in that the object is a living organism or a specific part of a living organism. [Configuration 10] The image processing apparatus according to any one of configurations 1 to 9, characterized in that the aforementioned shooting information is either the number of pixels in the vertical and horizontal directions of the region, or the lens characteristics of the interchangeable lens unit used when shooting the image, or both. [Composition 11] The system further includes a display means capable of displaying, in association with at least information that can identify the image and a value indicating the degree of focus, The image processing apparatus according to any one of configurations 1 to 10, characterized in that the display means switches and displays the display content based on the shooting information. [Composition 12] The system further includes a display means capable of displaying information that can identify the image in association with the determination result, Multiple image data captured using multiple interchangeable lens units are input to the aforementioned input means. The determination means performs, for each of the multiple image data, a determination of the focus state that is independent of the interchangeable lens unit and a determination of the focus state for each of the interchangeable lens units, respectively. The image processing apparatus according to any one of configurations 2 to 10, characterized in that the display means is capable of switching between displaying the determination result of the focus state independent of the interchangeable lens unit and the determination result of the focus state for each interchangeable lens unit. [Method 1] An image processing method performed by an image processing device, The input process involves inputting image data of the captured images, A focus calculation step, which calculates a value indicating the degree of focus of an object included in an image based on the image data input in the input step, based on the shooting information at the time the image was taken, An output step which outputs a value indicating the degree of focus calculated in the degree of focus calculation step, An image processing method characterized by having the following features. [program] A program characterized by causing a computer to execute the image processing method described in Method 1. [Explanation of Symbols]

[0084] 1 Image Processing Device 11. Data Input / Output Section (Input Means) 12 Object detection unit (extraction means) 13 Focus degree calculation unit (focus degree calculation means) 14 Data storage unit 15 Display section (output means) 16 Focus determination section (determination means) 2 cameras 2a Lens Unit (Interchangeable Lens Unit) 3 Display device (display means) 4. External storage

Claims

1. An input means into which the image data of the captured image is input, A focus calculation means calculates a value indicating the degree of focus of an object included in an image based on the image data input to the input means, based on the shooting information at the time the image was taken. An output means that outputs a value indicating the degree of focus calculated by the degree of focus calculation means, An image processing apparatus characterized by having

2. The system further includes determination means for determining the focus state of the object based on the value indicating the degree of focus, The image processing apparatus according to claim 1, characterized in that the output means outputs the determination result of the determination means along with the value indicating the degree of focus, or outputs the determination result of the determination means in place of the value indicating the degree of focus.

3. The image processing apparatus according to claim 2, characterized in that the focus determination means determines the focus state in multiple stages, and the determination thresholds for the multiple stages are changed according to the shooting information.

4. The image processing apparatus according to claim 1, characterized by having an extraction means for detecting the object from the image and extracting a region on the image that includes the object.

5. The image processing apparatus according to claim 4, characterized in that the focus degree calculation means converts the region extracted by the extraction means into a frequency domain, and performs a filter process on the converted frequency domain according to the visual characteristics to calculate a value indicating the degree of focus.

6. The image processing apparatus according to claim 5, wherein the focus degree calculation means performs a filter process according to the visual characteristics and the shooting information to calculate a value indicating the degree of focus.

7. The image processing apparatus according to claim 6, characterized in that the degree of focus calculation means calculates the degree of focus based on the difference between the image data showing the region extracted by the extraction means and the image data after the filtering process has been performed.

8. The image processing apparatus according to claim 1, characterized in that the focus degree calculation means calculates a value indicating the degree of focus using a machine learning model learned under learning conditions corresponding to the shooting information.

9. The image processing apparatus according to claim 1, characterized in that the object is a living organism or a specific part of a living organism.

10. The image processing apparatus according to claim 1, characterized in that the aforementioned shooting information is either the number of pixels in the vertical and horizontal directions of the region, or the lens characteristics of the interchangeable lens unit used when the image was captured, or both.

11. The system further includes a display means capable of displaying, in association with at least information that can identify the image and a value indicating the degree of focus, The image processing apparatus according to claim 1, characterized in that the display means switches and displays the display content based on the shooting information.

12. The system further includes a display means capable of displaying information that can identify the image in association with the determination result, Multiple image data captured using multiple interchangeable lens units are input to the aforementioned input means. The determination means performs, for each of the multiple image data, a determination of the focus state that is independent of the interchangeable lens unit and a determination of the focus state for each of the interchangeable lens units, respectively. The image processing apparatus according to claim 2, characterized in that the display means is capable of switching between displaying the determination result of the focus state independent of the interchangeable lens unit and the determination result of the focus state for each interchangeable lens unit.

13. An image processing method performed by an image processing device, The input process involves inputting image data of the captured images, A focus calculation step, which calculates a value indicating the degree of focus of an object included in an image based on the image data input in the input step, based on the shooting information at the time the image was taken, An output step which outputs a value indicating the degree of focus calculated in the degree of focus calculation step, An image processing method characterized by having the following features.

14. A program characterized by causing a computer to execute the image processing method described in claim 13.

Images