Image processing device, method, and program for estimating the degree of focus on a subject.

Abstract

The present invention aims to estimate the degree of focus of a specific part of a body. [Solution] An image processing apparatus comprising: information acquisition means for acquiring an image, camera information associated with the image, position information of a plurality of specific parts of a subject captured in the image, and focus information of at least one of the plurality of specific parts; estimation means for outputting an estimated value of focus using the image, the camera information, and the position information of the plurality of specific parts; loss calculation means for calculating a loss based on the estimated value and the focus information; and weight update means for updating the parameters of a model constituting the estimation means based on the loss.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, method, and program for estimating the focus degree with respect to a subject.

Background Art

[0002] When selecting a photograph with appropriate focus on a subject, it takes a great deal of effort to extract an image with a high focus degree from a large number of images. In particular, when determining whether a specific part of the subject is in focus, even more effort is required, such as enlarging and displaying the image.

[0003] Therefore, techniques for estimating and evaluating the focus degree have been proposed. In Patent Document 1, a technique is disclosed in which high-frequency components of a series of consecutive images are extracted, and an image with small blur or shake is displayed based on the integral value thereof. Further, in Patent Document 2, a technique is disclosed in which a subject region and a background region are specified, and the focus degree is evaluated by taking the difference in the focus degree in each region.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

[0006] The method described in Patent Document 1 cannot accurately determine the degree of focus of a specific part of the subject. Furthermore, the method described in Patent Document 2 determines the degree of focus from a macroscopic perspective of the subject and background, making it difficult to determine whether a specific part of the subject is in focus or not.

[0007] This invention has been made in view of the above problems, and aims to estimate the degree of focus of a specific part. [Means for solving the problem]

[0008] One embodiment of the present invention is an image processing apparatus comprising: information acquisition means for acquiring a first image, camera information associated with the first image, position information of a plurality of specific parts of a subject captured in the first image, and focus information of at least one of the plurality of specific parts; estimation means for outputting an estimated value of focus using the first image, the camera information, and the position information of the plurality of specific parts; loss calculation means for calculating a loss based on the estimated value and the focus information; and weight update means for updating the parameters of a model constituting the estimation means based on the loss.

[0009] Another aspect of the present invention is an image processing apparatus comprising: an image; an information acquisition means for acquiring camera information associated with the image; a detection means for outputting positional information of a specific part of a subject captured in the image; and an estimation means for outputting an estimated value of the degree of focus using the image, the camera information, and the positional information of the specific part. [Effects of the Invention]

[0010] According to the present invention, it is possible to estimate the degree of focus of a specific part. [Brief explanation of the drawing]

[0011] [Figure 1]It is a block diagram showing a configuration example of a learning device 100 according to a first embodiment. [Figure 2] It is a diagram showing specific part information and a cut-out image according to a first embodiment. [Figure 3] It is a flowchart of a learning process of a learning model by a learning device 100 according to a first embodiment. [Figure 4] It is a diagram showing data acquired from a learning dataset according to a first embodiment. [Figure 5] It is a diagram showing a network constituting a focus degree estimation model according to a first embodiment. [Figure 6] It is a diagram explaining the effect of the invention according to a first embodiment. [Figure 7] It is a block diagram showing a configuration example of an inference device 200 according to a first embodiment. [Figure 8] It is a flowchart of an inference process by an inference device 200 according to a first embodiment. [Figure 9] It is a diagram showing a network constituting a focus degree estimation model according to a modification example of a first embodiment. [Figure 10] It is a block diagram showing a configuration example of a learning device 110 according to a second embodiment. [Figure 11] It is a flowchart of a learning process of a learning model by a learning device 110 according to a second embodiment.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0013] (First Embodiment) As an image processing apparatus according to the first embodiment, a learning apparatus 100 that learns a model for estimating the focus degree of a subject in an image will be described. A configuration example of the learning apparatus 100 according to the present embodiment is shown in the block diagram of FIG. 1.

[0014] The learning apparatus 100 includes an image information acquisition unit 101 that is an information acquisition means, a cut-out image creation unit 102 that is a cut-out image creation means, a focus part estimation unit 103 that is a focus part estimation means, a loss calculation unit 104 that is a loss calculation means, and a weight update unit 105 that is a weight update means.

[0015] The image information acquisition unit 101 acquires a learning image, camera information, and correct value information associated with the learning image from a learning data set.

[0016] The cut-out image creation unit 102 creates a cut-out image by cutting out a region based on specific part information from the learning image.

[0017] The focus part estimation unit 103 inputs the cut-out image and the camera information into a focus degree estimation model configured by a neural network, and outputs an estimated value of the obtained focus degree.

[0018] The loss calculation unit 104 calculates a loss based on the estimated value of the focus degree and the focus degree information associated with the learning image.

[0019] The weight update unit 105 updates the parameters of the model that constitutes the focus degree estimation unit 103 based on the loss calculated by the loss calculation unit 104.

[0020] First, the learning data set used in the learning apparatus 100 will be described. The learning data set used in the learning apparatus 100 is composed of a plurality of learning images, camera information associated with each learning image, and correct value information.

[0021] At least one subject is shown in each of the plurality of learning images, and the subject has a specific part described later.

[0022] The camera information associated with each training image is metadata such as the shooting conditions recorded when the training image was captured. In this embodiment, the lens's F-number, the lens's focal length, the distance from the camera to the subject (hereinafter referred to as the subject distance), the sensor size, and the number of pixels in the captured image are treated as camera information, but the camera information is not limited to these. Furthermore, at least one of these pieces of information may be used as camera information.

[0023] The correct answer information consists of two parts: specific part information of the subject and focus degree information. Specific parts are characteristic parts of the subject; in this embodiment, the left and right pupils and nose of a person are treated as specific parts. The reason for treating the pupils as specific parts is that they are often the focus point when photographing people. The reason for treating the nose as a specific part is that it is a part of a person's face where a difference in depth of field between the pupils and nose is likely to occur. Depending on the subject, in addition to the left and right pupils and nose of a person, the left and right pupils and nose of an animal, or the headlights of a car can also be treated as specific parts.

[0024] Specific area information is information that indicates the region of a specific part of the learning image. In this embodiment, specific area information is given as information of the coordinates of the center position of the specific part and a rectangular area with a predetermined width and height centered on those coordinates, and is assigned to the left and right pupils and the nose. Figure 2 is a schematic diagram showing the specific area information assigned to the subject in this embodiment. Rectangular area 301 in Figure 2 is the rectangular area assigned to the left pupil of the person, rectangular area 302 is the rectangular area assigned to the right pupil of the person, and rectangular area 303 represents the rectangular area assigned to the nose of the person.

[0025] The focus level information is a numerical value indicating the degree of focus on a specific area in a training image. The focus level information may be given as a continuous value or as discrete labels. In this embodiment, the value range of the focus level information is [0,1], and the focus level information is defined as a continuous value where a value closer to 0 indicates a higher degree of focus. The focus level information may be labeled visually by a human, or it may be quantified using image processing techniques or other methods.

[0026] In this embodiment, focus information is assigned to only one of the left or right pupils, and the pupil to which focus information is assigned is called the first specific area. The other pupil is called the second specific area, and the nose is called the third specific area, but focus information is not assigned to the second or third specific area. However, the specific area to which focus information is assigned is not limited to the first specific area; for example, focus information may be assigned to multiple specific areas. In the following description, when simply referred to as specific area information or focus information, it refers to the specific area information or focus information associated with the training image mentioned in the most recent sentence.

[0027] Next, the model learning process by the learning device 100 according to this embodiment will be described. The model learning process by the learning device 100 having the configuration example shown in Figure 1 will be described according to the flowchart in Figure 3. The model learning process by the learning device 100 according to this embodiment is performed by repeating the processes S101 to S105 shown in the flowchart of Figure 3 a desired number of times, and the learning process is terminated when the repeated processes have been performed a predetermined number of times.

[0028] In step S101, the image information acquisition unit 101 acquires one training image, along with camera information and ground truth value information associated with that training image, from the training dataset. At this time, the training image is acquired randomly from the training dataset.

[0029] Figure 4 shows an example of data obtained from the training dataset. As mentioned above, the training dataset contains data that combines training images, camera information, and ground truth information. The camera information associated with the training images includes the camera's F-number, focal length, subject distance, sensor size, and pixel count at the time the training image was acquired. In addition, the ground truth information includes specific area information 1, specific area information 2, specific area information 3, and focus accuracy information for the first specific area. Here, specific area information 1 is the specific area information for the first specific area, and the notation (3574,1824,258,59) indicates that the center coordinates of the specific area are (3574,1824), the width of the rectangular area is 258 pixels, and the height is 59 pixels. The notation of specific area information is not limited to this method; any notation that allows for the identification of the specific area is acceptable.

[0030] In step S102, the cropped image creation unit 102 creates a cropped image by extracting a region based on specific part information from the training image. The method for creating the cropped image in this embodiment will be explained with reference to Figure 1.

[0031] First, a cropped image corresponding to the first specific area is created. Let r be the length of the longest side of the rectangular region 301 assigned to the first specific area. The area from which the cropped image is extracted is a rectangular region 304 whose center coordinates are the same as those of the rectangular region 301, and whose sides are all r. The extracted rectangular region 304 is resized to a size that can be input into the focus estimation model of the focus estimation unit 103, which will be described later. Cropped images are created for the second and third specific areas in the same manner. In the following explanation, the cropped images corresponding to the first, second, and third specific areas will be referred to as the first cropped image, the second cropped image, and the third cropped image, respectively.

[0032] In step S103, the focus estimation unit 103 inputs each of the resized cropped images and the camera information associated with the training image into a focus estimation model composed of a neural network, and outputs an estimated value of the focus.

[0033] Figure 5 shows an example of the configuration of the focus estimation model in this embodiment. In the example configuration shown in Figure 5, the first cropped image 401, the second cropped image 402, and the third cropped image 403 are each input to the first network for feature extraction. That is, the first cropped image 401 is input to the first network 404, the second cropped image 402 is input to the first network 405, and the third cropped image 403 is input to the first network 406. In this embodiment, the first network 405 and the first network 406 are networks that have weight parameters common to the first network 404. For the first network, for example, a CNN (Convolutional Neural Network) or a Transformer can be used. In this embodiment, focus information is assigned to the first specific area, but not to the second and third specific areas. Therefore, this configuration is adopted because it is not possible to update the weight parameters appropriately for the second and third specific areas independently.

[0034] The features output from the first networks 404, 405, and 406 are combined into a single tensor along the channel dimension (among width, height, and channel dimension), and input to the second network 407. The second network outputs an estimate of the focus level, and like the first network, it can use a CNN or a Transformer. The focus level estimate output from the second network 407 has dimensions corresponding to the number of focus level information points. In this embodiment, the focus level estimate is one-dimensional, i.e., a scalar value.

[0035] The bias parameter 408 of the first layer of the first network is set to a value calculated based on camera information, and the camera information is input to the focus estimation model through this parameter. In this embodiment, the value of the bias parameter is set using the following equation (1).

[0036]

number

[0037] However, α and β are parameters with the same number of dimensions as the weight parameters of the layer into which the bias parameters are input, and D is the depth of field. Here, the depth of field is the range in front of and behind the point of focus in which the image appears to be in focus, and is calculated based on camera information using the following equations (2), (3), and (4).

[0038]

number

[0039]

number

[0040]

number

[0041] However, D f D is the front depth of field. r is the rear depth of field, F is the lens's F-number (aperture value), f is the lens's focal length, and L is the subject distance. δ is the acceptable circle of confusion diameter, and here we will use a value that is 1.5 times the size of one pixel on the image sensor. The size of one pixel on the image sensor can be obtained by dividing the sensor size by the number of pixels. In the example in Figure 4, the sensor size is 36 mm ÷ 6000 pixels, so the size of one pixel on the image sensor is calculated to be 6 μm. The acceptable circle of confusion diameter is 1.5 times the size of one pixel, so it is calculated to be 9 μm. In the example in Figure 4, D f 1.26, D r Since the value is 1.33, the depth of field D can be calculated to be 2.59.

[0042] If the values ​​used in equations (3) and (4) cannot be obtained from the camera, they may be manually registered as camera information for each image. The same bias parameters are also set for the first network 405 and the first network 406. In this embodiment, camera information is input as the bias parameter of the first layer of the focus estimation model, but the method of inputting camera information is not limited to this. Specifically, camera information may be input as the bias parameter of another layer constituting the focus estimation model, or it may be input to the network together with the cropped image as a map of the same size as the cropped image. In addition, for the image input to the focus estimation model, the second and third cropped images may not be input, and the training image may be directly input instead of the first cropped image. In that case, specific part information may be input to the focus estimation model, and the region information of the specific part may be referenced within the focus estimation model. Alternatively, an image obtained by cropping a part of the training image to include the region of the specific part recorded in the specific part information may be input.

[0043] In S104, the loss calculation unit 104 calculates the loss based on the estimated focus level output from the focus level estimation model and the focus level information associated with the training image. In this embodiment, the squared error function shown in equation (5) below is used as the loss function.

[0044]

number

[0045] However, y is the estimated focus level output by the focus estimation model, and t is the focus level information recorded in the ground truth information. The sum for i is calculated for each piece of focus level information. In this embodiment, since focus level information is assigned only to the first specific part, the sum is not calculated. If the focus level information is discrete labels, for example, the cross-entropy error function shown in equation (6) below may be used.

[0046]

number

[0047] In S105, the weight update unit 105 updates the parameters of the model constituting the focus estimation unit 103. The weight parameters can be updated, for example, using the backpropagation method. At this time, the bias parameters α and β in equation (3) may be updated simultaneously with the weight parameters. As described above, in this embodiment, focus information is not assigned to the second specific area and the third specific area, and it is not possible to update the weight parameters appropriately on their own. Therefore, in this embodiment, only the weight parameters of the first network 404 and the second network 407 are updated, and the weight parameters of the first network 405 and the first network 406 are not updated.

[0048] In the above explanation, one training data point was acquired in a single training process, but it is also possible to acquire multiple training data points at once and perform the training process. This method of acquiring multiple training data points is called batch processing. When performing batch processing, processes S101 to S104 are performed for each acquired training data point, and the parameter update process in S105 is performed using the average of the losses calculated for each training data point.

[0049] Thus, in the model learning process according to this embodiment, the focus estimation model is learned using not only images of the specific area that is the target of focus, but also optical characteristic information obtained from images and camera information of multiple areas where a depth difference with the specific area is likely to occur. This makes it possible to incorporate images of each area, optical characteristic information, and the actual focus level as clues for estimating the focus level.

[0050] A concrete example of a focusing pattern will be explained using Figure 6, a schematic diagram showing how an imaging device captures a human face. In Figure 6, 501 is the first specific part (right pupil), 502 is the second specific part (left pupil), and 503 is the third specific part (nose). It is assumed that the first specific part 501 and the third specific part 503, which are within the depth of field D, are in focus. 504 is the imaging device.

[0051] Since the subject, the human face, is angled relative to the imaging device 504, the second specific part 502 falls outside the focus range when the depth of field D is within the range shown in Figure 6. Therefore, the degree of focus of the second specific part 502 cannot be a high value. In this invention, such focus patterns are used as training data for the focus estimation model. By performing training with such training data, a model can be constructed that can perform highly accurate focus estimation by recognizing minute differences in focus. In other words, it is possible to estimate the degree of focus of a specific part. Furthermore, even if only focus information for one specific part is available, the focus estimation can be trained using images of other specific parts for which focus information is not available, not just that specific part.

[0052] Next, we will describe an image processing device that functions as an inference device for estimating the degree of focus using the model trained using the method described above. The images handled by this inference device contain subjects of the same or similar types as those in the training images used to train the degree of focus estimation model, and these subjects are assumed to have the same specific parts as the specific part information recorded in the ground truth information of the training dataset.

[0053] Figure 7 shows a block diagram of an example configuration of the inference device 200 according to this embodiment. The inference device 200 includes an image information acquisition unit 101 which is an information acquisition means, a specific part detection unit 201 which is a detection means, a cropped image creation unit 102 which is a cropped image creation means, and a focus area estimation unit 103 which is a focus area estimation means, and is connected to a display unit 300.

[0054] The image information acquisition unit 101 acquires the inferred image to be used for focus estimation and the camera information associated with the inferred image.

[0055] The specific area detection unit 201 inputs the inferred image into the specific area detection model and outputs location information of the specific area.

[0056] The extracted image creation unit 102 creates an extracted image by extracting a region based on specific part information from the inferred image.

[0057] The focus estimation unit 103 inputs the cropped image and camera information into the focus estimation model and outputs an estimated value of the focus degree.

[0058] The display unit 300 is a screen display panel.

[0059] The inference process performed by the inference device 200 having the configuration example shown in Figure 7 will be explained according to the flowchart in Figure 8.

[0060] In S201, the image information acquisition unit 101 acquires the inferred image to be used for focus estimation and the camera information associated with the inferred image. The inferred image may be obtained by the imaging device, or it may be output from a memory that holds the image acquired by the imaging device.

[0061] In S202, the specific area detection unit 201 inputs the inferred image to the specific area detection model and outputs location information of the specific area. The specific area detection model is configured to detect specific areas of the same type as the specific area information recorded in the ground truth information of the training dataset, or a model that has been pre-trained is used. An example of a specific area detection model is the model described in Non-Patent Literature 1. The location information of the specific area is output for the first specific area, the second specific area, and the third specific area, and is in the same format as the specific area information recorded in the ground truth information of the training dataset. In this explanation, if the specific area detection model fails to output any of the first, second, or third specific areas, the inference process is stopped without performing further processing. Alternatively, the system may be configured to perform the inference process if at least the first specific area is detected among the first to third specific areas.

[0062] In S203, the extracted image creation unit 102 creates an extracted image by extracting a region based on specific part information from the inferred image. The method of extracting the image is the same as the image extraction process in the learning device described above.

[0063] In S204, the focus estimation unit 103 inputs the cropped image generated in S203 and camera information into the focus estimation model and outputs an estimated value of the focus. The model used as the focus estimation model is the model learned by the learning device described above. The input image may be a cropped image created by the user through manual cropping operations, instead of a cropped image created by the cropped image creation unit. Furthermore, if, in the learning process, the learning image and specific part information were directly input into the focus estimation model instead of a cropped image, and the model referenced the region information of the specific part internally, the same model may be used in this process as well.

[0064] In S205, the estimation device 200 displays at least one of the following on the display panel, which is the display unit 300: the inferred image, camera information, specific part information, cropped image, and the estimated focus level. The display panel is composed of a liquid crystal panel or an organic EL panel, and the user can visually confirm the information such as the inferred image and camera information input to the inferred device and the outputted estimated focus level.

[0065] (Modification of the first embodiment) As a modification of the learning device according to the first embodiment, we will describe a case in which images of a specific body part taken at different times are input to the focus depth model, and the focus depth estimation model is trained using a loss function based on the difference between the focus depths of each image. The configuration of the learning device and the learning process in this modification are the same as in the first embodiment, except for the configuration of the focus depth estimation model provided in the focus depth estimation unit and the loss function calculated in the loss calculation unit. Therefore, the configuration of the learning device and the learning process in this modification will be explained using Figures 2 and 3, respectively. Content common to the first embodiment will be omitted as appropriate, and the differences from the first embodiment will be mainly explained.

[0066] The learning process by the learning device according to this modified example will be explained in accordance with the flowchart in Figure 3.

[0067] In step S101, the image information acquisition unit 101 acquires two training images taken at different times from the training dataset, along with camera information and ground truth value information associated with each training image. The two acquired training images are designated as training image A, which is the first image, and training image B, which is the second image.

[0068] In S102, the cropped image creation unit 102 creates cropped images by extracting regions based on specific part information from each training image. The cropped image created from training image A is designated as cropped image A, and the cropped image created from training image B is designated as cropped image B.

[0069] In step S103, the focus estimation unit 103 inputs each cropped image and camera information into a focus estimation model composed of a neural network and outputs the obtained estimated focus value. In this modified example, in addition to the estimated focus value estimated in Embodiment 1, the difference between the estimated focus value of training image A and the estimated focus value of training image B is estimated.

[0070] Figure 9 shows an example of the configuration of the focus estimation model in this modified example. In this configuration example, the first cropped image A, 401, extracted from the training image A, is input to the first network 404. Similarly, the second cropped image A, 402, is input to the first network 405, and the third cropped image A, 406, is input to the first network 406.

[0071] The features output from the first network 404, the first network 405, and the first network 406 are combined into a single tensor along the channel dimension (among width, height, and channel dimension), and input to the second subsystem 409. The second subsystems 409 and 410 are obtained by dividing the second network 407 in the first embodiment into two subsystems separated by a predetermined layer. This configuration is adopted in order to utilize the intermediate features obtained from the second network. The second subsystem 410 outputs an estimated value of the degree of focus using the cropped image A as input, similar to the first embodiment.

[0072] The first cropped image B411, extracted from the training image B, is input to the first network 414; the second cropped image B412 is input to the first network 415; and the third cropped image B416 is input to the first network 416. The features output from the first network 414, the first network 415, and the first network 416 are combined along the channel dimension and input to the second subsystem 417. The features output from the second subsystem 409 and the second subsystem 417 are combined along the channel dimension and input to the third network 418.

[0073] The third network 418 outputs the difference between the focus level estimated from each specific area of ​​training image A and the focus level estimated from each specific area of ​​training image B as the focus difference estimate. The focus difference estimate has dimensions corresponding to the number of focus level information points. In this modified example, since there is one focus level information point for each training image, the dimension of the focus difference estimate is 1-dimensional, i.e., the focus difference estimate is a scalar value.

[0074] Although the bias parameter 408 is omitted in Figure 9, it is assumed that the bias parameter 408 is input to the focus estimation model in the same manner as in the first embodiment. Furthermore, the first network 414, the first network 415, and the first network 416 are networks that share weight parameters with the first network 404. The second subset network 417 is assumed to be a network that shares weight parameters with the second subset network 409.

[0075] In this modified version, the loss is calculated using a loss function that adds the sum of squared errors functions shown in equations (7) and (8) to the loss function shown in equation (1).

[0076]

number

[0077]

number

[0078] However, z is the estimated difference in focus output by the focus estimation model, and t a This is the focus information recorded in the ground truth information of training image A, t b is the focus information recorded in the ground truth information of training image B. The sum for i is calculated for each focus information. In this modified example, since focus information is assigned only to the first specific area, the sum is not calculated. w is a weighting parameter, and a desired real value should be set. If the focus information is discrete labels, for example, the cross-entropy error function shown below may be used.

[0079]

number

[0080] As a result, the focus estimation model learns to estimate the focus of a specific area in training image A, as well as the difference in focus between training image A and training image B for each specific area.

[0081] Thus, in the learning process described in this modified version, the model learns to estimate the difference in the degree of focus of a specific area in images taken at two different times. This makes it possible to estimate the degree of focus by considering the relative difference in the degree of focus of a specific area in the two images. Compared to training the model using only one image, this method enables the estimation of the degree of focus with higher accuracy, recognizing even minute differences in the degree of focus between consecutive images.

[0082] (Second Embodiment) In the first embodiment, the learning device used specific region information included in the training dataset to train the model. In this embodiment, instead of specific region information included in the training dataset, information obtained from a detector of a specific region is used. Explanations common to the first embodiment will be omitted, and the differences from the first embodiment will be explained mainly.

[0083] The learning device 110 according to this embodiment includes a specific part detection unit that detects a specific part, and performs learning processing using the specific part information detected by the specific part detection unit. An example of the configuration of the learning device according to this embodiment is shown in the block diagram of Figure 10, and a flowchart is shown in Figure 11.

[0084] The learning device 110 includes an image information acquisition unit 101, a specific area detection unit 106, a cropped image creation unit 102, a focus estimation unit 103, a loss calculation unit 104, and a weight update unit 105. The specific area detection unit 106 is a detection means that detects a specific area of ​​the input learning image and outputs specific area information.

[0085] In S101, the image information acquisition unit 101 acquires one training image, camera information associated with the training image, and correct value information from the training dataset.

[0086] In S106, the specific area detection unit 106 inputs the training image to the specific area detection model and outputs specific area information. The specific area detection model is configured to detect specific areas that include specific areas to which focus information has been assigned in the ground truth information of the training dataset, or a pre-trained model is used. Furthermore, the specific area information output by the specific area detection model is output in the same format as the specific area information recorded in the ground truth information of the training dataset in the first embodiment.

[0087] Furthermore, only a portion of the specific area information output by the specific area detection model may be used in subsequent processing. For example, information on specific areas located beyond a predetermined range from specific areas to which focus information has been assigned in the training dataset may not be used in subsequent processing. Unused specific area information may be deleted. Also, instead of obtaining all specific area information from the specific area detection model, some specific area information may be obtained from the ground truth information in the training dataset.

[0088] The processing from S102 onward is the same as in the first embodiment, so the explanation will be omitted.

[0089] Thus, in the learning device according to this embodiment, the learning process of the model is performed using the specific part information detected by the specific part detection model. This makes it possible to acquire and learn from other specific parts located around the specific part for which the degree of focus is to be estimated, even when the position and size information of a specific part is not recorded in the learning dataset.

[0090] Furthermore, the specific part detection unit 201 of the inference device 200 according to this embodiment shall use the same model as the specific part detection model constituting the specific part detection unit 106, or a model that is set or pre-trained to detect the same type of specific part.

[0091] (Other embodiments) Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or CPU or MPU, etc.) of that system or device read and execute the program.

[0092] It should be noted that the above embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0093] Furthermore, this disclosure includes the following components.

[0094] (Composition 1) Information acquisition means for acquiring a first image, camera information associated with the first image, positional information of multiple specific parts of a subject captured in the first image, and focus information of at least one of the multiple specific parts. An estimation means that outputs an estimated value of the degree of focus using the first image, the camera information, and the positional information of the plurality of specific parts, A loss calculation means that calculates the loss based on the estimated value and the focus information, An image processing apparatus characterized by having a weight update means for updating the parameters of the model constituting the estimation means based on the loss.

[0095] (Configuration 2) The information acquisition means further acquires a second image, camera information associated with the second image, positional information of multiple specific parts of the subject captured in the second image, and focus information of at least one of the multiple specific parts. The estimation means outputs an estimated value of the difference between the degree of focus for the first image and the degree of focus for the second image. The image processing apparatus according to configuration 1, wherein the loss calculation means calculates the loss based on the estimated value and the difference between the focus information associated with the first image and the focus information associated with the second image.

[0096] (Composition 3) The system further includes a cropped image creation unit that creates cropped images corresponding to the positional information of the multiple specific parts from the first image, The image processing apparatus according to configuration 1 or 2, wherein the estimation means is characterized by receiving the cropped image as input.

[0097] (Composition 4) An information acquisition means for acquiring an image and camera information associated with the said image, A detection means that outputs location information of a specific part of the subject shown in the aforementioned image, An image processing apparatus characterized by including an estimation means that outputs an estimated value of the degree of focus using the aforementioned image, the camera information, and the position information of the specific part.

[0098] (Composition 5) The detection means outputs location information of multiple specific parts, The image processing apparatus according to claim 4, characterized in that the estimation means outputs an estimated value of the degree of focus using the positional information of the plurality of specific parts.

[0099] (Composition 6) The image processing apparatus according to configuration 4 or 5, characterized by having a display unit that displays the aforementioned image and the estimated value of the degree of focus.

[0100] (Composition 7) The image processing apparatus according to any one of configurations 1 to 6, characterized in that the camera information is depth of field.

[0101] (Composition 8) The image processing apparatus according to configuration 7, characterized in that the depth of field is determined from the F-number, the focal length, and the subject distance.

[0102] (Composition 9) The estimation means is composed of a neural network. An image processing apparatus according to any one of configurations 1 to 8, characterized in that one of the layers constituting the neural network receives the input of the camera information.

[0103] (Composition 10) An image processing apparatus according to any one of configurations 1 to 3, characterized by having a detection means for outputting location information of the specific part.

[0104] (Composition 11) A program that causes a computer to function as an image processing device as described in any of configurations 1 to 10.

[0105] (Composition 12) An information acquisition step that acquires an image, camera information associated with the image, positional information of multiple specific parts of a subject captured in the image, and focus information of at least one of the multiple specific parts. An estimation step that outputs an estimated value of the degree of focus using the aforementioned image, the camera information, and the positional information of the plurality of specific parts, A loss calculation step that calculates the loss based on the estimated value and the focus information, A learning method for an image processing device, comprising: a weight update step for updating the parameters of a model used in the estimation step based on the loss; and a learning method for an image processing device.

[0106] (Composition 13) An information acquisition process that acquires an image and camera information associated with the image, A detection step that outputs location information of a specific part of the subject shown in the aforementioned image, An image processing method characterized by comprising an estimation step of outputting an estimated value of the degree of focus using the aforementioned image, the camera information, and the position information of the specific part.

[0107] (Composition 14) The image processing method according to configuration 12 or 13, characterized in that the camera information is depth of field.

[0108] (Composition 15) The image processing method according to configuration 14, characterized in that the depth of field is determined from the F-number, focal length, and subject distance.

[0109] (Composition 16) The aforementioned estimation process is performed using a neural network. The image processing method according to any one of configurations 12 to 15, characterized in that one of the layers constituting the neural network receives the input of the camera information.

[0110] (Composition 17) The image processing method according to configuration 12, characterized by having a detection step that outputs location information of the specified part.

[0111] (Composition 18) The detection step outputs location information of multiple specific parts, The image processing method according to configuration 13, characterized in that the estimation step outputs an estimated value of the degree of focus using the positional information of the plurality of specific parts.

[0112] (Composition 19) A program that causes a computer to perform any of the methods described in configurations 12 through 18. [Explanation of symbols]

[0113] 100 Learning Devices 101 Image Information Acquisition Unit 103 Focus degree estimation section 104 Loss calculation section 105 Weight update section

Claims

1. Information acquisition means for acquiring a first image, camera information associated with the first image, positional information of multiple specific parts of a subject captured in the first image, and focus information of at least one of the multiple specific parts. An estimation means that outputs an estimated value of the degree of focus using the first image, the camera information, and the positional information of the plurality of specific parts, A loss calculation means that calculates the loss based on the estimated value and the focus information, An image processing apparatus characterized by having a weight update means for updating the parameters of the model constituting the estimation means based on the loss.

2. An information acquisition means for acquiring an image and camera information associated with the said image, A detection means that outputs location information of a specific part of the subject shown in the aforementioned image, An image processing apparatus characterized by including an estimation means that outputs an estimated value of the degree of focus using the aforementioned image, the camera information, and the position information of the specific part.

3. The information acquisition means further acquires a second image, camera information associated with the second image, positional information of multiple specific parts of the subject captured in the second image, and focus information of at least one of the multiple specific parts. The estimation means outputs an estimated value of the difference between the degree of focus for the first image and the degree of focus for the second image. The image processing apparatus according to claim 1, characterized in that the loss calculation means calculates the loss based on the estimated value and the difference between the focus information associated with the first image and the focus information associated with the second image.

4. The system further includes a cropped image creation unit that creates cropped images corresponding to the positional information of the multiple specific parts from the first image, The image processing apparatus according to claim 1, characterized in that the estimation means receives the cropped image as input.

5. The image processing apparatus according to claim 2, characterized in that it has a display unit that displays the aforementioned image and the estimated value of the degree of focus.

6. The image processing apparatus according to claim 1 or 2, characterized in that the camera information is depth of field.

7. The image processing apparatus according to claim 6, characterized in that the depth of field is determined from the F-number, the focal length, and the subject distance.

8. The estimation means is composed of a neural network. The image processing apparatus according to claim 1 or 2, characterized in that one of the layers constituting the neural network receives the input of the camera information.

9. The image processing apparatus according to claim 1, characterized in that it has a detection means for outputting location information of the specific part.

10. The detection means outputs location information of multiple specific parts, The image processing apparatus according to claim 2, characterized in that the estimation means outputs an estimated value of the degree of focus using the positional information of the plurality of specific parts.

11. A program that causes a computer to function as an image processing device according to claim 1 or 2.

12. An information acquisition step that acquires an image, camera information associated with the image, positional information of multiple specific parts of a subject captured in the image, and focus information of at least one of the multiple specific parts. An estimation step that outputs an estimated value of the degree of focus using the aforementioned image, the camera information, and the positional information of the plurality of specific parts, A loss calculation step that calculates the loss based on the estimated value and the focus information, A learning method for an image processing device, comprising: a weight update step for updating the parameters of a model used in the estimation step based on the loss; and a learning method for an image processing device.

13. An information acquisition process that acquires an image and camera information associated with the image, A detection step that outputs location information of a specific part of the subject shown in the aforementioned image, An image processing method characterized by comprising an estimation step of outputting an estimated value of the degree of focus using the aforementioned image, the camera information, and the position information of the specific part.

14. The image processing method according to claim 12 or 13, characterized in that the camera information is depth of field.

15. The image processing method according to claim 14, characterized in that the depth of field is determined from the F-number, the focal length, and the subject distance.

16. The aforementioned estimation process is performed using a neural network. The image processing method according to claim 12 or 13, characterized in that one of the layers constituting the neural network receives the input of the camera information.

17. The image processing method according to claim 12, characterized in that it has a detection step of outputting location information of the specific part.

18. The detection step outputs location information of multiple specific parts, The image processing method according to claim 13, characterized in that the estimation step outputs an estimated value of the degree of focus using the positional information of the plurality of specific parts.

19. A program that causes a computer to perform the method described in claim 12 or 13.

Images