Image classification device, image classification method, and program

Abstract

This image classification device comprises: a contribution estimation unit that estimates the contribution of each partial image included in an input image to the classification of the input image; an image selection unit that preferentially selects a partial image with a high contribution; and an image classification unit that classifies the input image including the selected partial image by classifying the selected partial image. The image classification device may further comprise: a magnification estimation unit that estimates a shooting magnification of an input image; an image extraction unit that extracts a partial image from an input image at a size corresponding to the shooting magnification; and a size standardization unit that standardizes the sizes of partial images. The contribution estimation unit may estimate the contribution of each partial image of standardized size.

Description

Image classification device, image classification method, and program

[0001] The present invention relates to an image classification device, an image classification method, and a program.

[0002] The captured images may be classified. For example, a machine learning model may be used to classify bone marrow biopsy images to determine whether the person from whom the bone marrow tissue was taken has acute myeloid leukemia (AML) (see Non-Patent Document 1).

[0003] Eckardt, JN., Middeke, JM, Riechert, S. et al. "Deep learning detects acute myeloid leukemia and predicts NPM1 mutation status from bone marrow smears," Leukemia 36, ​​111-118 (2022).

[0004] However, constructing a training sample containing multiple training images (training data) for use in machine learning requires significant costs. For example, training samples must be prepared for detecting components (e.g., blood cells) captured in the training images, estimating the properties of the components detected in the training images, and finally classifying the training images.

[0005] In particular, the task of detecting components captured in an image requires a very large cost (labor) to construct a training sample. When a large number of components are captured in the training image, the required cost is proportional to the number of components captured in the training image. Thus, there is a problem in that input images cannot be classified unless a large cost is incurred to construct a training sample for machine learning.

[0006] In view of the above circumstances, the present invention aims to provide an image classification device, an image classification method, and a program that are capable of classifying input images while reducing the cost required to construct learning samples for machine learning.

[0007] One aspect of the present invention is an image classification device that includes a contribution estimation unit that estimates the contribution of each partial image included in an input image to the classification of the input image, an image selection unit that preferentially selects partial images with a high contribution, and an image classification unit that classifies the input image including the selected partial images by classifying the selected partial images.

[0008] One aspect of the present invention is an image classification method executed by an image classification device, which includes the steps of estimating the contribution of each partial image included in an input image to the classification of the input image, preferentially selecting the partial images with the highest contribution, and classifying the input image including the selected partial images by classifying the selected partial images.

[0009] One aspect of the present invention is a program for causing a computer to execute the steps of: estimating the contribution of each partial image contained in an input image to the classification of the input image; preferentially selecting the partial images with the highest contribution; and classifying the input image including the selected partial images by classifying the selected partial images.

[0010] The present invention makes it possible to classify input images even when the cost required for constructing learning samples for machine learning is reduced.

[0011] 1 is a diagram illustrating an example of the configuration of an image classification system in an embodiment; FIG. 2 is a flowchart illustrating an example of the operation of an image classification device in a learning process in an embodiment; FIG. 3 is a flowchart illustrating an example of the operation of an image classification device in an inference process in an embodiment; FIG. 4 is a diagram illustrating an example of a receiver operating characteristic curve of an image classification method according to a known technique; FIG. 5 is a diagram illustrating an example of a receiver operating characteristic curve of an image classification method in an embodiment; and FIG. 6 is a diagram illustrating an example of the hardware configuration of an image classification device in an embodiment.

[0012] An embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a diagram showing an example configuration of an image classification system 1 according to the embodiment. In the learning process, the image classification system 1 is a system that performs machine learning of each model (magnification estimator, contribution estimator, and image classifier) ​​using training images. In the inference process after the learning process, the image classification system 1 is a system that classifies an input image (inference image) using each trained model.

[0013] The image classification system 1 includes a storage device 2 and an image classification device 3. The storage device 2 is, for example, a server device. The image classification device 3 includes an acquisition unit 31, a magnification estimation unit 32, an image extraction unit 33, a size unification unit 34, a contribution estimation unit 35, an image selection unit 36, and an image classification unit 37. The image classification device 3 may include the storage device 2.

[0014] First, the image classification system 1 during the learning process will be described. The storage device 2 stores learning samples. The learning samples include input images (learning images) during the learning process. The learning samples (teacher data) may further include learning labels (correct answer labels) that represent correct answers. The learning samples may further include additional information such as information on the shooting magnification (lens magnification) of the learning images (hereinafter referred to as "learning shooting magnification information").

[0015] The image classification device 3 in the learning process is a device that executes machine learning of each model (magnification estimator, contribution estimator, and image classifier). The acquisition unit 31 acquires learning samples from the storage device 2. The acquisition unit 31 outputs learning images to the magnification estimation unit 32, the image extraction unit 33, and the size unification unit 34. The acquisition unit 31 may output the learning images to the image classification unit 37. The acquisition unit 31 may output shooting magnification information for learning to the magnification estimation unit 32.

[0016] The magnification estimation unit 32 (magnification estimator) estimates the photographing magnification of the learning image (input image). The magnification estimation unit 32 estimates the photographing magnification of the learning image, for example, based on predetermined magnification visual estimation data. In microscopes, blood cells and the like are often photographed using a lens selected from lenses with various magnifications. Therefore, the magnification estimation unit 32 estimates which lens with the selected magnification has been selected.

[0017] The magnification estimation unit 32 compares the estimated photographing magnification with the accurate magnification indicated by the learning photographing magnification information. That is, the magnification estimation unit 32 calculates the difference (magnification estimation loss) between the estimated photographing magnification and the magnification predetermined in the learning label (correct label). The magnification estimation unit 32 updates the coefficients (model parameters) of the magnification estimator so as to reduce the magnification estimation loss. The learning (coefficient update) of the magnification estimation unit 32 is performed repeatedly.

[0018] When the learning of the magnification estimation unit 32 is completed, the image extraction unit 33 extracts one or more partial images from the training image (input image) at a size (cropping size) corresponding to the estimated shooting magnification. If the estimated shooting magnification is, for example, 10x, each partial image is extracted from random coordinates in the training image at a size of, for example, "64 x 64." If the estimated shooting magnification is, for example, 20x, each partial image (cropped image) is extracted from random coordinates in the training image at a size of, for example, "128 x 128." If the estimated shooting magnification is, for example, 40x, each partial image is extracted from random coordinates in the training image at a size of, for example, "256 x 256." If the estimated shooting magnification is, for example, 100x, each partial image is extracted from random coordinates in the training image at a size of, for example, "640 x 640."

[0019] The size unification unit 34 unifies (resizes) the size of each partial image extracted from the training image to, for example, 64 x 64. This makes the apparent size of the components captured in each partial image roughly uniform, facilitating the learning of image features. Each partial image is labeled with its contribution to the classification of the training image.

[0020] The contribution estimation unit 35 (contribution estimator) estimates the contribution of each partial image included in the learning image (input image) to the classification of the learning image. Here, the contribution estimation unit 35 may estimate the average value of the contributions at multiple coordinates in the partial image as the contribution (average contribution) of that partial image.

[0021] The partial images are labeled so that partial images with a higher contribution to the classification of the training image are preferentially extracted from the training image. Here, important components in the image do not need to be explicitly detected, and the feature quantities of important components in the image do not need to be explicitly extracted. Therefore, labeling to assign a contribution level to each partial image can be performed at a lower cost than labeling to detect components in the training image. Note that such labeling may not be necessary.

[0022] The contribution estimation unit 35 calculates the difference (contribution estimation loss) between the estimated contribution and a predetermined contribution (labeled contribution). The contribution estimation unit 35 updates the coefficients (model parameters) of the contribution estimator so as to reduce the contribution estimation loss. The learning (coefficient update) of the contribution estimation unit 35 is repeatedly performed.

[0023] As described above, learning by the magnification estimation unit 32 is completed before learning by the contribution estimation unit 35. Furthermore, every time learning by the contribution estimation unit 35 is executed, before that execution, magnification estimation by the magnification estimation unit 32, image extraction by the image extraction unit 33, and size unification by the size unification unit 34 are executed as described above.

[0024] When the learning of the contribution estimation unit 35 is completed, the image selection unit 36 ​​focuses on partial images that are important for classification by the image classification unit 37 based on the estimated contribution. That is, the image selection unit 36 ​​preferentially selects one or more partial images (important partial images) with high estimated contributions from the learning images. For example, the image selection unit 36 ​​determines partial images with a contribution equal to or greater than a predetermined contribution threshold as partial images with high contributions. The contribution threshold is, for example, predetermined randomly from real numbers between 0 and 1.

[0025] The image classification unit 37 (image classifier) ​​classifies the selected partial image, thereby classifying the training images including the selected partial image. For example, if the partial image is classified into the image group for cases of acute myeloid leukemia, the training images including the partial image are also classified into the image group for cases of acute myeloid leukemia.

[0026] The image classification unit 37 calculates the difference (classification estimation loss) between the classification result of the training image and the classification result predetermined for the training label (correct label). The image classification unit 37 updates the coefficients (model parameters) of the classification estimator so as to reduce the classification estimation loss. The learning (coefficient update) of the image classification unit 37 is performed repeatedly.

[0027] Next, the image classification system 1 for the inference process will be described. The storage device 2 stores input images (images for inference) used in the inference process. The storage device 2 may store information on the magnification of the images for inference (hereinafter referred to as "image magnification information for inference") for each image for inference.

[0028] The image classification device 3 in the inference process is a device that classifies an input image using each trained model (magnification estimator, contribution estimator, and image classifier). The acquisition unit 31 acquires an input image (image for inference) in the inference process from the storage device 2. The acquisition unit 31 outputs the image for inference to the magnification estimation unit 32, the image extraction unit 33, and the size unification unit 34. The acquisition unit 31 may output the image for inference to the image classification unit 37. The acquisition unit 31 may output information on the shooting magnification of the image for inference (hereinafter referred to as "shooting magnification information for inference") to the magnification estimation unit 32.

[0029] The magnification estimation unit 32 (magnification estimator) estimates the photographing magnification of the inference-use image (input image). The magnification estimation unit 32 estimates the photographing magnification of the inference-use image based on, for example, predetermined magnification visual estimation data.

[0030] The image extraction unit 33 extracts partial images from the inference image (input image) at a size (cropping size) according to the estimated imaging magnification. If the estimated imaging magnification is, for example, 10x, each partial image is extracted from random coordinates in the inference image at a size of, for example, "64 x 64." If the estimated imaging magnification is, for example, 20x, each partial image (cropped image) is extracted from random coordinates in the inference image at a size of, for example, "128 x 128." If the estimated imaging magnification is, for example, 40x, each partial image is extracted from random coordinates in the inference image at a size of, for example, "256 x 256." If the estimated imaging magnification is, for example, 100x, each partial image is extracted from random coordinates in the inference image at a size of, for example, "640 x 640."

[0031] The size unifying unit 34 unifies (resizes) the size of each partial image extracted from the inference image to, for example, a size of 64 × 64. This makes it easier to infer image features because the apparent sizes of the components captured in each partial image are roughly uniform.

[0032] The contribution estimation unit 35 (contribution estimator) estimates the contribution of each partial image included in the image for inference (input image) to the classification of the image for inference. Here, the contribution estimation unit 35 may estimate the average value of the contributions at multiple coordinates in the partial image as the contribution (average contribution) of that partial image.

[0033] Based on the estimated contribution, the image selection unit 36 ​​focuses on partial images that are important for classification by the image classification unit 37. That is, the image selection unit 36 ​​preferentially selects partial images with high estimated contributions (important partial images) from the inference images. For example, the image selection unit 36 ​​determines partial images with contributions equal to or greater than a predetermined contribution threshold as partial images with high contributions.

[0034] The image classification unit 37 (image classifier) ​​classifies the selected partial image, thereby classifying the inference image including the selected partial image. For example, if a partial image is classified into the image group for cases of acute myeloid leukemia, the inference image including that partial image is also classified into the image group for cases of acute myeloid leukemia. The image classification unit 37 records the classification results of the inference image in the storage device 2.

[0035] Next, an example of the operation of the image classification device 3 during the learning process will be described. Fig. 2 is a flowchart showing an example of the operation of the image classification device 3 during the learning process in this embodiment.

[0036] <Preparation Process of Learning Samples> The acquisition unit 31 acquires learning samples including learning images and the like from the storage device 2 (step S101).

[0037] The acquisition unit 31 acquires the learning image “x ｉ ", learning label (correct label) "y ｉ ", learning magnification information "s ｉ " is a learning sample "(x ｉ , y ｉ , s ｉ ) is used as a set of learning samples (teaching data).

[0038] The acquisition unit 31 acquires the important partial image “z ｉ ", which also contains the training sample "(x ｉ , y ｉ , s ｉ , z ｉ A set of "important partial images for learning" may be used as a learning sample set. Important partial images for learning are partial images (important partial images) that have a high degree of contribution to the classification of the learning images. Each pixel of the important partial images for learning is previously associated with a pixel value indicating the degree of contribution to the classification.

[0039] The acquisition unit 31 acquires the learning numerical data “d ｉ ", which also contains the training sample "(x ｉ , y ｉ , s ｉ , d ｉ The acquisition unit 31 may use a set of important partial images "z ｉ" and learning numerical data "d ｉ ", which also contains the training sample "(x ｉ , y ｉ , s ｉ , d ｉ , z ｉ ) may be used as a training sample set.

[0040] The learning imaging magnification information is information that indicates the imaging magnification of the learning image (the magnification of the lens used for imaging). The learning numerical data is numerical data that can be used to infer the class labels of the learning images, such as peripheral blood test data. In order for the image classifier of the image classification unit 37 to learn in advance, a set of learning samples including pairs of learning images and learning labels (correct labels) may be added to the learning sample set.

[0041] Hereinafter, the total number of learning samples is, for example, N. The number of learning labels (correct labels) is, for example, K.

[0042] Hereinafter, the "i∈{1, ..., N}"-th learning image is "x ｉ ∈R Ｈ×Ｗ The learning label (correct label) is written as "y ｉ ∈{1, 2, ..., K}". The learning magnification information is expressed as "s ｉ ∈R∪{1, ..., M}". The important partial images for learning are denoted as "z ｉ ∈R Ｈ×Ｗ The learning data is written as "d ｉ ∈R Ｄ " is written as ".

[0043] <Magnification Estimation Process> The magnification estimation unit 32 estimates the photographing magnification of the learning image (step S102).

[0044] Input: learning image and coefficients of the magnification estimator Output: information on estimated shooting magnification for learning (hereinafter referred to as "estimated magnification information for learning"). When shooting magnification information for learning (accurate magnification information) is obtained, the magnification estimation unit 32 may output the shooting magnification information for learning (accurate magnification information) to the image extraction unit 33 instead of outputting the estimated magnification information for learning to the image extraction unit 33.

[0045] The magnification estimation unit 32 acquires training images from the acquisition unit 31. The magnification estimation unit 32 estimates the shooting magnification of the input training images using a magnification estimator that has been trained in advance using the coefficients of the magnification estimator. The estimation of the shooting magnification can be formulated as a regression problem that estimates the shooting magnification based on the image. Furthermore, if the number of shooting magnifications is narrowed down to M in advance, the estimation of the shooting magnification can also be formulated as an M-class classification problem. In either case, for example, a known image input neural network (e.g., "ResNet" and "Vision Transformer") can be used as the magnification estimator.

[0046] The magnification estimation unit 32 outputs the numerical value of the photographing magnification to the image extraction unit 33. The magnification estimation unit 32 may output a class label corresponding to the photographing magnification to the image extraction unit 33. The magnification estimation unit 32 may output a certainty of the class label corresponding to the photographing magnification to the image extraction unit 33.

[0047] <Process of Calculating Magnification Estimation Loss> When the coefficients of the magnification estimator have not converged, the magnification estimation unit 32 calculates a magnification estimation loss for learning (step S103).

[0048] Input: Estimated magnification information for learning and shooting magnification information for learning Output: Magnification estimation loss for learning The smaller the difference between the estimated magnification information for learning and the shooting magnification information for learning, the smaller the magnification estimation loss.

[0049] Hereinafter, symbols placed above letters in mathematical expressions or functions (hereinafter referred to as "mathematical expressions, etc.") will be written before the letters. For example, the symbol "^" placed above a letter in a mathematical expression, etc. will be written below as "^s" before the letter "s". For example, the symbol "-" placed above a letter in a mathematical expression, etc. will be written below as "(-)s" before the letter "s".

[0050] In order to evaluate whether the inference of the learning estimated magnification information was a desirable inference (whether the coefficients of the magnification estimator are desirable values), the magnification estimation unit 32 calculates the learning magnification estimation loss based on the learning estimated magnification information and the learning shooting magnification information.

[0051] For example, the magnification estimation unit 32 may use the learning photographing magnification information “s ｉ " and the estimated magnification information for learning "^s ｉ " is calculated as the magnification estimation loss for learning. For example, when the shooting magnification is a real value, the magnification estimation unit 32 may calculate, for example, a mean square error as the magnification estimation loss for learning. For example, when the shooting magnification is expressed by a class or by a confidence level corresponding to each class, the magnification estimation unit 32 may calculate, for example, "cross entropy" or "Kullback-Leibler divergence" as the magnification estimation loss for learning.

[0052] <Process for updating the coefficients of the magnification estimator (if the coefficients of the magnification estimator have not converged after execution, processing returns to the magnification estimation process.)> If the coefficients of the magnification estimator have not converged, the magnification estimation unit 32 updates the coefficients of the magnification estimator (step S104).

[0053] Input: scaling factor estimation loss for learning and current coefficients of the scaling factor estimator Output: updated coefficients of the scaling factor estimator (model parameters) The scaling factor estimator 32 updates the coefficients of the scaling factor estimator based on the scaling factor estimation loss.

[0054] The scaling factor estimation unit 32 (scaling factor estimator) may include a neural network. The scaling factor estimation loss for training is back-propagated to this neural network. In response to this, the scaling factor estimator 32 updates the coefficients of the scaling factor estimator.

[0055] <Process of Extracting Partial Images> The image extracting unit 33 extracts partial images of sizes according to the shooting magnification from the learning image (step S105). The size unifying unit 34 unifies (resizes) the sizes of the partial images (step S106).

[0056] Input: training image and estimated magnification information for training Output: collection of partial images extracted from the training image (collection of partial images for training) The image extraction unit 33 extracts one or more partial images from the input training image. Here, the image extraction unit 33 extracts partial images of a size determined according to the input estimated magnification information from the training image. The size unification unit 34 unifies (resizes) the sizes of the extracted partial images so that each partial image has the same size.

[0057] For example, if there are two cases of estimated magnifications for learning (10x or 20x) and the estimated magnification information for learning is 10x, the image extraction unit 33 extracts a partial image of, for example, 64x64 pixels from the learning image. If the estimated magnification information for learning is 20x, the image extraction unit 33 extracts a partial image of, for example, 128x128 pixels from the learning image.

[0058] The image extraction unit 33 extracts B partial images for learning "x ｉ，ｊ ∈R ＾Ｈ×＾Ｗ " to the training image "x ｉ ∈R Ｈ×Ｗ In this way, the image extraction unit 33 extracts the partial images for learning from the set "{^x ｉ，ｊ} Ｂ ｊ＝１ " to create a

[0059] The size unification unit 34 reduces the partial images of 128 x 128 pixels to partial images of 64 x 64 pixels so that the final sizes of the partial images are the same. This makes it possible to unify the sizes of the same structural elements (objects) captured in the partial images even if the learning images are captured at different magnifications. The positions (coordinates) from which the partial images are extracted in the learning images may be fixed positions or random positions.

[0060] <Contribution Degree Estimation Process> The contribution degree estimation unit 35 estimates the contribution degree (importance) to classification for each partial image included in the set of partial images for learning (step S107).

[0061] Input: A set of partial images for learning and coefficients of the contribution estimator (model parameters) Output: Estimated contributions for learning (hereinafter referred to as "estimated contributions for learning")

[0062] The contribution estimation unit 35 estimates the contribution of each partial image included in the set of training partial images to classification based on the coefficients of the contribution estimator. The estimated training contribution represents the degree to which each partial image, which is an element of the set of training partial images, contributes to classification. The contribution estimation can be formulated as a regression problem that estimates the contribution based on an image. For example, a known image input neural network (e.g., "ResNet" and "Vision Transformer") can be used as the contribution estimator.

[0063] <Process of Calculating Loss of Contribution Estimation> When the coefficients of the contribution estimator have not converged, the contribution estimator 35 calculates a contribution estimation loss (step S108).

[0064] Input: Estimated contribution for learning and important partial image for learning Output: Contribution estimation loss for learning The contribution estimation unit 35 calculates the contribution estimation loss for learning based on the contribution (pixel value of the important partial image) associated with each pixel of the important partial image for learning and the estimated contribution for learning. The smaller the difference between the contribution associated with each pixel of the important partial image for learning and the estimated contribution for learning, the smaller the contribution estimation loss for learning.

[0065] In order to evaluate whether the inference of the estimated contribution for learning is a desirable inference (whether the coefficient of the contribution estimator is a desirable value), the contribution estimation unit 35 calculates the contribution estimation loss for learning based on the estimated contribution for learning and the important partial image for learning. For example, the contribution estimation unit 35 calculates the contribution estimation loss for learning based on the important partial image for learning "z ｉ "The contribution rate estimated based on "(-)z ｉ ” and the estimated contribution for learning “^z ｉ The difference (deviation) between " and " is calculated as the contribution estimation loss for training.

[0066] The contribution estimation unit 35, for example, detects the position (coordinates) in the learning image from which the learning partial image used to calculate the estimated contribution for learning was extracted. The contribution estimation unit 35 extracts a partial image from that position in the important partial image for learning. The contribution estimation unit 35 may use the average value of the contribution for each pixel in the extracted partial image (pixel value of the important partial image) as the estimated contribution for learning. Furthermore, the contribution estimation unit 35 may calculate, for example, a mean square error as the contribution estimation loss for learning.

[0067] <Process for updating the coefficients of the contribution estimator (if the coefficients of the contribution estimator have not converged after execution, processing returns to the contribution estimation process.)> If the coefficients of the contribution estimator have not converged, the contribution estimation unit 35 updates the coefficients of the contribution estimator (step S109).

[0068] Input: Contribution estimation loss for training and current coefficients of the contribution estimator Output: Updated coefficients (model parameters) of the contribution estimator The contribution estimation unit 35 updates the coefficients of the contribution estimator based on the contribution estimation loss for training.

[0069] The contribution estimation unit 35 (contribution estimator) may include a neural network. The contribution estimation loss for training is back-propagated to this neural network. As a result, the contribution estimation unit 35 updates the coefficients of the contribution estimator.

[0070] <Partial Image Selection Process (After execution, the process may return to the partial image extraction process so that multiple partial images are extracted.)> The image selection unit 36 ​​selects partial images based on the contribution degree (step S110).

[0071] Input: A set of partial images for training and estimated contributions for training. Output: A set of updated partial images. The more a partial image contributes to classification, the more likely it is to be selected.

[0072] The image selection unit 36 ​​calculates the contribution of each partial image included in the learning partial image set based on the estimated contribution for learning. The image selection unit 36 ​​updates the learning partial image set by removing partial images with small contributions from the learning partial image set. For example, the image selection unit 36 ​​removes partial images with contributions below a predetermined threshold from the learning partial image set. For example, the image selection unit 36 ​​may remove partial images with contributions below a generated random number (random contribution threshold) from the learning partial image set.

[0073] <Image Classification Process> The image classification unit 37 classifies the selected partial image, thereby classifying the learning images including the selected partial image (step S111).

[0074] Input: A set of partial images for training, and coefficients (model parameters) of the image classifier Furthermore, training images may be additionally input to the image classification unit 37. Furthermore, training numerical data may be additionally input to the image classification unit 37. Output: Classification results of training images The image classification unit 37 may output estimated class labels to the storage device 2 as classification results of the training images. The image classification unit 37 may output confidence levels for each class label to the storage device 2 as classification results of the training images.

[0075] The image classification unit 37 classifies the set of partial images for training using an image classifier that has been trained in advance using the coefficients of the image classifier. In this way, the image classification unit 37 classifies the training images from which the set of partial images for training has been extracted. In other words, the image classification unit 37 estimates the class labels of the training images from which the set of partial images for training has been extracted.

[0076] The image classification unit 37 may classify the training images from which the set of training partial images has been extracted, based on the training images additionally input from the acquisition unit 31. The image classification unit 37 may classify the training images from which the set of training partial images has been extracted, based on the training numerical data additionally input from the acquisition unit 31.

[0077] As the image classifier, for example, a known image input neural network (such as "ResNet" and "Vision Transformer") can be used.

[0078] The image classification unit 37 acquires a set of partial images for learning (plurality of partial images) from the image selection unit 36. Furthermore, the image classification unit 37 may acquire, from the acquisition unit 31, images for learning in a process prior to the process in which the partial images are extracted.

[0079] The image classification unit 37 partially uses the image input neural network for each partial image included in the set of partial images for training. In this way, the image classification unit 37 extracts feature quantities of the partial images from the partial images. The image classification unit 37 converts all feature maps obtained by the feature quantity extraction into a single feature map. The image classification unit 37 classifies the partial images for training by inputting the feature maps to one or more fully connected layers in the image classifier.

[0080] When the additionally input training image is used for classification, the image classification unit 37 adjusts the size of the additionally input training image so that it is the same size as the partial image. The image classification unit 37 inputs the training image whose size has been adjusted to the image classifier.

[0081] Furthermore, when numerical data is used for classification, the image classifier 37 connects the converted numerical data, for example, using one or more fully connected layers, to a feature map obtained from the partial image. The image classifier 37 classifies the partial image for training using one or more fully connected layers in the image classifier.

[0082] <Classification Loss Calculation Process> The image classification unit 37 calculates the classification estimation loss (learning classification loss) (step S112).

[0083] Input: Classification results of training images and training labels (correct labels) Output: Training classification loss If the classification results of training images (estimated labels) and the training labels (correct labels) match, the value of the training classification loss is small. In other words, if the classification results of training images (estimated labels) and the training labels (correct labels) differ, the value of the training classification loss is large.

[0084] To evaluate whether the inference of the classification results of the training images is desirable (whether the coefficients of the image classifier are desirable values), the image classifying unit 37 calculates a training classification loss based on the classification results of the training images and the training labels. For example, the image classifying unit 37 calculates a training classification loss based on the training label "y ｉ " and the classification result of the training image "^y ｉ The difference (deviation) between the image classification loss and the image classification loss is calculated as the training classification loss. The image classification unit 37 may calculate, for example, "cross entropy" or "Kullback-Leibler divergence."

[0085] <Process of Updating Classifier Coefficients (If the classifier coefficients have not converged after execution, the process returns to the image classification process.)> The image classifying unit 37 updates the classifier coefficients (step S113).

[0086] Input: training classification loss and current coefficients of the image classifier. Output: updated coefficients (model parameters) of the image classifier. The image classifier unit 37 updates the coefficients of the image classifier based on the training classification loss.

[0087] The image classifier 37 may include a neural network, through which the training classification loss is backpropagated, thereby updating the coefficients of the image classifier.

[0088] Next, an example of the operation of the image classification device 3 during the inference process will be described. Fig. 3 is a flowchart showing an example of the operation of the image classification device 3 during the inference process in an embodiment. The inference process is executed after the learning process is completed.

[0089] <Magnification Estimation Process> The magnification estimation unit 32 estimates the shooting magnification of the inference-use image (step S201).

[0090] Input: Image for inference and coefficients of the magnification estimator Output: Information on estimated shooting magnification for inference (hereinafter referred to as ``estimated magnification information for inference'') When shooting magnification information for inference (accurate magnification information) can be obtained, the magnification estimation unit 32 may output the obtained shooting magnification information for inference to the image extraction unit 33 instead of outputting the estimated magnification information for inference to the image extraction unit 33.

[0091] The magnification estimation unit 32 acquires an inference image from the acquisition unit 31. The magnification estimation unit 32 estimates the shooting magnification of the input inference image using a magnification estimator that has been trained in advance using the coefficients of the magnification estimator. The estimation of the shooting magnification can be formulated as a regression problem that estimates the shooting magnification based on the image. Furthermore, if the number of shooting magnifications is narrowed down to M in advance, the estimation of the shooting magnification can also be formulated as an M-class classification problem. In either case, for example, a known image input neural network (e.g., "ResNet" and "Vision Transformer") can be used as the magnification estimator.

[0092] The magnification estimation unit 32 outputs the numerical value of the photographing magnification to the image extraction unit 33. The magnification estimation unit 32 may output a class label corresponding to the photographing magnification to the image extraction unit 33. The magnification estimation unit 32 may output a certainty of the class label corresponding to the photographing magnification to the image extraction unit 33.

[0093] <Process of Extracting Partial Images> The image extracting unit 33 extracts partial images of a size according to the shooting magnification from the inference image (step S202). The size unifying unit 34 unifies (resizes) the sizes of the partial images (step S203).

[0094] Input: Image for inference and estimated magnification information for inference Output: Set of partial images extracted from the image for inference (set of partial images for inference) The image extraction unit 33 extracts one or more partial images from the input image for inference. Here, the image extraction unit 33 extracts partial images of a size determined according to the input estimated magnification information from the image for inference. The size unification unit 34 unifies (resizes) the sizes of the extracted partial images so that each partial image has the same size.

[0095] For example, if there are two cases of estimated magnifications for inference (10x or 20x) and the estimated magnification information for inference is 10x, the image extraction unit 33 extracts a partial image of, for example, 64x64 pixels from the image for inference. If the estimated magnification information for inference is 20x, the image extraction unit 33 extracts a partial image of, for example, 128x128 pixels from the image for inference.

[0096] The image extraction unit 33 extracts B partial images "x ｉ，ｊ ∈R ＾Ｈ×＾Ｗ " is used as the inference image "x ｉ ∈R Ｈ×Ｗ In this way, the image extraction unit 33 extracts the partial images for inference from the set "{^x ｉ，ｊ} Ｂ ｊ＝１ " to create a

[0097] The size unification unit 34 reduces the partial images of 128 x 128 pixels to partial images of 64 x 64 pixels so that the final size of each partial image is the same. This makes it possible to unify the size of the same structural element (object) captured in the partial images even if the magnification of the inference image is different. The position (coordinates) from which the partial image is extracted in the inference image may be a fixed position or a random position.

[0098] <Contribution Degree Estimation Process> The contribution degree estimation unit 35 estimates the contribution degree (importance) to classification for each partial image included in the set of partial images for inference (step S204).

[0099] Input: A set of partial images for inference and coefficients of the contribution estimator (model parameters) Output: Estimated contributions for inference (hereinafter referred to as "estimated contributions for inference")

[0100] The contribution estimation unit 35 estimates the contribution of each partial image included in the set of partial images for inference to classification based on the coefficients of the contribution estimator. The estimated contribution for inference represents the degree to which each partial image, which is an element of the set of partial images for inference, contributes to classification. The contribution estimation can be formulated as a regression problem that estimates the contribution based on an image. For example, a known image input neural network (e.g., "ResNet" and "Vision Transformer") can be used as the contribution estimator.

[0101] <Partial Image Selection Process (After execution, the process may return to the partial image extraction process so that multiple partial images are extracted (selected))> The image selection unit 36 ​​selects partial images based on the contribution degree (step S205).

[0102] Input: A set of partial images for inference and estimated contribution for inference. Output: A set of updated partial images. The more a partial image contributes to classification, the more likely it is to be selected.

[0103] The image selection unit 36 ​​calculates the contribution of each partial image included in the set of partial images for inference based on the estimated contribution for inference. The image selection unit 36 ​​updates the set of partial images for inference by removing partial images with small contributions from the set of partial images for inference. For example, the image selection unit 36 ​​removes partial images with contributions below a predetermined threshold from the set of partial images for inference. For example, the image selection unit 36 ​​may remove partial images with contributions below a generated random number (random contribution threshold) from the set of partial images for inference.

[0104] <Image Classification Process> The image classification unit 37 classifies the selected partial image, thereby classifying the inference image including the selected partial image (step S206).

[0105] Input: A set of partial images for inference, and coefficients (model parameters) of the image classifier Furthermore, images for inference and numerical data for inference may be additionally input to the image classification unit 37. Output: Classification results of images for inference The image classification unit 37 may output estimated class labels to the storage device 2 as classification results of the images for inference. The image classification unit 37 may output confidence levels for each class label to the storage device 2 as classification results of the images for inference.

[0106] The image classification unit 37 classifies the set of partial images for inference using an image classifier that has been trained in advance using the coefficients of the image classifier. In this way, the image classification unit 37 classifies the images for inference from which the set of partial images for inference has been extracted. In other words, the image classification unit 37 estimates the class labels of the images for inference from which the set of partial images for inference has been extracted.

[0107] The image classification unit 37 may classify the inference-use images from which a set of partial images for inference has been extracted, based on the inference-use images additionally input from the acquisition unit 31. The image classification unit 37 may classify the inference-use images from which a set of partial images for inference has been extracted, based on the inference-use numerical data additionally input from the acquisition unit 31.

[0108] As the image classifier, for example, a known image input neural network (such as "ResNet" and "Vision Transformer") can be used.

[0109] The image classification unit 37 acquires a set of partial images for inference (plurality of partial images) from the image selection unit 36. Furthermore, the image classification unit 37 may acquire, from the acquisition unit 31, images for inference in a process prior to the process in which the partial images are extracted.

[0110] The image classification unit 37 partially uses the image input neural network for each partial image included in the set of partial images for inference. In this way, the image classification unit 37 extracts feature quantities of the partial images from the partial images. The image classification unit 37 converts all feature maps obtained by the feature quantity extraction into a single feature map. The image classification unit 37 classifies the partial images for inference by inputting the feature maps to one or more fully connected layers in the image classifier.

[0111] When the additionally input image for inference is to be used for classification, the image classification unit 37 adjusts the size of the additionally input image for inference so that it is the same size as the partial image. The image classification unit 37 inputs the adjusted size image for inference to the image classifier.

[0112] Furthermore, if numerical data is used for classification, the image classifier 37 may, for example, use one or more fully connected layers to connect the converted numerical data to a feature map obtained from the subimage. The image classifier 37 may use one or more fully connected layers in the image classifier to classify the subimage for inference.

[0113] As described above, the contribution estimation unit 35 estimates the contribution of each partial image included in the input image to the classification of the input image. In the learning process, the input image is an image for learning. In the inference process after the learning process, the input image is an image for inference. The image selection unit 36 ​​preferentially selects partial images with high contributions. For example, the image selection unit 36 ​​determines that a partial image having a contribution equal to or greater than a random contribution threshold value among real values ​​between 0 and 1 is a partial image with high contribution. The image classification unit 37 classifies the selected partial images, thereby classifying the input image including the selected partial images.

[0114] This makes it possible to classify input images while reducing the cost required to build machine learning training samples.

[0115] The magnification estimation unit 32 estimates the shooting magnification of the input image. The image extraction unit 33 extracts partial images from the input image at a size corresponding to the shooting magnification. The size unification unit 34 unifies the sizes of the partial images. The contribution estimation unit 35 estimates the contribution degree for each partial image of unified size.

[0116] This makes it possible to make the apparent sizes of the components captured in each partial image roughly uniform, facilitating learning of image features.

[0117] (Example of Experimental Results) 472 inference samples, including bone marrow biopsy images for inference and peripheral blood test data, were collected for an experiment of the image classification method. The bone marrow biopsy images for inference included images of bone marrow aspirates aspirated from bone marrow fluid and images of hematopoietic tissues, including bone tissue. Each inference sample was pre-labeled with a correct answer label indicating whether the person from whom the bone marrow tissue was collected by bone marrow biopsy suffered from acute myeloid leukemia (AML) or myelodysplastic syndrome (MDS).

[0118] The performance of this task of inferring the correct label was compared between an image classification method according to a known technique and the image classification method of the embodiment. In the following, the image classification method according to the known technique is a method for classifying images of bone marrow biopsies of acute myeloid leukemia.

[0119] The tasks for which performance was compared were binary classification tasks, namely, classifying bone marrow biopsy images as being of acute myeloid leukemia or myelodysplastic syndrome. Therefore, performance on these tasks was evaluated using the area under the receiver operating characteristic curve (AUC). AUC corresponds to the area under the right corner of the receiver operating characteristic curve (ROC curve).

[0120] FIG. 4 is a diagram showing an example of a receiver operating characteristic curve (ROC curve) of an image classification method according to a known technique. In FIG. 4, the inference process was performed using only the inference images (input images) in the inference sample. On the left side of FIG. 4, the receiver operating characteristic curve "Macro avg" of the image classification method according to the known technique is represented by a thick solid line. Specifically, the receiver operating characteristic curve shown on the left side of FIG. 4 indicates the performance of the task of inferring whether an image is a bone marrow biopsy image of acute myeloid leukemia.

[0121] The right side of Fig. 4 shows receiver operating characteristic curves for the image classification method according to the embodiment. The receiver operating characteristic curve "AUC_AML" shown on the right side of Fig. 4 indicates the performance of the task of inferring whether an image is a bone marrow biopsy image of acute myeloid leukemia. The receiver operating characteristic curve "AUC_MDS" shown on the right side of Fig. 4 indicates the performance of the task of inferring whether an image is a bone marrow biopsy image of myelodysplastic syndrome.

[0122] The receiver operating characteristic curve of the image classification method of the embodiment is located to the upper right compared to the receiver operating characteristic curve of the image classification method according to the known technology. Therefore, the image classification method of the embodiment can more accurately infer whether an image is a bone marrow biopsy image of acute myeloid leukemia. Specifically, the AUC of the image classification method according to the known technology is 0.70. In contrast, the AUC of the image classification method of the embodiment is high at 0.82.

[0123] Furthermore, the image classification method according to the embodiment is as accurate in determining whether an image is a bone marrow biopsy image of myelodysplastic syndrome as it is in determining whether an image is a bone marrow biopsy image of acute myeloid leukemia. Specifically, the AUC of the image classification method according to the embodiment is as high as 0.80.

[0124] FIG. 5 is a diagram showing an example of a receiver operating characteristic curve (ROC curve) of an image classification method according to an embodiment. In FIG. 5, an inference process was performed using an inference image (input image) and peripheral blood test data (numerical data) in an inference sample. The receiver operating characteristic curve "AUC_AML" shown in FIG. 5 indicates the performance of the task of inferring whether an image is a bone marrow biopsy image of acute myeloid leukemia. The receiver operating characteristic curve "AUC_MDS" shown in FIG. 5 indicates the performance of the task of inferring whether an image is a bone marrow biopsy image of myelodysplastic syndrome.

[0125] By performing the inference process using not only the inference image but also the numerical data, the image classification method of the embodiment can achieve higher performance than when the inference process is performed using only the inference image. Specifically, the AUC of the image classification method of the embodiment is high at 0.93 for the performance of the task of inferring whether an image is a bone marrow biopsy image for acute myeloid leukemia. Also, the AUC of the image classification method of the embodiment is high at 0.91 for the performance of the task of inferring whether an image is a bone marrow biopsy image for myelodysplastic syndrome.

[0126] The horizontal dashed line illustrated in Figure 5 indicates that 95% of the samples that are actually positive were inferred to be positive. In other words, the horizontal dashed line illustrated in Figure 5 indicates a true positive rate (TPR) of 0.95. When the true positive rate is 0.95, the false positive rate is approximately 0.4. This indicates that in a situation where 95% of samples are correctly identified as positive, the rate at which negative samples are incorrectly identified as positive is only approximately 40%.

[0127] (Hardware Configuration) FIG. 6 is a diagram illustrating an example of the hardware configuration of the image classification device 3 according to an embodiment. The image classification device 3 is realized as software by a processor 101, such as a CPU (Central Processing Unit), executing a program stored in a storage device 103 having a non-volatile recording medium (non-transitory recording medium) and a memory 102. The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), and a CD-ROM (Compact Disc Read Only Memory), and non-transitory recording media such as a storage device built into a computer system, such as a hard disk or a solid-state drive (SSD). A communication unit 104 executes predetermined communication processing.

[0128] The image classification device 3 may be realized using hardware (accelerator) including an electronic circuit (electronic circuit or circuitry) using, for example, an LSI (Large Scale Integrated circuit), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0129] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0130] The present invention is applicable to a system for classifying (identifying) images.

[0131] 1...image classification system, 2...storage device, 3...image classification device, 31...acquisition unit, 32...magnification estimation unit, 33...image extraction unit, 34...size unification unit, 35...contribution estimation unit, 36...image selection unit, 37...image classification unit, 101...processor, 102...memory, 103...storage device, 104...communication unit

Claims

1. An image classification device comprising: a contribution estimation unit that estimates the contribution of each partial image included in an input image to the classification of the input image; an image selection unit that preferentially selects partial images with a high contribution degree; and an image classification unit that classifies the input image including the selected partial images by classifying the selected partial images.

2. An image classification device as described in claim 1, further comprising: a magnification estimation unit that estimates a shooting magnification of the input image; an image extraction unit that extracts the partial image from the input image at a size corresponding to the shooting magnification; and a size unification unit that unifies the sizes of the partial images, wherein the contribution degree estimation unit estimates the contribution degree for each of the partial images of the unified size.

3. The image classification device according to claim 1, wherein the image selection unit determines that a partial image having a contribution rate equal to or greater than a random contribution rate threshold is a partial image having a high contribution rate.

4. An image classification method executed by an image classification device, comprising: a step of estimating the contribution of each partial image contained in an input image to the classification of the input image; a step of preferentially selecting the partial images with a high contribution; and a step of classifying the input image including the selected partial images by classifying the selected partial images.

5. A program for causing a computer to execute the steps of: estimating the contribution of each partial image contained in an input image to the classification of the input image; preferentially selecting the partial images having a high contribution; and classifying the input image including the selected partial images by classifying the selected partial images.

Images