Method and medium for identifying and locating an image using triplet loss and prediction regions

Abstract

Methods and media for identifying and localizing images using triplet loss and predicted regions. A method and system for classifying image features using a neural network is provided. The method includes training the neural network using a triplet loss process, including receiving an anchor image, selecting a positive image and a negative image, generating an image embedding associated with each of the anchor image, the positive image, and the negative image, classifying image features extracted from the anchor image based on the image embedding of the anchor image, determining image label locations associated with the classified image features, extracting features associated with the determined image label locations, and classifying the features associated with the determined image label locations; and using a weighted loss and combining a multi-label loss with a local image classification loss and a triplet loss.

Description

Technical Field

[0001] The present disclosure relates to image analysis and, more particularly, to systems and methods for automatically identifying and locating image regions. Background Art

[0002] In existing applications, the identification and localization of diseases in medical images has been applied to the segmentation of diseases in medical images. Furthermore, this approach can be used to associate the textual description of a disease in a medical report with the image region discussed in the report. Existing methods already allow for the automatic segmentation of organs. By understanding which diseases are present in an image and the approximate location of each disease, organ segmentation methods can be applied to disease segmentation. This, in turn, improves the efficiency of measuring the size of diseases in medical images.

[0003] Furthermore, the ability to perform disease localization in the prior art also allows for linking or highlighting the location of a disease or abnormality described in a radiologist's or physician's report to a medical image. This can help improve the understanding of medical reports by physicians and their patients. While there are numerous medical datasets with disease annotated, few have disease location annotations. Therefore, it would be desirable to develop a system that requires minimal, if any, annotation of disease location (e.g., a bounding box indicating a diseased area). Summary of the Invention

[0004] Aspects of the present application may include a method for classifying image features using a neural network. The method includes the following steps: training the neural network using a triplet loss process, including: receiving an anchor image and an image label associated with the anchor image, the anchor image to be used to train the neural network; selecting a positive image and an image label associated with the positive image, the positive image having at least one image label shared with the anchor image; selecting a negative image and an image label associated with the negative image, the negative image having no image label shared with the anchor image; using an image feature embedding network to generate an image embedding associated with each of the anchor image, the positive image, and the negative image; classifying image features extracted from the anchor image based on the image embedding of the anchor image by the neural network; and calculating a triplet loss based on the image embedding associated with each of the anchor image, the positive image, and the negative image to determine a classification weight for the image features; and using the trained neural network to classify image features of a test data image that does not have any associated labeled features.

[0005] Additional aspects of the present application may include a method for classifying image features using a neural network. The method includes the steps of training a neural network using a triplet loss process, including: receiving an anchor image and an image label associated with the anchor image, the anchor image to be used to train the neural network; generating an image embedding associated with the anchor image using an image feature embedding network; classifying image features extracted from the anchor image based on the image embedding of the anchor image using the neural network; determining image label locations associated with the classified image features; extracting features associated with the determined image label locations; classifying the features associated with the determined image label locations using the neural network; and determining a local image classification loss associated with the features associated with the determined image label locations; and classifying image features of a test data image that does not have any associated labeled features using the trained neural network.

[0006] Additional aspects of the present application may include a non-transitory computer-readable medium encoded with instructions for causing a computer to perform a method of classifying image features using a neural network. The method includes the steps of training a neural network using a triplet loss process, comprising: receiving an anchor image and an image label associated with the anchor image, the anchor image to be used for training the neural network; selecting a positive image and an image label associated with the positive image, the positive image having at least one image label shared with the anchor image; selecting a negative image and an image label associated with the negative image, the negative image having no image label shared with the anchor image; generating an image embedding associated with each of the anchor image, the positive image, and the negative image using an image feature embedding network; classifying image features extracted from the anchor image based on the image embedding of the anchor image by the neural network; and determining an image label position associated with the classified image feature; extracting features associated with the determined image label position; classifying the features associated with the determined image label position using the neural network; determining a local image classification loss associated with the features associated with the determined image label position; calculating a triplet loss based on the image embedding associated with each of the anchor image, the positive image, and the negative image to determine a classification weight for the image features; and combining the local image classification loss and the triplet loss using the weighted loss; and classifying image features of a test data image that does not have any associated labeled features using the trained neural network.

[0007] Another aspect of the present application may include a computing device for classifying image features using a neural network. The computing device may include a device for training the neural network using a triplet loss process, wherein training the neural network using the triplet loss process includes: receiving an anchor image and an image label associated with the anchor image, the anchor image to be used to train the neural network; selecting a positive image and an image label associated with the positive image, the positive image having at least one image label shared with the anchor image; selecting a negative image and an image label associated with the negative image, the negative image having no image label shared with the anchor image; using an image feature embedding network to extract image features and generate an image embedding associated with each of the anchor image, the positive image, and the negative image; and performing a classification on the image features extracted from the anchor image based on the image embedding of the anchor image by the neural network. classify image features; and determine an image label position associated with the classified image features; extract features associated with the determined image label position; classify the features associated with the determined image label position using a neural network; determine a local image classification loss associated with the features associated with the determined image label position; calculate a triplet loss based on an image embedding associated with each of an anchor image, a positive image, and a negative image to determine a classification weight for the image features; and use the weighted loss and combine the local image classification loss and the triplet loss; use the trained neural network to classify image features of a test data image that does not have any associated labeled features. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] This patent or application file contains at least one drawing drawn in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0009] Figure 1 A schematic representation showing training a multi-label disease classifier using triplet loss according to an example implementation of the present application is shown.

[0010] Figure 2 A schematic representation showing training an image classifier using local disease predictions according to an example implementation of the present application is shown.

[0011] Figure 3 Schematic representation showing training of an image classifier using local disease prediction according to other example implementations of the present application.

[0012] Figure 4 A flow chart illustrating a process of classifying an image using a trained neural network according to an example implementation of the present application.

[0013] Figure 5 and Figure 6 Shown is a graphical representation of evaluation results for an example implementation of the present application.

[0014] Figure 7 and Figure 8 An example heat map is shown along with the predicted and ground truth locations of diseases using an example implementation of the present application.

[0015] Figure 9 An example computing environment is shown with an example computer device suitable for use with some example implementations. DETAILED DESCRIPTION

[0016] The following detailed description provides further details of the drawings and example implementations of the present application. For clarity, the numbers and descriptions of redundant elements between the drawings are omitted. The terms used throughout the specification are provided as examples and are not intended to be limiting. For example, the use of the term "automatic" may relate to fully automatic or semi-automatic implementations, involving user or operator control of specific aspects of the implementation, depending on the desired implementation of the ordinary technician in the art who practices the implementation of the present application. In addition, sequential terms such as "first", "second", "third", etc. may be used only for marking purposes in the specification and claims and should not be limited to indicating that the described actions or items occur in the described order. Without departing from the scope of the present application, the actions or items may be sorted into different orders or may be executed in parallel or dynamically.

[0017] In this application, the term "computer-readable medium" may include a local storage device, a cloud-based storage device, a remote server, or any other storage device that may be apparent to one of ordinary skill in the art.

[0018] As mentioned above, the identification and localization of diseases in medical images has been applied to the segmentation of diseases in medical images and can also be used to associate textual descriptions of diseases in medical reports with the image regions in question. Existing automatic organ segmentation methods can be used when the presence of diseases in an image and the approximate location of each disease are known. However, annotation of disease locations is rarely available. Therefore, it would be desirable to develop a system that requires minimal, if any, annotation of disease locations (e.g., a bounding box indicating the diseased region).

[0019] Example implementations of the present application can provide improved classification and localization of diseases in medical images. In some example implementations of the present application, object localization processing can be used to automatically identify image regions that contain relevant organs (e.g., lungs located in the chest), thereby removing errors caused by incorrectly identifying diseases in areas where relevant organs are not (e.g., shoulders or arms). In other example implementations, triplet loss processing can be applied to the classification model so that the model learns in a more continuous manner how much the distance between pairs of images containing different diseases increases. In addition, in some example implementations, the estimated disease area can be fed back to the model during training to better learn how to ensure disease size localization of the disease in medical images. In some example implementations, all of these techniques can be combined.

[0020] Although the example implementations of the present application may be described in the context of disease identification in medical images, the example implementations are not limited to analysis of medical images. Instead, the example implementations may be applied to other types of image analysis that may be apparent to those of ordinary skill in the art.

[0021] Figure 1 A schematic representation 100 is shown of training a multi-label disease classifier using a triplet loss according to an example implementation of the present application. In some example implementations, medical images often contain regions larger than the area where disease is present. For example, a chest x-ray often includes the patient's shoulder and arm. In these example implementations, a spatial transformer network can be used to crop the medical image to focus on the area where disease is present. This step may be performed in some example implementations, or omitted in some example implementations.

[0022] Furthermore, as shown, in some example implementations, a triplet loss technique may be adapted as part of a multi-label classification system. Specifically, the triplet loss may be used to force images with at least one common disease to be close together in feature space, while images without any common disease may be far apart in feature space. The triplet loss may minimize the distance between an image to be classified, or anchor image 105, and another image of the same type (e.g., same patient, same disease, or other similar characteristics). Figure 1 As shown, images of the same type are shown as images containing at least one common disease (e.g., positive images 107). In addition, images whose intersection of disease labels is empty are considered different types; these serve as examples of negative images 109. Using the triplet loss, for each input anchor image 105, one positive image 107 and one negative image 109 are randomly selected from the same training batch. In some example implementations, hard example mining can be used to improve the performance of the triplet loss technique discussed in more detail below.

[0023] like Figure 1 As shown, once a positive image 107 and a negative image 109 are randomly selected, each image (e.g., anchor image 105, positive image 107, and negative image 109) is processed by a spatial transformer network to transform each image to allow image comparison. For example, anchor image 105 is processed by the spatial transformer network at 111. Additionally, positive image 107 is processed by the spatial transformer network at 113. Additionally, negative image 109 is processed by the spatial transformer network at 115.

[0024] The spatial transformer network can be a differentiable module that applies cropping, translation, rotation, scaling, and skew to the input image (feature map) and requires only 6 parameters. Specifically, in an example implementation, the input chest x-ray image 105 with width W and height H can be represented by x=(x i ,x j ) i＝1,2 ;:::;W ,j =1,2,:::,H. The spatial transformer network outputs a 6-dimensional vector at 111, which can be parameterized as an affine transformation A Θ In the affine case, the point-by-point transformation is:

[0025]

[0026] Where (x t i ,y t i ) is the target coordinate in the output feature map, (x s i ,y s i ) is the source coordinate in the input image (feature map), A Θ is an affine transformation that allows cropping, translation, and isotropic scaling of the original image. During training, the model learns s (scaling) as well as t x and t y (Translation) value.

[0027] After the spatial transformer network processes each image (e.g., anchor image 105, positive image 107, and negative image 109), each image is processed by the image feature embedding network to embed image features for disease classification to generate image embeddings. Figure 1 As shown, an image embedding 123 is generated from the anchor image 105 by the image feature embedding network at 117. Similarly, an image embedding 125 is generated from the positive image 107 by the embedding network at 119. Furthermore, an image embedding 127 is generated from the negative image 109 by the embedding network at 121.

[0028] In some example implementations, a disease locator may be used to include disease location information as an embedded feature to embed a disease location feature. Figure 2 and Figure 3 Discuss in more detail.

[0029] The image embedding 123 created from the anchor image 105 may then be passed through a classifier network 129 to calculate a loss factor based on the labels assigned to the anchor image 105 in the training data. In other words, a multi-label loss factor 131 is calculated based on the difference between the assigned labels associated with the anchor image 105 from the training data and the classification of the anchor image 105 by the classifier network 129.

[0030] In addition to the multi-label loss factor 131, a second loss factor, the triplet loss factor 133, can be calculated as follows. The two losses are then combined using a weighted sum, giving equal weight to the overall image classification loss and the local image classification loss. Alternatively, the weights can be learned during training.

[0031] In some example implementations, considering a triple constraint set X may be considered as follows:

[0032] X={(x i ,x + i, x - i )|(x i ,x + i )∈P;(x i, x - i )∈N;i=1,2, …,;M} (Equation 2)

[0033] Where P contains positive pairs, N contains negative pairs, and M represents the cardinality of the entire triple set. The similarity function of two samples is expressed as l2 distance:

[0034] dist(x i ,x j )=||x i ;x j || 2 2 (Formula 3)

[0035] In some example implementations, cosine distance may alternatively be used.

[0036] In some example implementations, given an anchor x i , the hinge loss of a triplet can be defined as:

[0037] l(x i ,x + I,x - i )=[dist(x i ,x + i )-dist(x i ,x - i )+m] + (Formula 4)

[0038] where m is the margin imposed between positive and negative pairs. Hence, the global triplet loss L triplet The minimization over all possible triplets in the training set can be calculated as:

[0039] L triplet =Σ (xi,x+i,x-i)∈x l(x i ,x + i, x - i ) (Formula 5)

[0040] Finally, the total loss L for classification integrated with triplet learning is cls It can be expressed as:

[0041] L cls =βL binary +(1-β)L triplet (Equation 6)

[0042] Among them L binary is the binary cross entropy loss for traditional training of multi-label classification, and β is the coefficient.

[0043] In some example implementations, a disease location algorithm can be used to train a disease classifier to further improve the performance of disease identification and localization. For example, an existing image region localization algorithm such as CAM or GradCAM can be used to train a classifier. Both algorithms compute and generate a "heat map" from the network activations used for class or disease prediction, where regions in the image that support the classification have larger values. Since multi-label classification is performed, the algorithm (e.g., GradCAM, CAM, or other algorithms) can be run separately for each identified class. Figure 2 and Figure 3 The use of this localization to improve disease identification is discussed in more detail.

[0044] Figure 2A schematic representation 200 is shown of training an image classifier using local disease predictions according to an example implementation of the present application. In some example implementations, localized disease generation can be used to check whether the predicted disease location is consistent with the classifier. In these example implementations, if the image is incorrectly cropped, the classification may be incorrect. In other words, if the image has been cropped to exclude the diseased region, the disease classification may be incorrect.

[0045] Similar to About Figure 1 In the discussed model, the anchor image 105 (the image analyzed for training purposes) is passed through the spatial transformer network 111 to transform the image to a common image scale. The transformed image from the spatial transformer network 111 is then passed through the image feature embedding network 117 to embed image features for disease classification to generate an image embedding 123. As described below, the image embedding 123 includes disease location information that will be used by the disease locator 202 to embed disease location features.

[0046] Similar to Figure 1 Using the model described in

[15] , the image embedding 123 created from the anchor image 105 may then be passed through a classifier network 129 to compute a loss factor based on the labels assigned to the anchor image 105 in the training data. In other words, a multi-label loss factor 131 (also referred to as a whole image computed loss) is computed based on the difference between the assigned labels associated with the anchor image 105 from the training data and the classification of the anchor image 105 by the classifier network 129.

[0047] In addition to calculating the classifier loss (whole image classification loss 131) based on classifying the disease in the entire input image, an example implementation may crop the image using the predicted bounding box for each disease before classification. Specifically, as shown, the locator 202 determines the predicted location 206 of the disease and crops the image to the area of the predicted location of the disease to generate a local image embedding 204. Specifically, the locator 202 extracts regional features within the bounding box of the predicted location of the local disease in the anchor image 105. The feature values of the area outside the bounding box are set to 0. The modified features or local image embedding 204 represent the image within the bounding box and are input into the second classifier. An alternative approach would be to mask the non-disease area using, for example, a value of 0, and use the masked image as the input to the classifier. It should be noted that by selecting features (or features from localization) within the predicted bounding box, multiple layers of computation can be skipped, resulting in greater efficiency.

[0048] In some example implementations, a CAM localization algorithm may be used, where the activations of the last layer of the DenseNet are extracted. Alternatively, a GradCAM localization algorithm may be used in example implementations. Different localization methods may be used in different example implementations. In some example implementations, the maximum value over all features may be used for normalization. In other example implementations, the difference between the maximum and minimum values over all features may be used for normalization. In each case, locations of feature activations greater than a fixed threshold may be identified. A rectangular bounding box is calculated around the threshold activation, and features outside the bounding box are zeroed. The resulting feature map is then fed into a classifier 205, which is compared to the original classifier (see Figure 2 ) share weights. Alternatively, a separate classifier initialized with trained classifier parameters (classification layer only) can be used to classify the local image. In another alternative example implementation, the local region of the image determined by the localizer 202 can be determined from any layer of the image feature embedding network (not the feature embedding immediately before the classifier) representation (DENSENET representation).

[0049] In some example implementations, the loss 131 from whole image classification and the loss 208 from local image classification are combined as a weighted sum 212 using the following Equation 7:

[0050] L total =αL whole +(1-α)L local (Equation 7)

[0051] Where α is varied from 0 to 0.5 over multiple epochs so that the effect of local image loss gradually increases. For example, in some example implementations, α may be increased over 20 epochs. Thereafter, α may be kept constant.

[0052] In some example implementations, the trained model can be initialized using the parameters of a trained baseline model. The trained baseline model can be obtained by training for a fixed number of iterations or by training until the loss is asymptotic on a validation dataset. This initialization or pre-training is used so that in the first iteration, the estimated local regions are more likely to be correct, and therefore the local features are more likely to contain some information. If no initialization is performed, then in the initial iterations, the local region estimates are purely noise, and therefore the model gives very noisy local features, which will increase the difficulty during the early training iterations.

[0053] In some example implementations, Figure 2 The predicted local feature model shown can be compared with Figure 1 Specifically, the triplet model can be trained separately first to provide a pre-trained classification model, similar to pre-training a binary classifier. Then, Figure 1 The classifier network 129 in Figure 2 The two classifiers 129 and 205 and the localizer 202 shown are replaced. The loss will then be a combination of the whole image loss (multi-label loss 131), the local image loss 208, and the triplet loss 133. The combined model aims to learn a better feature space to distinguish multiple diseases.

[0054] However, in an alternative example implementation, the Figure 2 The predicted local feature model without Figure 1 In other example implementations, it can be used alone Figure 1 The triplet loss prediction model without Figure 2 The predicted local feature model.

[0055] Figure 3 A schematic representation 300 is shown of training an image classifier using local disease prediction according to other example implementations of the present application. Figure 3 The model is similar to Figure 2 A similar description is provided below for the model of . In some example implementations, local disease generation can be used to check whether the predicted location of the disease is consistent with the classifier. In these example implementations, if the image is not correctly cropped, the classification may be incorrect. In other words, if the image has been cropped to not include the disease area, the disease classification may be incorrect.

[0056] Similar to About Figure 1 and Figure 2 In the discussed model, the anchor image 105 (the image analyzed for training purposes) is passed through the spatial transformer network 111 to transform the image to a common image scale. The transformed image from the spatial transformer network 111 is then passed through the image feature embedding network 117 to embed image features for disease classification to generate an image embedding 123. As described below, the image embedding 123 includes disease location information that will be used by the disease locator 202 to embed disease location features.

[0057] Similar to About Figure 1 and Figure 2 Using the described model, the image embedding 123 created from the anchor image 105 may then be passed through a classifier network 129 to compute a loss factor based on the labels assigned to the anchor image 105 in the training data. In other words, a multi-label loss factor 131 (also referred to as a whole image computed loss) is computed based on the difference between the assigned labels associated with the anchor image 105 from the training data and the classification of the anchor image 105 by the classifier network 129.

[0058] In addition to calculating the classifier loss (whole image classification loss 131) based on classifying the disease in the entire input image, an example implementation may crop the image using the predicted bounding box for each disease before classification. Specifically, as shown, the locator 302 determines the predicted location of the disease and crops the image to the area of the predicted location of the disease to generate a local image embedding 304. Specifically, the locator 302 extracts regional features within the bounding box of the predicted location of the local disease in the anchor image 105. The feature values outside the bounding box are set to 0. The modified features or local image embedding 304 represent the image within the bounding box and are input into the second classifier. An alternative approach would be to mask the non-disease area using, for example, a value of 0, and use the masked image as the input to the classifier. It should be noted that by selecting features (or features from localization) within the predicted bounding box, multiple layers of computation can be skipped, resulting in greater efficiency.

[0059] In addition, Figure 3 In the model, the image embedding 304 can be fed back into the image feature embedding network 117 to improve the image embedding of the features extracted from the anchor image 105.

[0060] In some example implementations, a CAM localization algorithm may be used, where the activations of the last layer of the DenseNet are extracted. Alternatively, a GradCAM localization algorithm may be used in example implementations. Different normalization methods may be used in different example implementations. In some example implementations, the maximum value over all features may be used for normalization. In other example implementations, the difference between the maximum and minimum values over all features may be used for normalization. In each case, locations where feature activations are greater than a fixed threshold may be identified. A rectangular bounding box is calculated around the threshold activation, and features outside the bounding box are zeroed. The resulting feature map is then fed into a classifier 305, which is compared to the original classifier (see Figure 3 ) share weights. Alternatively, a separate classifier initialized with the trained classifier parameters (classification layer only) can be used to classify the partial image. The loss 131 from the entire image classification and the loss 308 from the partial image classification are combined as a weighted sum 312 using the following equation 8:

[0061] L total =αL whole +(1-α)L local (Equation 8)

[0062] Where α is varied from 0 to 0.5 over multiple epochs so that the effect of local image loss gradually increases. For example, in some example implementations, α may be increased over 20 epochs. Thereafter, α may be kept constant.

[0063] In some example implementations, the trained model can be initialized using the parameters of a trained baseline model. The trained baseline model can be obtained by training for a fixed number of iterations or by training until the loss on the validation dataset is asymptotic. This initialization or pre-training is used so that in the first iteration, the estimated local regions are more likely to be correct, and therefore the local features are more likely to contain some information. If initialization is not performed, then in the initial iterations, the local region estimates are purely noise, and therefore the model gives very noisy local features, which will increase the difficulty during the early training iterations.

[0064] In some example implementations, Figure 3 The predicted local feature model shown can be compared with Figure 1 Specifically, the triplet model can be trained separately first to provide a pre-trained classification model, similar to pre-training a binary classifier. Then, Figure 1 The classifier network 129 in Figure 3 The two classifiers 129 and 305 and the localizer 302 shown are replaced. The loss will then be a combination of the whole image loss (multi-label loss 131), the local image loss 308, and the triplet loss 133. The combined model aims to learn a better feature space to distinguish multiple diseases.

[0065] However, in an alternative example implementation, the Figure 3 The predicted local feature model without Figure 1 In other example implementations, it can be used alone Figure 1 The triplet loss prediction model without Figure 3 The predicted local feature model.

[0066] Figure 4 A flow chart illustrating a process 400 for classifying an image using a trained neural network according to an example implementation of the present application. The process 400 may be performed by a computing device in a computing environment. For example, Figure 9 The computing device 905 of the computing environment 900 can perform the process 400.

[0067] like Figure 4 As shown, at 405, a neural network is trained using labeled image samples. Training may involve using a plurality of labeled image samples labeled with image labels rather than image label location information (e.g., information identifying the region within the image where the image label is located). For example, the labeled image samples may be medical images from a publicly available collection of medical images (e.g., the NIH CHESTXRAY14 dataset) labeled with disease labels or other diagnostic information.

[0068] In some example implementations, the training of the neural network at 405 may involve: Figure 1 The image classifier is trained using triplet loss as shown in FIG. In other example implementations, the training of the neural network at 405 may involve the following: Figure 2 or Figure 3 In other example implementations, the training of the neural network at 405 may involve combining first Figure 1 The image classifier is trained using triplet loss as shown in Figure 2 or Figure 3 Shown using local disease prediction to train an image classifier.

[0069] After training the neural network, an image (e.g., a test data image) is selected for analysis at 410. In some example implementations, the selected test data image may be a new subject (e.g., a new patient) that is different from the subject of the images in the training image data and is suspected of having the same type of characteristics as the training (e.g., possibly the same disease). The selected test data image may be the same subject as the subject of the images in the training data (e.g., the same patient). Additionally, the test image data and the training image data may be two-dimensional slices of a three-dimensional image.

[0070] At 415, the trained neural network is applied to the selected image to determine an image feature classification associated with the selected image. For example, the neural network may analyze the selected image to determine a disease classification associated with the selected image. In some example implementations, applying the trained neural network to the selected image may involve using a special transformer network to transform the image to have a constant image orientation and a scale consistent with the training data used to train the image data.

[0071] Optionally, at 420, the trained neural network may also determine an image label location associated with the determined image feature classification (e.g., an area within the image where the image label is located). For example, the trained neural network may also detect an image region associated with the disease classification determined at 415.

[0072] The determined image feature classification may be provided or exported to a user at 425. For example, a disease classification report may be generated to be attached to a medical report associated with the selected medical image.

[0073] At 430, the determined image tag location (e.g., the area within the image where the image tag is located) can also be optionally provided or exported to the user. For example, a bounding box identifying the diseased area of the selected medical image can be attached to a medical report. After the image feature classification is provided to the user at 425, or after the determined image tag location is optionally provided or exported to the user at 430, process 400 can end.

[0074] Evaluation of example implementations

[0075]

[0076] Table 1: Summary of classification and localization performance of two baseline models, our model, and the Google model that, unlike other models, uses some labeled bounding boxes during training. The best performing model without labeled bounding boxes is shown in bold. The best performing model among all models is shown in bold italics.

[0077] For the evaluation of the example implementation, the NIH ChestXray14 dataset was used. For the triplet loss experiments, the task was multi-label classification of the 14 labeled diseases in the dataset. The second task was to use GradCAM to localize 8 diseases in the absence of local features in the dataset.

[0078] For the experiment using local features, the example implementation is compared with the binary model for three diseases. This should more clearly indicate whether the local features improve the performance of each disease and can not confuse the influence of other diseases. Although the binary model is used for the local feature experiment to check whether the local features are helpful, the method can be integrated into the STN+ triplet loss model. The disease is cardiac hypertrophy, mass and tubercle. The classification test set used for these is identical to the test data of the three diseases in the multi-label classification. The localization test set is a subset of the test data with the bounding box of the mark.

[0079] STN and triplet loss results

[0080] Two baseline models, ChestX-ray14(NIH) and CheXNext, were used to compare with the example implementation model (STN+Triplet). The baseline model ChestX-ray14(NIH) was compared using four image analysis models (i.e., AlexNet, GoogLeNet, VGGNet-16, and ResNet-50) to compute image representations. CheXNext showed that using DenseNet121 outperformed the model used by ChestX-ray14(NIH). The two baseline models performed disease classification and then performed localization. The example implementation model was also compared with the Google model [2], which used 70% of the bounding box labels to train its localization model. Bounding box labels are difficult to obtain because doctors need to manually label them, which is beyond common practice.

[0081] Table 1 shows the classification and localization performance of two baseline models, ChestXray14 and CheXNext, and the Google model using additional data. Among the models that do not use some labeled bounding boxes during training, the example implementation model (STN+Triplet) performs best overall for both classification and localization. In addition, the example implementation model also demonstrates performance close to that of the Google model, which utilizes more labeled data during training.

[0082] Figure 5 and Figure 6 More detailed performance of the four models for each disease is shown in . Figure 5 Shown are the classification accuracy of the baseline model, the example implementation model, and the Google model using some labeled bounding boxes during training. Figure 6 Shown are the localization performance measured using IOU for a baseline model, an example implementation model, and a Google model that use some labeled bounding boxes during training.

[0083] Training using local features

[0084] For these experiments, the effectiveness of training with local features was compared to a baseline model consisting of a DenseNet121 and a single-layer disease classifier. Separate binary classifiers were created for each disease to evaluate whether using local features alongside features from the entire image improved performance.

[0085]

[0086] Table 2: Classification accuracy of the baseline model and models that compute local features using two different methods of normalizing activation maps. The best tested model accuracy is in bold.

[0087] Table 2 above shows the classification accuracy of the baseline model and two models using local features calculated using two different normalization methods for different diseases. For accuracy assessment, when using local features with any normalization method across all three diseases, consistent improvements in test accuracy were noted. In contrast, the best performing validation accuracy was the baseline model, but the best validation performance did not translate into the best test performance. The performance difference between validation and testing of the two models using local features was smaller than that of the baseline. The smaller difference may indicate that local features help the model generalize better to new data.

[0088]

[0089] Table 3: Localization test performance of different models evaluated as the percentage of intersection over union (IoU) greater than a given threshold. The best model localization performance is in bold.

[0090] Table 3 compares the localization performance of the same models as in Table 2. With an IoU threshold of 0.1, the baseline model outperforms both local feature models only for cardiac hypertrophy. For three cases, the norm1 method normalized by the minimum activation value outperforms the baseline, while the norm2 method outperforms the baseline in two cases.

[0091] From the results in Tables 2 and 3, it is noted that using the predicted local features consistently improves test classification accuracy across the three tested diseases. Furthermore, the local feature model slightly outperforms the baseline model: each has the best localization performance for two diseases, one of which has an IoU threshold of 0.5, a more stringent criterion requiring higher overlap. The baseline model has the best performance for one disease, which has a less stringent criterion with an IoU threshold of 0.1.

[0092] Qualitative results

[0093] Figure 7 and Figure 8 An example heatmap showing the use of a multi-label triplet model and the predicted (red box) and true (green box) locations of the disease. Figure 7 An example is shown when the probability of predicting the true disease is greater than 0.5 and there is at least 0.5 IoU between the true and predicted disease bounding boxes. Figure 8 Examples of incorrect predictions are shown. The first two columns show cases where the bounding box is correct but the predicted probability of the disease is less than 0.5 (incorrect). The third column shows cases where the bounding box is incorrect and the predicted probability of the disease may or may not be greater than 0.5.

[0094] Sample computing environment

[0095] Figure 9 An example computing environment 900 is shown with an example computing device 905 suitable for use with some example implementations. The computing device 905 in the computing environment 900 may include one or more processing units, cores, or processors 910, memory 915 (e.g., RAM, ROM, etc.), internal storage 920 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 925, any of which may be coupled on a communication mechanism or bus 930 for communicating information or embedded in the computing device 905.

[0096] The computing device 905 may be communicatively coupled to an input / interface 935 and an output device / interface 940. Either or both of the input / interface 935 and the output device / interface 940 may be wired or wireless interfaces and may be detachable. The input / interface 935 may include any device, component, sensor, or interface (physical or virtual) that can be used to provide input (e.g., buttons, a touch screen interface, a keyboard, a pointing / cursor control, a microphone, a camera, Braille, a motion sensor, an optical reader, etc.).

[0097] Output devices / interfaces 940 may include displays, televisions, monitors, printers, speakers, Braille, and the like. In some example implementations, input / interfaces 935 (e.g., user interfaces) and output devices / interfaces 940 may be embedded in or physically coupled to computing device 905. In other example implementations, other computing devices may serve as or provide functionality for input / interfaces 935 and output devices / interfaces 940 for computing device 905. These elements may include, but are not limited to, well-known AR hardware inputs to allow a user to interact with an AR environment.

[0098] Examples of computing device 905 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, server devices, other computers, kiosks, televisions embedded with and / or coupled to one or more processors, radios, etc.).

[0099] The computing device 905 can be communicatively coupled to external storage 945 and a network 950 (e.g., via I / O interface 925) for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. The computing device 905 or any connected computing device can function as, provide services for, or be referred to as a server, client, thin server, general-purpose machine, special-purpose machine, or another label.

[0100] I / O interface 925 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11xs, universal system bus, WiMAX, modems, cellular network protocols, etc.) for communicating information to and / or from at least all connected components, devices, and networks in computing environment 900. Network 950 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0101] The computing device 905 may use and / or communicate with computer-usable or computer-readable media (including transitory and non-transitory media). Transitory media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., magnetic disks and tapes), optical media (e.g., CD ROMs, digital video disks, Blu-ray discs), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage devices), and other non-volatile storage devices or memories.

[0102] The computing device 905 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. The computer-executable instructions can be retrieved from transitory media, as well as stored on and retrieved from non-transitory media. The executable instructions can originate from one or more of any programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0103] The processor 910 can execute under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, which include a logic unit 955, an application programming interface (API) unit 960, an input unit 965, an output unit 970, a spatial transformer 975, a feature embedder 980, an image classifier 985, a localizer 990, and an inter-unit communication mechanism 995 for different units to communicate with each other, with the OS, and with other applications (not shown).

[0104] For example, the output unit 970, the spatial transformer 975, the feature embedder 980, the image classifier 985, and the localizer 990 may implement Figure 6 One or more of the processes shown and implemented Figures 1 to 4 The design, functionality, configuration or implementation of the units and elements described may vary and are not limited to the description provided.

[0105] In some example implementations, when information is received or instructions are executed by the API unit 960, they may be communicated to one or more other units (e.g., the spatial transformer 975, the feature embedder 980, the image classifier 985, and the localizer 990). For example, the spatial transformer 975 may transform one or more images to rotate and scale the images to a common orientation. The feature embedder 980 may extract and embed image features from the transmitted image. The image classifier 985 may classify features of the transformed image based on the embedded image features. The localizer 990 determines the image location associated with the embedded image features to assist in image classification.

[0106] In some cases, in some of the example implementations described above, the logic unit 955 can be configured to control the flow of information between the units and direct the services provided by the API unit 960, the spatial transformer 975, the feature embedder 980, the image classifier 985, and the localizer 990. For example, the flow of one or more processes or implementations can be controlled by the logic unit 955 alone or in combination with the API unit 960.

[0107] Although some example implementations have been shown and described, these example implementations are provided in order to convey the subject matter described herein to those familiar with the art. It should be understood that the subject matter described herein can be implemented in various forms and is not limited to the example implementations described. The subject matter described herein can be practiced without those specifically defined or described matters or with other or different elements or matters that are not described. Those familiar with the art will understand that these exemplary implementations can be changed without departing from the subject matter defined in the appended claims and their equivalents as described herein.

Claims

1. A method for classifying image features using a neural network, the method comprising the following steps: The neural network is trained using triplet loss processing, and the steps of training the neural network include: receiving an anchor image and an image label associated with the anchor image, the anchor image to be used for training the neural network; selecting a positive image and an image tag associated with the positive image, the positive image having at least one image tag shared with the anchor image; selecting a negative image and an image label associated with the negative image, the negative image having no image label shared with the anchor image; generating an image embedding associated with each of the anchor image, the positive image, and the negative image using an image feature embedding network, the image embedding being embedded with image features for disease classification; classifying, by the neural network, image features extracted from the anchor image based on the image embedding of the anchor image; computing a multi-label loss based on the classified image features extracted from the anchor image associated with the image embedding and the image label associated with the anchor image; computing a triplet loss based on the image embedding associated with each of the anchor image, the positive image, and the negative image to determine a classification weight for the image features; Using a weighted loss and combining the multi-label loss with the calculated triplet loss; and using the trained neural network to classify image features of a test data image without any associated labeled features.

2. The method according to claim 1, wherein The step of training the neural network further comprises: A spatial transformer network is used to transform each of the anchor image, the positive image, and the negative image by scaling, rotating, and cropping to establish a common view orientation.

3. The method according to claim 1, wherein The step of training the neural network further comprises: determining image label locations associated with the classified image features; extracting features associated with the determined image tag locations; classifying the features associated with the determined image label locations using the neural network; determining a local image classification loss associated with the features associated with the determined image label locations; and The multi-label loss is combined with the local image classification loss and the triplet loss using a weighted loss sum.

4. The method of claim 3, further comprising identifying image label locations on test image data using the trained neural network.

5. The method according to claim 3, wherein The step of training the neural network further comprises: generating a new image embedding for the anchor image based on the extracted features associated with the determined image tag positions; and The features extracted from the anchor image are classified based on the new image embedding by the neural network.

6. The method according to claim 3, wherein: The anchor image, the positive image, and the negative image are each a medical diagnostic image associated with one or more diseases.

7. A method for classifying image features using a neural network, the method comprising the following steps: The neural network is trained using triplet loss processing, and the steps of training the neural network include: receiving an anchor image and an image label associated with the anchor image, the anchor image to be used for training the neural network; generating an image embedding associated with the anchor image using an image feature embedding network, the image embedding being embedded with image features for disease classification; classifying image features extracted from the anchor image based on the image embedding of the anchor image by the neural network; determining image label locations associated with the classified image features; extracting features associated with the determined image tag locations; classifying the features associated with the determined image tag locations using the neural network; and determining a local image classification loss associated with features associated with the determined image label locations; The trained neural network is used to classify image features of test data images that do not have any associated labeled features.

8. The method according to claim 7, wherein: The step of training the neural network further comprises: computing a multi-label loss based on the classified image features extracted from the anchor image associated with the image embedding and the image label associated with the anchor image; and The multi-label loss is combined with the local image classification loss using a weighted loss sum.

9. The method of claim 7, further comprising identifying image label locations on test image data using the trained neural network.

10. The method according to claim 8, wherein The step of training the neural network further comprises: generating a new image embedding for the anchor image based on the extracted features associated with the determined image tag positions; and The features extracted from the anchor image are classified based on the new image embedding by the neural network.

11. The method according to claim 7, wherein: The step of training the neural network further comprises: The anchor images are transformed using a spatial transformer network by scaling, rotating, and cropping to establish a common view orientation.

12. A non-transitory computer-readable medium encoded with instructions for causing a computer to perform a method for classifying image features using a neural network, the method comprising the steps of: The neural network is trained using triplet loss processing, and the steps of training the neural network include: receiving an anchor image and an image label associated with the anchor image, the anchor image to be used for training the neural network; selecting a positive image and an image tag associated with the positive image, the positive image having at least one image tag shared with the anchor image; selecting a negative image and an image label associated with the negative image, the negative image having no image label shared with the anchor image; generating an image embedding associated with each of the anchor image, the positive image, and the negative image using an image feature embedding network, the image embedding being embedded with image features for disease classification; classifying, by the neural network, image features extracted from the anchor image based on the image embedding of the anchor image; and determining image label locations associated with the classified image features; extracting features associated with the determined image tag locations; classifying the features associated with the determined image label locations using the neural network; determining a local image classification loss associated with features associated with the determined image label locations; calculating a triplet loss based on the image embedding associated with each of the anchor image, the positive image, and the negative image to determine a classification weight for the image features; and combining the local image classification loss and the triplet loss using a weighted loss; The trained neural network is used to classify image features of test data images that do not have any associated labeled features.

13. The non-transitory computer-readable medium of claim 12, wherein: The step of training the neural network further comprises: computing a multi-label loss based on the classified image features extracted from the anchor image associated with the image embedding and the image label associated with the anchor image; and The multi-label loss is combined with the local image classification loss and the triplet loss using a weighted loss sum.

14. The non-transitory computer-readable medium of claim 13, wherein: The step of training the neural network further comprises: A spatial transformer network is used to transform each of the anchor image, the positive image, and the negative image by scaling, rotating, and cropping to establish a common view orientation.

15. The non-transitory computer-readable medium of claim 13, wherein: The method also includes identifying image label locations on test image data using the trained neural network.

16. The non-transitory computer-readable medium of claim 15, wherein: The step of training the neural network further comprises: generating a new image embedding for the anchor image based on the extracted features associated with the determined image tag positions; and The features extracted from the anchor image are classified based on the new image embedding by the neural network.

17. The non-transitory computer-readable medium of claim 16, wherein: The anchor image, the positive image, and the negative image are each a medical diagnostic image associated with one or more diseases.

Images