Learning device, learning method, and program

Abstract

A training device 1X mainly includes a feature amount acquisition means 33X and a training means 36X. The feature amount acquisition means 33X acquires: a feature amount of an abnormal region of a training image which includes the abnormal region and which is an image used for training an abnormality detection model that is a machine learning model; and a feature amount of an abnormality-related region which is a region determined on the basis of the abnormal region. The training means 36X trains the abnormality detection model on the basis of the degree of similarity between the feature amount of the abnormal region and the feature amount of the abnormality-related region. The invention can be used to assist a user in decision making, etc.

Description

Learning device, learning method, and storage medium

[0001] The present disclosure relates to the technical fields of a learning device, a learning method, and a storage medium for learning a model that performs inference regarding anomaly detection.

[0002] Techniques relating to machine learning of object detection models have been known for some time. For example, Non-Patent Document 1 discloses Feature Pyramid Networks, which is a neural network architecture that performs inference using a multi-scale feature map. Furthermore, Patent Document 1 discloses a technique for detecting a suspected lesion region from an intracavity image obtained by capturing an image of an organ in a living body using a machine learning model.

[0003] Japanese Patent Application Laid-Open No. 2008-093172

[0004] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection.In CVPR, 2017.

[0005] When training an anomaly detection model for anomaly detection, it is difficult to train the anomaly detection model accurately when the abnormal region to be detected is similar to other regions or when there is insufficient training data.

[0006] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide a learning device, a learning method, and a storage medium capable of learning an anomaly detection model that performs anomaly detection with high accuracy.

[0007] One aspect of the learning device is a learning device having: a feature acquisition means for acquiring features of an abnormal region of a learning image containing an abnormal region, the image being used for learning an anomaly detection model, and features of an anomaly-related region that is a region defined based on the abnormal region; and a learning means for learning the anomaly detection model based on the degree of similarity between the features of the abnormal region and the features of the anomaly-related region.

[0008] One aspect of the learning method is a learning method in which a computer acquires features of an abnormal region of a learning image, the image being an image used for learning an anomaly detection model, the abnormal region including the abnormal region, and features of an anomaly-related region that is an area defined based on the abnormal region, and learns the anomaly detection model based on the degree of similarity between the features of the abnormal region and the features of the anomaly-related region.

[0009] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute a process of acquiring features of an abnormal region of a learning image that includes an abnormal region, the image being used for training an anomaly detection model, and acquiring features of an anomaly-related region that is a region defined based on the abnormal region, and training the anomaly detection model based on the degree of similarity between the features of the abnormal region and the features of the anomaly-related region.

[0010] As an example of an effect of the present disclosure, it is possible to learn an anomaly detection model that performs anomaly detection with high accuracy.

[0011] 1 shows a schematic configuration of a learning system; FIG. 2 shows a hardware configuration of a learning device; FIG. 3 is a functional block diagram of a learning device related to learning of a lesion detection model; FIG. 4 is a diagram showing an outline of processing performed in a model execution unit; FIG. 5 shows a learning image clearly indicating the processing result by the feature acquisition unit when a lesion peripheral region is selected based on anchor boxes; FIG. 6 is an example of a flowchart showing an overview of processing performed by the learning device in the first embodiment; (A) shows a learning image clearly indicating the correspondence between each block in the layer where the most similar block exists and the learning image; (B) shows a learning image clearly indicating the correspondence between each block in the layer where the upper layer corresponding block exists and the learning image; FIG. 7 is a block diagram of a learning device in the second embodiment; and FIG. 8 is an example of a flowchart executed by the learning device in the second embodiment.

[0012] Hereinafter, embodiments of a learning device, a learning method, and a storage medium will be described with reference to the drawings.

[0013] <First embodiment> (1) System configuration Fig. 1 shows a schematic configuration of a learning system 100. As shown in Fig. 1, the learning system 100 is a system that learns a model for detecting an abnormality from an image of a subject in which an abnormality is to be detected, and mainly includes a learning device 1 and a storage device 2.

[0014] Hereinafter, as a representative example, learning of a lesion detection model that detects image regions suspected of having a lesion (also referred to as "lesion regions") from endoscopic images captured during an endoscopic examination using a human subject will be described. Examples of endoscopes that are the subject of the present disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, vascular endoscopes, and epidural endoscopes. Note that the model to be learned is not limited to the lesion detection model described above, and may be a model that detects lesion regions from any medical image other than an endoscope, or a model that detects suspected abnormal regions from images captured during a visual inspection of a non-human subject (visual inspection images).

[0015] The learning device 1 learns (trains) the lesion detection model based on the learning data D1 stored in the storage device 2, and updates the model information D2 stored in the storage device 2.

[0016] The storage device 2 is a memory that stores various information necessary for the processing of the learning device 1. The storage device 2 has learning data D1 and model information D2.

[0017] The learning data D1 is learning (training) data used for learning by the learning device 1. The learning data D1 has multiple records, and each record includes a learning image, which is an image to be input to the lesion detection model during learning, and correct answer data indicating the correct answer to be output by the lesion detection model when the learning image is input to the lesion detection model. In the present embodiment, as an example, the lesion detection model outputs an inference result including region information indicating a detected region in the image input to the lesion detection model and a class into which the region is classified, and the correct answer data indicates the correct region information of the lesion region in the learning image and the correct class into which the lesion region is classified. Here, the "region information of the region" and the "correct region information of the lesion region" indicate bounding boxes surrounding the region. Furthermore, the "class into which the region is classified" and the "correct class into which the lesion region is classified" indicate the class (category) into which the region indicated by the paired region information is classified, and may indicate whether or not it is a lesion region, or may indicate the type of lesion if it is a lesion region in addition to whether or not it is a lesion region.

[0018] The model information D2 is information about the lesion detection model and includes parameters that are updated by learning performed by the learning device 1. Here, the lesion detection model is a model (engine) that, when an endoscopic image is input to the lesion detection model, outputs an inference result regarding the lesion area in the input image. In other words, it is a model that has learned the relationship between the input image (i.e., the endoscopic image) and the lesion area in that image. The inference result output by the lesion detection model includes, for example, area information indicating a bounding box surrounding the detected area, a class indicating the classification result of the area within the bounding box, and a confidence level indicating the likelihood of the class classification.

[0019] The lesion detection model is a machine learning model (including a statistical model, the same applies hereinafter) having any architecture, such as a neural network or a support vector machine. A typical example of a neural network used in the architecture of the lesion detection model is a Feature Pyramid Network. Examples of architectures that the lesion detection model has include Feature Pyramid Network, as well as Featured image pyramid, Fully Convolutional Network, SegNet, U-Net, V-Net, Mask R-CNN, DeepLab, AlexNet, VGG, ResNet, SqueezeNet, DenseNet, Inception, GoogleNet, ShuffleNet, MobileNet, ResNeXt, Wide ReNet, and NASNet. When the lesion analysis model is constructed using a neural network, the model information D2 includes various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number of filters and filter size in each layer, and the weight of each element of each filter.

[0020] Examples of pathologies that can be detected by the lesion detection model include the following (a) to (f): (a) head and neck: pharyngeal cancer, malignant lymphoma, papilloma; (b) esophagus: esophageal cancer, esophagitis, hiatal hernia, Barrett's esophagus, esophageal varices, esophageal achalasia, esophageal submucosal tumor, benign esophageal tumor; (c) stomach: gastric cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor; (d) duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma; (e) small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease; (f) large intestine: large intestine cancer, large intestine neoplastic disease, large intestine inflammatory disease, large intestine polyp, large intestine polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids.

[0021] The storage device 2 may be an external storage device such as a hard disk connected to or built into the learning device 1, a storage medium such as a flash memory, or a server device that communicates data with the learning device 1. The storage device 2 may also be composed of multiple storage devices, and may have the above-mentioned storage units distributed among them.

[0022] The configuration of learning system 100 shown in Figure 1 is an example, and various modifications may be made. For example, learning device 1 and storage device 2 may be implemented as a single device. In another example, learning device 1 may be implemented as a plurality of devices. In this case, the plurality of devices that make up learning device 1 exchange information necessary to execute pre-assigned processing between them via direct wired or wireless communication or via communication via a network.

[0023] (2) Hardware Configuration Fig. 2 shows an example of the hardware configuration of the learning device 1. The learning device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 19.

[0024] The processor 11 executes a program stored in the memory 12 to function as a controller (arithmetic unit) that performs overall control of the learning device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0025] Memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Memory 12 also stores programs executed by learning device 1. Some of the information stored in memory 12 may be stored in an external storage device, such as storage device 2, that can communicate with learning device 1, or in a storage medium that is detachable from learning device 1. Memory 12 may also store information stored in storage device 2 instead.

[0026] The interface 13 is an interface for electrically connecting the learning device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0027] (3) Overview of Lesion Detection Model Learning Next, the learning of the lesion detection model will be described in detail. In summary, the learning device 1 calculates a loss based on the error between the ground truth data and the inference result output by the lesion detection model when a learning image is input to the lesion detection model, as well as a loss based on the degree of similarity in feature quantities between the lesion area identified by the ground truth data and the area surrounding the lesion area. The learning device 1 then updates the parameters of the lesion detection model to optimize the loss obtained by combining the two losses described above. This allows the learning of a lesion detection model that can accurately detect lesion areas whose features are similar to those of the surrounding areas.

[0028] Hereinafter, the loss corresponding to the error between the ground truth data and the inference result output by the lesion detection model when a training image is input to the lesion detection model will be referred to as the "first loss L1." Furthermore, the loss based on the similarity of the features between the lesion area identified by the ground truth data and the area surrounding the lesion area (also referred to as the "lesion periphery area") will be referred to as the "second loss L2." Furthermore, the loss obtained by combining the first loss L1 and the second loss L2 will be referred to as the "combined loss L."

[0029] Figure 3 is a functional block diagram of the learning device 1 related to learning of a lesion detection model. The processor 11 of the learning device 1 functionally includes a model execution unit 31, a first loss calculation unit 32, a feature acquisition unit 33, a second loss calculation unit 34, an integrated loss calculation unit 35, and a learning unit 36. Note that in Figure 3, blocks that exchange data are connected by solid lines, but the combination of blocks that exchange data is not limited to that shown in Figure 3. The same applies to other functional block diagrams described below.

[0030] The model execution unit 31 extracts training images to be used for training from the training data D1 and inputs them into a lesion detection model based on the model information D2, thereby acquiring an inference result output by the lesion detection model. The model execution unit 31 supplies the acquired inference result to the first loss calculation unit 32. The model execution unit 31 also supplies intermediate data (e.g., information related to features, anchor boxes, etc.) generated in the lesion detection model before acquiring the inference result to the feature acquisition unit 33.

[0031] 4 is a diagram illustrating the process performed by the model execution unit 31. The model execution unit 31 inputs a training image to a lesion detection model to which parameters included in model information D2 have been applied. In this case, the lesion detection model outputs a set of region information indicating a bounding box surrounding the detected region, the class of the detected region, and a confidence level regarding the classification of that class. Here, the lesion detection model outputs an inference result including a set of region information A1, class B1, and confidence level C1, and a set of region information A2, class B2, and confidence level C2. This inference result is supplied to the first loss calculation unit 32 and used to calculate the first loss L1.

[0032] The lesion detection model also includes at least a feature extraction unit 311 and an anchor box generation unit 312 .

[0033] The feature extraction unit 311 generates feature amounts of the training images input to the lesion detection model. For example, if the lesion detection model has a network architecture based on Feature Pyramid Networks, the feature extraction unit 311 performs a convolution operation on the training images, which are input images, to generate a multi-scale (i.e., multiple layers with different granularities) feature map. Note that the feature map is a map of feature amounts (feature vectors) and corresponds to a tensor obtained by convolution.

[0034] The anchor box generation unit 312 places anchor boxes of multiple sizes around each anchor point placed on the feature map. These anchor boxes correspond to a set of prior distribution candidates for bounding box prediction. In this case, the anchor box generation unit 312 outputs information about the anchor boxes (including the feature values ​​and class probabilities corresponding to the anchor boxes). Note that, as will be described later, if the lesion peripheral region is defined in block units on the feature map, the anchor box generation unit 312 is not essential.

[0035] The processing results of the feature extraction unit 311 and the anchor box generation unit 312 are supplied as intermediate outputs to the feature acquisition unit 33 and the feature acquisition unit 33 .

[0036] Referring again to FIG. 3, the functional blocks of the learning device 1 related to learning the lesion detection model will be described.

[0037] The first loss calculation unit 32 obtains, from the training data D1, ground truth data corresponding to the training images input to the lesion detection model, and calculates a first loss L1 corresponding to the error between the obtained ground truth data and the inference result output by the lesion detection model supplied from the model execution unit 31. In this case, the first loss L1 may be calculated based on any loss function, such as the mean squared error, the mean absolute error, the square root of the mean squared error, the mean squared logarithmic error, the Huber loss, the Poisson loss, the hinge loss, or the Kullback-Leibler divergence. The first loss calculation unit 32 supplies the calculated first loss L1 to the integrated loss calculation unit 35.

[0038] Based on the correct answer data, the feature acquisition unit 33 identifies the lesion area and the surrounding area in the learning image input to the lesion detection model, and acquires the feature amounts corresponding to the identified lesion area and surrounding area.

[0039] First, a method for identifying the lesion peripheral region will be described. For example, the feature amount acquisition unit 33 identifies the lesion peripheral region in units of blocks for which feature amounts are defined in the feature map. In another example, the feature amount acquisition unit 33 selects anchor boxes that will become the lesion peripheral region from the anchor boxes generated by the anchor box generation unit 312. In these examples, the feature amount acquisition unit 33 identifies, as the lesion peripheral region, blocks or anchor boxes that satisfy predetermined criteria for determining the lesion peripheral region.

[0040] In a first example of the criteria for determining the lesion peripheral area, the feature acquisition unit 33 determines, as the lesion peripheral area, a block or anchor box whose distance from the bounding box of the lesion area based on the ground truth data is within a predetermined threshold. The above-mentioned "distance" may be the Lp distance (including Euclidean distance) on the image, or a value based on any other distance function. Information regarding the above-mentioned "predetermined threshold" is pre-stored, for example, in the storage device 2 or memory 12. In this way, the feature acquisition unit 33 determines the block or anchor box that will become the lesion peripheral area based on its physical proximity to the lesion area.

[0041] In this case, preferably, the feature acquisition unit 33 identifies a block or anchor box corresponding to the lesion area, and then determines a lesion peripheral area from blocks or anchor boxes other than the block or anchor box identified as the lesion area. In this case, the feature acquisition unit 33 determines, as the lesion area, a block or anchor box that is determined to substantially match the bounding box of the lesion area indicated by the correct answer data. In this case, for example, the feature acquisition unit 33 determines, as the lesion area, a block or anchor box whose evaluation index (also referred to as a "match evaluation index") for the degree of match between the areas, such as IoU (Intersection over Union), is equal to or greater than a predetermined threshold. Hereinafter, for ease of explanation, it is assumed that the larger the value of the match evaluation index, the higher the degree of match between the two compared areas.

[0042] In a second example of the criteria for determining a peripheral lesion area, feature acquisition unit 33 defines, as a peripheral lesion area, a block or anchor box for which a match evaluation index such as IoU with a lesion area (specifically, a bounding box of a lesion area) based on ground truth data falls within a predetermined range. The above-mentioned "predetermined range" is defined to be a range of values ​​of the match evaluation index that does not correspond to a lesion area and is determined to be close to a lesion area, and the defined range is stored in advance in storage device 2, memory 12, or the like.

[0043] Next, a supplementary explanation will be given regarding the acquisition of feature amounts. Feature amount acquisition unit 33 acquires feature amounts corresponding to the lesion area and the lesion-periphery area from the feature extraction results output by feature extraction unit 311. If the lesion area and the lesion-periphery area are identified by anchor boxes, feature amount acquisition unit 33 may acquire feature amounts associated with the anchor boxes from anchor box generation unit 312. Then, feature amount acquisition unit 33 supplies the feature amounts of the lesion area and the lesion-periphery area to second loss calculation unit 34.

[0044] The second loss calculation unit 34 calculates the second loss L2 based on the degree of similarity (also referred to as "similarity S") between the feature amounts of the lesion area and the feature amounts of the perilesional area. The similarity S may be any index that measures the similarity of the feature amounts, such as the cosine distance, the inverse of the Lp distance, or the inverse of KL-divergence. In the following description, the similarity S will be assumed to have a higher value the higher the degree of similarity. Note that the second loss calculation unit 34 may calculate an index that has a lower value the higher the degree of similarity, instead of the similarity S.

[0045] In this case, the second loss calculation unit 34 calculates the second loss L2 from the similarity S based on a predetermined equation, a lookup table, or the like that defines the relationship between the similarity S and the second loss L2 so that the similarity S and the second loss L2 have a negative correlation. As a result, the second loss calculation unit 34 reduces the second loss L2 as the similarity S decreases. For example, the second loss calculation unit 34 calculates the second loss L2 using the similarity S as a loss function as follows: L2 = S Note that if an index that takes a lower value as the degree of similarity increases is calculated instead of the similarity S, the second loss calculation unit 34 calculates the second loss L2 using, for example, a loss function that takes the reciprocal of the similarity S.

[0046] Here, a supplementary explanation will be given on the calculation of the similarity S when there are N lesion peripheral regions (N is an integer of 2 or more).

[0047] In this case, in a first calculation example of the similarity S, the second loss calculation unit 34 calculates the similarity between the lesion area and each of the multiple lesion-peripheral areas (i.e., N similarities), and uses the sum of the N similarities as the similarity S used in calculating the second loss L2. In a second example, the second loss calculation unit 34 calculates the similarity between the lesion area and each of the multiple lesion-peripheral areas (i.e., N similarities), and uses a representative value (average, minimum, median, etc.) of the N similarities as the similarity S used in calculating the second loss L2. For example, when the maximum value is used as the representative value, the similarity between the lesion area and the lesion-peripheral area that is most similar to the lesion area is adopted as the similarity S.

[0048] The integrated loss calculation unit 35 calculates the integrated loss L by integrating (adding up) the first loss L1 calculated by the first loss calculation unit 32 and the second loss L2 calculated by the second loss calculation unit 34. In this case, the integrated loss calculation unit 35 calculates the integrated loss L from the first loss L1 and the second loss L2 based on an equation, a lookup table, or the like that defines the relationship between the first loss L1 and the second loss L2 and the integrated loss L so that there is a positive correlation between the first loss L1, the second loss L2, and the integrated loss L. In this case, when the second loss L2 is fixed, the integrated loss L increases as the first loss L1 increases, and when the first loss L1 is fixed, the integrated loss L increases as the second loss L2 increases.

[0049] For example, if the contribution rate of the second loss L2 to the integrated loss L is "α", the integrated loss calculation unit 35 calculates the integrated loss L using the following formula: L = (1 - α) L1 + α L2 Note that the integrated loss calculation unit 35 is not limited to the above formula, and may determine the integrated loss L from the first loss L1 and the second loss L2 based on any formula or lookup table that defines the relationship between the first loss L1 and the second loss L2 and the integrated loss L.

[0050] The learning unit 36 ​​determines the parameters of the lesion detection model using a gradient descent method, an error backpropagation method, or the like so as to optimize the integrated loss L. More preferably, the learning unit 36 ​​determines the parameters of the lesion detection model so as to minimize the integrated loss L. The learning unit 36 ​​then updates the model information D2 by reflecting the determined parameters of the lesion detection model in the model information D2. The learning unit 36 ​​updates the model information D2 for each record in the training data D1.

[0051] Each of the components, namely, the model execution unit 31, the first loss calculation unit 32, the feature acquisition unit 33, the second loss calculation unit 34, the integrated loss calculation unit 35, and the learning unit 36, can be realized, for example, by the processor 11 executing a program. Alternatively, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as needed. At least a portion of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least a portion of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. At least a portion of the components may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0052] (4) Selection of Lesion Periphery Region Next, a specific example of selection of the lesion peripheral region by the feature amount acquisition unit 33 will be described.

[0053] Figure 5 shows a training image that clearly shows the processing results by feature acquisition unit 33 when a lesion perifacial region is selected based on an anchor box. The training image shown in Figure 5 clearly shows anchor boxes generated by anchor box generation unit 312 and bounding boxes of lesion regions based on ground truth data. Here, the bounding boxes of lesion regions based on ground truth data are represented by solid lines, anchor boxes identified as lesion perifacial regions by feature acquisition unit 33 are represented by dashed lines, and anchor boxes that are regions other than the lesion region and the lesion perifacial region (also referred to as "other regions") are represented by dashed lines.

[0054] In the example of Figure 5, the feature acquisition unit 33 recognizes the bounding box of the lesion area (which may be the anchor box with the highest matching evaluation index) based on the ground truth data. The feature acquisition unit 33 also identifies anchor boxes surrounding the bounding box of the lesion area as the lesion peripheral area based on an index (distance or matching evaluation index) that measures their physical proximity to the bounding box of the lesion area. Although the anchor boxes in Figure 5 do not overlap each other, in reality, many overlapping anchor boxes are generated.

[0055] In this way, the learning device 1 identifies a lesion-periphery area that is physically close to the lesion area, and learns a lesion detection model using the second loss L2 based on the similarity S between the features of the identified lesion-periphery area and the features of the lesion area, thereby making it possible to promote the learning of features of positive cases.

[0056] The effect of selecting a peripheral lesion region will be further explained. Generally, the number of anchor boxes classified as lesion regions (i.e., positive examples) is overwhelmingly smaller than the number of anchor boxes classified as non-lesion regions (i.e., negative examples). In this case, the contribution of negative examples (especially the contribution of negative examples corresponding to the "other regions" whose features differ significantly from those of the lesion region) is large in the calculation of the first loss L1. Therefore, if the parameters of a lesion detection model are trained to optimize only the first loss L1, negative examples will be trained more favorably than positive examples, resulting in a problem of poor progress in learning the features of positive examples. Furthermore, some lesions, such as flat lesions (superficial lesions), lack distinctive global color and shape compared to their surroundings and therefore have few training samples due to their low prevalence and low case numbers. In this case, a lesion detection model that does not adequately learn the features of positive examples may be generated.

[0057] Taking the above into consideration, in this embodiment, the learning device 1 identifies as a perilesional region a region in which the lesion detection model is particularly desired to learn differences in features from the lesion region, and calculates a second loss L2 based on the similarity S of the feature amounts between the lesion region and the perilesional region. The learning device 1 then uses the second loss L2 related to the features of positive examples in addition to the first loss L1 to train the lesion detection model so as to optimize the integrated loss L obtained by integrating the first loss L1 and the second loss L2. This enables the learning device 1 to train the lesion detection model so as to effectively balance positive examples and negative examples within a single image.

[0058] (5) Processing Flow FIG. 6 is an example of a flowchart showing an outline of the processing executed by the learning device 1 in the first embodiment.

[0059] First, the learning device 1 acquires a pair of a learning image and correct answer data corresponding to one record used for learning the lesion detection model from the learning data D1 stored in the storage device 2 (step S11).

[0060] Next, learning device 1 acquires an inference result of the lesion detection model based on model information D2 stored in storage device 2 and the training image acquired in step S11 (step S12). In this case, learning device 1 acquires an inference result output by the lesion detection model by inputting the training image acquired in step S11 into the lesion detection model that reflects model information D2. Learning device 1 then calculates a first loss L1 based on the inference result of the lesion detection model and the ground truth data acquired in step S11 (step S13).

[0061] Furthermore, the learning device 1 executes steps S14 to S17 before, after, or in parallel with the processing of step S13. First, the learning device 1 identifies a lesion area and a perilesional area based on the ground truth data acquired in step S11 (step S14). Then, the learning device 1 acquires feature amounts of the identified lesion area and feature amounts of the perilesional area (step S15). Then, the learning device 1 calculates a similarity S between the feature amounts of the lesion area and the feature amounts of the perilesional area (step S16). Then, the learning device 1 calculates a second loss L2 based on the similarity S calculated in step S16 (step S17).

[0062] After executing steps S13 to S17, the learning device 1 calculates an integrated loss L based on the first loss L1 calculated in step S13 and the second loss L2 calculated in step S17 (step S18).The learning device 1 then updates the parameters of the lesion detection model stored in model information D2 based on the integrated loss L calculated in step S18 (step S19).In this case, the learning device 1 determines the updated parameters of the lesion detection model so as to optimize the integrated loss L.

[0063] The learning device 1 then determines whether or not to terminate learning (step S20). For example, the learning device 1 determines that learning should terminate when all records in the learning data D1 have been used to train the lesion detection model, or when other predetermined learning termination conditions are met. If the learning device 1 determines that learning should terminate (step S20; Yes), it terminates the processing of the flowchart. On the other hand, if the learning device 1 determines that learning should not terminate (step S20; No), it returns the processing to step S11. In this case, in step S11, the learning device 1 extracts records from the learning data D1 that have not been used in training the lesion detection model.

[0064] (6) Modifications Next, suitable modifications of the above-described embodiment will be described. The following modifications may be applied in combination to the above-described embodiment.

[0065] (Variation 1) When using a lesion detection model having an architecture that generates a multi-scale feature map having multiple layers (e.g., Feature Pyramid Networks), the learning device 1 may calculate a similarity S based on the feature amounts between layers in the feature map.

[0066] In the following, we assume that the higher the layer in a multi-scale feature map, the higher the granularity (coarser the area). That is, the area in the learning image represented by each block is larger in the higher layer. In this case, the features in the higher layers are dominated by broader features, while the features in the lower layers are dominated by features closer to the interior of the lesion.

[0067] The feature acquisition unit 33 first identifies a layer of the feature map that has a block (also referred to as the "most similar block") that is closest to the bounding box of the lesion area. In this case, the feature acquisition unit 33 calculates, for example, a match evaluation index (e.g., IoU) with respect to the bounding box of the lesion area for each block in each layer, and determines the block with the highest match evaluation index as the most similar block. Note that if anchor boxes are defined in each layer, the feature acquisition unit 33 may use the anchor box with the highest match evaluation index instead of the most similar block for subsequent processing.

[0068] The feature amount acquiring unit 33 then identifies a block in an upper layer that corresponds to the most similar block (also referred to as an "upper-layer corresponding block"), acquires the feature amount of the identified upper-layer corresponding block as the feature amount of the lesion peripheral region, and acquires the feature amount of the most similar block as the feature amount of the lesion region. The second loss calculating unit 34 then calculates the similarity S between the feature amount of the lesion peripheral region and the feature amount of the lesion region, and calculates a second loss L2 corresponding to the similarity S.

[0069] Here, the upper layer corresponding block may be a block corresponding to the most similar block in the layer one level above the most similar block, or may be a block corresponding to the most similar block in all or a predetermined number of layers above the most similar block.

[0070] 7A shows a training image that clearly indicates the correspondence between each block in the layer containing the most similar block and the training image, while FIG. 7B shows a training image that clearly indicates the correspondence between each block in the layer containing the corresponding block in the upper layer and the training image.

[0071] In this case, feature amount acquisition unit 33 determines block 70, which has the highest matching evaluation index with respect to the bounding box of the lesion area, as the most similar block, and determines corresponding block 71 in the layer above the layer in which the most similar block exists as the upper-layer corresponding block. Feature amount acquisition unit 33 then acquires the feature amount corresponding to block 70 as the feature amount of the lesion area, and the feature amount corresponding to block 71 as the feature amount of the peripheral-lesion area.

[0072] In addition to calculating the similarity between the feature amounts of the upper-layer corresponding block and the feature amounts of the most similar block (also referred to as "upper-layer similarity"), the learning device 1 may also calculate the similarity between the feature amounts of the most similar block and the feature amounts of the corresponding block in the layer below the most similar block (also referred to as "lower-layer similarity"). In this case, for example, the learning device 1 determines the sum of the upper-layer similarity and the lower-layer similarity as similarity S, and calculates the second loss L2 based on this similarity S. Instead of determining the sum of the upper-layer similarity and the lower-layer similarity as similarity S, the learning device 1 may determine similarity S based on an arbitrary formula or lookup table so that it has a positive correlation with the upper-layer similarity and the lower-layer similarity.

[0073] In this modified example, the learning device 1 also preferably calculates the similarity S and the second loss L2, and learns a lesion detection model that takes the second loss L2 into account, thereby facilitating the learning of the features of positive cases.

[0074] (Variant 2) Instead of identifying the lesion-periphery area based on its physical proximity to the lesion area, the feature acquisition unit 33 may identify an associated area of ​​the lesion area (also called a "lesion-associated area") that corresponds to the lesion-periphery area based on its color proximity.

[0075] In this case, for example, the feature acquisition unit 33 uses an index value (also referred to as a "reference color index value") indicating the degree of one or more reference colors as a scale to identify blocks or anchor boxes similar to the bounding box of the lesion area as the lesion-related region. Specifically, for example, the feature acquisition unit 33 identifies blocks or anchor boxes other than the blocks or anchor boxes identified as the lesion area that have reference color index values ​​within a predetermined difference from the reference color index value of the bounding box of the lesion area as the lesion-related region. The predetermined difference is, for example, a default value stored in advance in the storage device 2 or memory 12. The reference color index value may be any of the RGB values ​​contained as pixel values ​​in each pixel of the learning image, or may be a value derived from the RGB values.

[0076] Then, the second loss calculation unit 34 calculates a similarity S based on the feature amount of the lesion region and the feature amount of the lesion-related region, and calculates a second loss L2 based on the similarity S.

[0077] This modification also enables the learning device 1 to learn a lesion detection model based on the second loss L2, thereby facilitating learning of the features of positive cases.

[0078] (Modification 3) The model to be learned is not limited to a lesion detection model that receives an endoscopic image as input and outputs an inference result regarding the detection of a lesion area.

[0079] The model to be trained may be any anomaly detection model that takes an image of a subject to be detected for anomaly as input and outputs an inference result regarding a region suspected of having an abnormality (anomaly region). For example, the anomaly detection model may take an image taken during a visual inspection (visual inspection image) as input and output an inference result regarding the detection of an abnormal region.

[0080] In this case, the feature amount acquiring unit 33 acquires feature amounts of abnormal regions based on the correct data instead of feature amounts of the lesion region, and acquires feature amounts of anomaly-related regions that are close to the abnormal region in physical distance or color characteristics instead of feature amounts of the lesion-related region. The second loss calculating unit 34 then calculates the similarity S between the feature amounts of the abnormal region and the feature amounts of the anomaly-related region, and sets the second loss L2 based on the similarity S.

[0081] In this way, the learning device 1 can also learn anomaly detection models other than lesion detection models by taking into account the second loss L2 according to the similarity S, thereby promoting the learning of the features of positive cases.

[0082] Furthermore, for example, a countermeasure may be determined based on a model generated by machine learning the correspondence between the anomaly detection results of a subject and the countermeasures, and the inference results regarding the anomaly detection of a subject output by the anomaly detection model. The method for determining a countermeasure is not limited to the above-described method. The method may be used to support user decision-making, etc. Note that the determination of a countermeasure and the output of information regarding the countermeasure may be performed by the learning device 1, or may be performed by any device other than the learning device 1 that references the model information of the trained anomaly detection model.

[0083] 8 is a block diagram of a learning device 1X according to a second embodiment. The learning device 1X mainly includes a feature acquisition unit 33X and a learning unit 36X. The learning device 1X may be composed of multiple devices.

[0084] The feature acquisition unit 33X acquires feature amounts of abnormal regions of training images that contain abnormal regions and feature amounts of anomaly-related regions that are regions defined based on the abnormal regions, and is an image used for training an anomaly detection model that is a machine learning model. The feature acquisition unit 33X can be, for example, the feature acquisition unit 33 in the first embodiment (including modified examples, the same applies hereinafter).

[0085] The learning means 36X learns the anomaly detection model based on the degree of similarity between the feature amounts of the abnormal region and the feature amounts of the anomaly-related region. The learning means 36X can be, for example, the learning unit 36 ​​in the first embodiment.

[0086] 9 is an example flowchart showing a processing procedure in the second embodiment. The feature acquisition unit 33X acquires feature amounts of abnormal regions of training images containing abnormal regions, which are images used for training the anomaly detection model, which is a machine learning model, and feature amounts of anomaly-related regions, which are regions defined based on the abnormal regions (step S21). Next, the learning unit 36X trains the anomaly detection model based on the degree of similarity between the feature amounts of the abnormal regions and the feature amounts of the anomaly-related regions (step S22).

[0087] According to the second embodiment, the learning device 1X can learn an anomaly detection model by taking into consideration the degree of similarity between the feature amounts of the abnormal region and the feature amounts of the anomaly-related region.

[0088] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0089] In addition, part or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0090] [Supplementary Note 1] A learning device comprising: a feature acquisition means for acquiring feature amounts of an abnormal region of a training image containing an abnormal region, the image being used for training an anomaly detection model, and feature amounts of an anomaly-related region that is a region defined based on the abnormal region; and a learning means for training the anomaly detection model based on the degree of similarity between the feature amounts of the abnormal region and the feature amounts of the anomaly-related region. [Supplementary Note 2] The learning device according to Supplementary Note 1, comprising: a first loss calculation means for calculating a first loss based on ground truth data corresponding to the training image and an inference result output by the anomaly detection model when the training image is input; and a second loss calculation means for calculating a second loss based on the degree of similarity, wherein the learning means trains the anomaly detection model based on the first loss and the second loss. [Supplementary Note 3] The learning device according to Supplementary Note 2, wherein the second loss calculation means increases the second loss as the degree of similarity decreases. [Supplementary Note 4] The learning device according to Supplementary Note 2, wherein the learning means determines parameters of the anomaly detection model so as to optimize an integrated loss obtained by integrating the first loss and the second loss. [Supplementary Note 5] The learning device according to Supplementary Note 1, wherein the feature acquisition means identifies an area of ​​the learning image present around the abnormal area as the anomaly-related area. [Supplementary Note 6] The learning device according to Supplementary Note 1, wherein the anomaly detection model generates a map of feature amounts in multiple layers with different granularities, and the feature acquisition means extracts, as the feature amounts of the anomaly-related area, feature amounts corresponding to the abnormal area in a layer with a higher granularity than a layer that extracts the feature amounts of the abnormal area. [Supplementary Note 7] The learning device according to Supplementary Note 1, wherein the feature acquisition means identifies, as the anomaly-related area, an area of ​​the learning image similar in color to the abnormal area. [Supplementary Note 8] The learning device according to Supplementary Note 1, wherein the learning images are medical images, and the anomaly detection model is a model that performs inference regarding the detection of a lesion area in the medical image. [Supplementary Note 9] The learning device according to Supplementary Note 1, wherein the anomaly detection model is a machine learning model, and the learning means performs machine learning on the anomaly detection model.[Supplementary Note 10] A learning method in which a computer acquires feature amounts of abnormal regions of learning images that include an abnormal region, the learning images being images used for learning an anomaly detection model, and feature amounts of an anomaly-related region that is a region defined based on the abnormal region, and learns the anomaly detection model based on the degree of similarity between the feature amounts of the abnormal region and the anomaly-related region. [Supplementary Note 11] A storage medium that stores a program that causes a computer to execute a process of acquiring feature amounts of the abnormal regions of learning images that include an abnormal region, the images being used for learning an anomaly detection model, and feature amounts of the anomaly-related region that is a region defined based on the abnormal region, and learning the anomaly detection model based on the degree of similarity between the feature amounts of the abnormal region and the anomaly-related region.

[0091] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference.

[0092] 1, 1X learning device 2 storage device 11 processor 12 memory 13 interface 100 learning system D1 learning data D2 model information

Claims

1. a feature amount acquiring means for acquiring a feature amount of an abnormal region of a learning image, the image being used for learning an anomaly detection model, the abnormal region including the abnormal region, and a feature amount of an anomaly-related region that is a region determined based on the abnormal region; learning means for learning the anomaly detection model based on the degree of similarity between the feature amount of the anomaly region and the feature amount of the anomaly-related region; A learning device having the above configuration.

2. a first loss calculation means for calculating a first loss based on correct answer data corresponding to the learning image and an inference result output by the anomaly detection model when the learning image is input; a second loss calculation means for calculating a second loss based on the degree of similarity, The learning device according to claim 1 , wherein the learning means learns the anomaly detection model based on the first loss and the second loss.

3. The learning device according to claim 2 , wherein the second loss calculation means increases the second loss as the degree of similarity decreases.

4. The learning device according to claim 2 , wherein the learning means determines parameters of the anomaly detection model so as to optimize an integrated loss obtained by integrating the first loss and the second loss.

5. The learning device according to claim 1 , wherein the feature amount acquiring means identifies an area of ​​the learning image that exists around the abnormal area as the abnormality-related area.

6. The anomaly detection model generates a map of feature quantities in multiple layers with different granularities; The learning device according to claim 1 , wherein the feature acquisition means extracts, as the feature of the anomaly-related region, a feature corresponding to the abnormal region in a layer with a higher granularity than a layer that extracts the feature of the abnormal region.

7. The learning device according to claim 1 , wherein the feature amount acquiring means identifies, as the anomaly-related region, a region of the learning image that has a color similar to that of the abnormal region.

8. the training images are medical images, The learning device according to claim 1 , wherein the anomaly detection model is a model that performs inference regarding detection of a lesion area in the medical image.

9. the anomaly detection model is a machine learning model, The learning device according to claim 1 , wherein the learning means performs machine learning on the anomaly detection model.

10. The computer acquiring a feature amount of an abnormal region of a learning image used for learning an anomaly detection model, the learning image including the abnormal region, and a feature amount of an anomaly-related region that is a region determined based on the abnormal region; learning the anomaly detection model based on the degree of similarity between the feature amount of the anomaly region and the feature amount of the anomaly-related region; How to learn.

11. acquiring a feature amount of an abnormal region of a learning image used for learning an anomaly detection model, the learning image including the abnormal region, and a feature amount of an anomaly-related region that is a region determined based on the abnormal region; a program that causes a computer to execute a process of learning the anomaly detection model based on the degree of similarity between the feature amount of the abnormal region and the feature amount of the anomaly-related region;