Method for generating a learning model
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- OLYMPUS MEDICAL SYST CORP
- Filing Date
- 2023-02-08
- Publication Date
- 2026-07-31
Smart Images
Figure 0007898120000016 
Figure 0007898120000017 
Figure 0007898120000018
Abstract
Description
Technical Field
[0005] ,
[0001] The present invention relates to a method for generating a learning model law and the like.
Background Art
[0002] The accuracy of image diagnosis of early gastric cancer largely depends on the experience of doctors. In particular, the detection of flat lesions is considered difficult due to the diversity of the gastric mucosa background. Experts suppress the findings to be noted according to the sites where gastric cancer is likely to occur or the background gastric mucosa, and conduct observations by utilizing their experience and knowledge. However, it is difficult for non-experts with shallow experience and knowledge to accurately diagnose. It is conceivable to use AI (Artificial Intelligence) to support such diagnosis. It is assumed that multiple AIs will be used for image diagnosis of early gastric cancer, and among them, gastric cancer risk AI is assumed. Gastric cancer risk AI detects the gastric cancer risk from the infection status of Helicobacter pylori or the atrophy status of the background gastric mucosa. Patent Document 1 discloses a method for discriminating the infection status of Helicobacter pylori from an image using machine learning.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
[0006] Although the above explanation uses the diagnosis of Helicobacter pylori infection as an example, similar challenges arise when a case is judged as a whole from multiple images acquired from multiple sites or under multiple imaging conditions. [Means for solving the problem]
[0007] One aspect of the present disclosure relates to a method for generating a learning model for inferring the state of a region captured in an input endoscopic image from the input endoscopic image, wherein an anchor image selected from N endoscopic images (N is an integer of 3 or more) each having a ground truth label assigned to it, and N-1 endoscopic images from the N endoscopic images excluding the anchor image are input to the learning model, and the learning model outputs an i-th predicted probability distribution, which is the predicted probability of each class of the state corresponding to the i-th endoscopic image, from the i-th endoscopic image (i is an integer of 1 or more and less than or equal to N) of the N endoscopic images, thereby outputting N predicted probability distributions including an anchor predicted probability distribution corresponding to the previous anchor image and N-1 predicted probability distributions corresponding to the N-1 endoscopic images, and the method for generating a learning model to update the weight parameters of the learning model using a loss function based on the error between the anchor predicted probability distribution and the ground truth label assigned to the anchor image, and the distance between the anchor predicted probability distribution and the distribution calculated from the N-1 predicted probability distributions.
[0008] Another aspect of this disclosure relates to an endoscope system including a memory for storing a trained model generated by the above-described method for generating a learning model, and a processor for inferring the state of a region shown in an endoscope image by performing inference processing using the trained model on the endoscope image. [Brief explanation of the drawing]
[0009] [Figure 1] An example of an image obtained in a particular case, and the diagnostic result for that case. [Figure 2] A diagram illustrating supervised learning using knowledge distillation with separated teacher and student models. [Figure 3] An example configuration of the learning system and endoscope system of this embodiment. [Figure 4] Flowchart of the learning model generation process. [Figure 5] The flow of the inference process. [Figure 6] Detailed flow of fine-tuning learning. [Figure 7] Detailed flow of model-based computations during fine-tuning learning. [Figure 8] Detailed flow of calculating the loss function in fine-tuning learning. [Figure 9] Detailed flow of the inference process. [Figure 10] Detailed flow chart for pre-training. [Modes for carrying out the invention]
[0010] The following describes this embodiment. Note that the embodiment described below is not intended to unduly limit the scope of the claims. Furthermore, not all configurations described in this embodiment are necessarily essential components of this disclosure.
[0011] 1. Imaging diagnosis of Helicobacter pylori and knowledge distillation. The following description of this embodiment will use the case of diagnosing the status of Helicobacter pylori infection in the stomach using imaging as an example. However, the imaging diagnostic method of this embodiment can be applied when performing a comprehensive imaging diagnosis from multiple images having multiple locations or imaging conditions.
[0012] Figure 1 shows an example of images obtained in a single case and the diagnostic results for that case. Let's assume that 100 images, images 1 to 100, are obtained by imaging various parts of the stomach using an endoscope. 100 is just one example, and the number of images can be arbitrary. Let a1 to a100 be the parts or imaging conditions shown in images 1 to 100. Some images may have overlapping parts or imaging conditions, but overall, images 1 to 100 are images taken of various parts and under various imaging conditions. The parts here refer to parts of the stomach, such as the cardia, upper stomach, middle stomach, lower stomach, or pylorus. The imaging conditions are, for example, the distance between the endoscope and the subject, the direction in which the subject is photographed, the brightness of the image, the noise of the image, how the subject is illuminated, or the degree of stomach expansion due to air insufflation.
[0013] The physician comprehensively diagnoses the Helicobacter pylori infection status by looking at images 1 to 100. For example, the physician checks the entire stomach and excludes inappropriate images, and determines the Helicobacter pylori infection status from the state of mucosal atrophy visible in the endoscopic images. For example, normal mucosa is judged as uninfected, weak or strong atrophy as current infection, and regenerating epithelium as past infection. Uninfected means that the person has not been infected with Helicobacter pylori in the past and is not currently infected. Current infection means that the person is currently infected with Helicobacter pylori. Past infection means that the person was infected with Helicobacter pylori in the past but is not currently infected. Based on these judgments, let's say the physician diagnoses past infection from images 1 to 100. However, even if a case is diagnosed as past infection, individual images may show a mixture of images that appear to depict various infection states. For example, in a case diagnosed as past infection, there may be images that appear to depict past infection as described in the case, and images that appear to depict a mucosal state close to uninfected. In the example in Figure 1, for example, image 2 appears to depict a mucosal state close to uninfected.
[0014] In the diagnosis of Helicobacter pylori infection status, it is conceivable to make the experience or knowledge of a skilled doctor available to an unskilled doctor by making the diagnosis AI (Artificial Intelligence). At this time, it is difficult to infer the accurate infection status from images having the appearances of various infection states. For example, in supervised learning using only hard target loss, an input image and a correct label are input into a learning model, and the learning model infers the probability of each class from the input image. Then, the hard target loss, which is the error between the inferred probability distribution and the distribution of the correct label, is calculated, and the learning model is updated so as to minimize the hard target loss. By such a method, a learning model for inferring the infection status from individual images can be obtained, so the learning model will infer "already infected" from an image 2 that appears to be in a mucosal state close to uninfected in FIG. 1. However, since the diagnosis result of "already infected" is originally obtained comprehensively from images 1 to 100, it is difficult to accurately infer by the learning model as described above.
[0015] In the present embodiment, by generating a learning model by self-knowledge distillation applying the method of knowledge distillation, the Helicobacter pylori infection status can be accurately diagnosed. First, the method of knowledge distillation in which a teacher model and a student model are separated will be described below, and then the configuration of the present embodiment and the method of self-knowledge distillation of the present embodiment will be described.
[0016] FIG. 2 is a diagram for explaining supervised learning of knowledge distillation in which a teacher model and a student model are separated.
[0017] In knowledge distillation, first, the teacher model is learned, and then the student model is learned using the output of the teacher model. The teacher model is larger in capacity and higher in accuracy than the student model. The student model is smaller in capacity than the teacher model, but the goal is to have the same level of accuracy as the teacher model. The capacity of a model means the file capacity of the program in which the model is described, or the number of nodes of the neural network included in the model, etc.
[0018] It is assumed that the teacher model has already been learned through supervised learning. Here, a model that classifies images into classes 1 to 5 is taken as an example. For example, the images are pictures of animals such as dogs, and classes 1 to 5 are dogs, cats, hamsters, fish, cars, etc. The teacher model outputs a probability distribution for classes 1 to 5 by inferring the probability of each class from the input image.
[0019] The student model outputs a probability distribution for classes 1 to 5 by inferring the probability of each class from the input image. The error between the probability distribution output by the student model and the distribution of the correct labels is calculated as the hard target loss. Also, the error between the probability distribution output by the student model and the probability distribution output by the teacher model is calculated as the soft target loss. Then, a loss function is calculated from the hard target loss and the soft target loss, and the student model is updated so that the loss function decreases. By performing the update with a large number of images, the student model is learned.
[0020] Through such supervised learning of knowledge distillation, the following advantages (i) and (ii) can be obtained. (i) The correct label is a distribution in which only the probability of one class among classes 1 to 5 is 1. On the other hand, the probability distribution output by the teacher model has probabilities distributed not only in the correct class but also in various classes, and has information not included in the correct label. For example, when the input image may also resemble classes other than the correct class, a probability distribution including the probability of that class is output from the teacher model. Since the student model learns the probability distribution output by the teacher model, it can learn more information than when learning only the correct label.
[0021] (ii) The student model learns more information from the probability distribution output by the teacher model. As a result, it becomes possible to create a student model with a small capacity but with the same high accuracy as the large-capacity teacher model.
[0022] The model creation method using self-knowledge distillation in this embodiment utilizes the characteristics described in (i) above. This improves diagnostic accuracy in diagnosing Helicobacter pylori infections, where individual images may appear to represent various infection states even in cases of a specific infection condition.
[0023] 2. Learning system and endoscopic system Figure 3 shows an example configuration of the learning system and endoscope system of this embodiment. Note that the endoscope system 200 does not need to be connected to the learning system 100 during the learning phase, nor does the learning system 100 need to be connected to the endoscope system 200 during the inference phase.
[0024] The learning system 100 is a system that performs learning processing on the learning model 121, and is, for example, an information processing device such as a PC or a server. Alternatively, the learning system 100 may be a cloud system in which multiple information processing devices are connected via a network. The learning system 100 includes a processor 110, memory 120, display 130, and operating device 140.
[0025] Memory 120 stores the learning model 121 and a program for executing the learning process on the learning model 121. The learning model 121 may include a program describing the algorithm and data used within that program. For example, the learning model 121 may include a neural network such as a CNN (Convolutional Neural Network). In that case, the learning model 121 includes a program describing the algorithm of the neural network and weight parameters between the nodes of the neural network. The processor 110 performs the learning process on the learning model 121 stored in memory 120 by executing the learning process program stored in memory 120.
[0026] The processor 110 includes hardware. The processor 110 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microcomputer, or a DSP (Digital Signal Processor). Alternatively, the processor 110 may be an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor 110 may consist of one or more of the CPU, GPU, microcomputer, DSP, ASIC, and FPGA. The memory 120 is, for example, a semiconductor memory which is volatile memory or non-volatile memory. Alternatively, the memory 120 may be a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive.
[0027] Furthermore, a non-temporary information storage medium, which is a computer-readable medium, may store the learning model 121 and the learning process program. The information storage medium is, for example, an optical disc, a memory card, a hard disk drive, or a semiconductor memory. The semiconductor memory is, for example, ROM or non-volatile memory. The learning system 100 performs various processes of this embodiment based on the learning model 121 and the learning process program stored in the information storage medium.
[0028] The display 130 is a liquid crystal display or an EL (Electro Luminescence) display, etc., and displays various information related to the learning process. The operating device 140 is a device that receives operation input from the user to the learning system 100, and is, for example, a pointing device, a keyboard, or a touch panel.
[0029] The endoscopic system 200 determines the Helicobacter pylori infection status from endoscopic images using a trained model 221. The endoscopic system 200 includes a processor 210, memory 220, display 230, controller 250, and endoscope 260.
[0030] The endoscope 260 is inserted into the body and captures images of the inside of the body. The endoscope 260 may be detachable from the controller 250. The endoscope 260 includes, for example, an image sensor for capturing images, an objective lens for forming an image of the subject on the image sensor, a light guide for guiding illumination light, and an illumination lens for emitting the guided illumination light. The endoscope 260 may also include a water supply channel or a forceps channel, etc. In determining the status of Helicobacter pylori infection, the endoscope 260 is assumed to be a flexible endoscope for the upper gastrointestinal tract for imaging the stomach, etc. However, the endoscope 260 may be a flexible endoscope for the lower gastrointestinal tract or a rigid surgical endoscope, etc., depending on the application.
[0031] The controller 250 performs various control and image processing operations on the endoscope system 200. For example, the controller 250 performs image processing on the endoscopic image acquired from the endoscope 260 and displays the processed endoscopic image on the display 230. Alternatively, the controller 250 transmits the endoscopic image to the processor 210, receives the result of the Helicobacter pylori infection status determination from the processor 210, and displays the determination result on the display 230. The display 230 is a liquid crystal display or an EL (Electro-Luminescence) display, etc.
[0032] Memory 220 stores the trained model 221. The trained model 221 is the trained model 121 that has been trained by the training system 100. The processor 210 uses the trained model 221 stored in memory 220 to determine the Helicobacter pylori infection status from the endoscopic image.
[0033] The processor 210 includes hardware. The processor 210 is, for example, a CPU, GPU, microcomputer, or DSP. Alternatively, the processor 210 may be an ASIC or FPGA. The processor 210 may consist of one or more of the CPU, GPU, microcomputer, DSP, ASIC, and FPGA. The memory 220 is, for example, a semiconductor memory, magnetic memory device, or optical memory device. The processor 210 and memory 220 are, for example, the processor and memory of an information processing device provided separately from the controller 250. Alternatively, the processor 210 and memory 220 may be the processor and memory provided within the enclosure of the controller 250.
[0034] Although Figure 3 shows an example where the learning system 100 is provided separately from the endoscope system 200, the endoscope system 200 may also function as the learning system 100. In that case, the processor 210 of the endoscope system 200 performs the learning process for the learning model 121.
[0035] Figure 4 shows the flow of the learning model generation process. In step S10, the processor 110 of the learning system 100 performs pre-training on the pre-trained model 21 using the training data, which is the endoscopic image IMP. Pre-training is, for example, self-supervised learning, as described later, but is not limited to that. For example, pre-training may be supervised learning, in which case the training data includes endoscopic images and the correct labels attached to those endoscopic images.
[0036] In step S20, the processor 110 performs fine-tuning learning on the learning model 121 obtained in pre-training using the training data, which are endoscopic images IM_1 to IM_N. N is an integer greater than or equal to 3. Fine-tuning learning is supervised learning of self-knowledge distillation, which will be described later. The training data includes endoscopic images IM_1 to IM_N and the correct labels assigned to each endoscopic image. The endoscopic images used for fine-tuning learning may be the same as or different from the endoscopic images used for pre-training.
[0037] Figure 5 shows the inference process flow. In step S30, the endoscope 260 acquires an endoscopic image of the stomach using IMscope. In step S40, the processor 210 of the endoscope system 200 inputs the endoscopic image IMscope to the trained model 221 obtained through fine-tuning learning, and causes the trained model 221 to determine the Helicobacter pylori infection status.
[0038] Figure 6 shows the detailed flow of the fine-tuning learning process S20.
[0039] In step S100, the processor 110 of the learning system 100 performs calculations using the model. Specifically, the processor 110 inputs N endoscopic images IM_1 to IM_N into the learning model 121. Each endoscopic image is assigned a correct label indicating the state of Helicobacter pylori infection. The endoscopic images IM_1 to IM_N constitute a batch, which is a group of images input into the learning model 121 at once. A batch is composed of, for example, endoscopic images from various cases, various stomach locations, and various imaging conditions.
[0040] The processor 110 sets one of the endoscopic images IM_1 to IM_N as the anchor image. Figure 6 shows an example where IM_1 is set as the anchor image, but the anchor image can be any of IM_1 to IM_N. The anchor image is the input to the student side, but in the fine-tuning learning of this embodiment, self-knowledge distillation is used, so the teacher side and the student side are not separated, and the learning model 121 plays the roles of both the teacher side and the student side.
[0041] The learning model 121 infers the predicted probability distribution p1 from the anchor image IM_1. We will also refer to p1 as the anchor predicted probability distribution. Similarly, the learning model 121 infers the predicted probability distributions p2 to pN from the remaining N-1 endoscopic images IM_2 to IM_N. Let i be an integer between 1 and N, inclusive. The predicted probability distribution pi is a distribution in which the predicted probability is shown for each of multiple classes. Suppose we classify the Helicobacter pylori infection status into, for example, uninfected, currently infected, and previously infected. In this case, the predicted probability distribution pi is a vector (pi(1), pi(2), pi(3)) consisting of the predicted probability pi(1) for the uninfected class, the predicted probability pi(2) for the currently infected class, and the predicted probability pi(3) for the previously infected class. The sum of the elements of this vector is normalized to 1.
[0042] In Figure 6, the "SG" indicated by the arrow "p2~pN" stands for Stop Gradient, meaning that the weight parameter update using gradient descent is stopped for p2~pN. In other words, the weight parameter update is performed to improve the accuracy of the anchor prediction probability distribution p1, and the prediction probability distributions p2~pN are used only for calculating the loss function. More specifically, the hard target loss is calculated from the anchor prediction probability distribution p1, and the prediction probability distributions p2~pN are used for calculating the soft target loss.
[0043] In step S200, the processor 110 calculates the loss function LSKD from the predicted probability distributions p1 to pN. Specifically, in step S210, the processor 110 calculates the soft target loss from the predicted probability distributions p1 to pN, and in step S220, it calculates the hard target loss from the anchor predicted probability distribution p1 and the ground truth label TGp1, and calculates the sum of the hard target loss and the soft target loss as the loss function LSKD. The ground truth label TGp1 is the ground truth label assigned to the endoscopic image IM_1, and, similar to the predicted probability distribution, is a distribution in which the probability of each of the multiple classes is indicated. For example, the ground truth label TGp1 is a vector (TGp1(1), TGp1(2), TGp1(3)) consisting of the probability of the uninfected class TGp1(1), the probability of the currently infected class TGp1(2), and the probability of the previously infected class TGp1(3). The probability that any of the probabilities TGp1(1), TGp1(2), and TGp1(3) is "1" is true, and the probability of any other probabilities is "0".
[0044] The hard target loss is a loss function based on the error between the anchor prediction probability distribution p1 and the distribution of the ground truth label TGp1. The hard target loss is, for example, the cross-entropy loss, but is not limited to that; any function that can evaluate the above-mentioned inter-distribution error will suffice.
[0045] The soft target loss is a loss function based on the distance between the anchor prediction probability distribution p1 and the distribution calculated from the prediction probability distributions p2~pN. For example, the learning model 121 calculates the similarity of features between each endoscopic image IM_1~IM_N and calculates a soft target from this similarity and the prediction probability distributions p1~pN. At this time, the prediction probability distribution pj is weighted according to the similarity between the features of endoscopic image IM_i and endoscopic image IM_j, and the soft target Q corresponding to the prediction probability distribution pi is calculated from this weighted prediction probability distribution pj. The soft target loss is, for example, the KL divergence between the prediction probability distributions p1~pN and the soft target Q. For i=1, the KL divergence is related to the distance between the anchor prediction probability distribution p1 and the distribution calculated from the prediction probability distributions p2~pN (the soft target Q corresponding to the anchor prediction probability distribution p1). However, the soft target loss is not limited to the KL divergence; any function that can evaluate the above-mentioned distance between distributions is acceptable.
[0046] In step S300, the processor 110 updates the weight parameters of the learning model 121 to minimize the loss function LSKD. The update method is, but is not limited to, gradient descent or backpropagation. The processor 110 grows the learning model 121 by repeating steps S100 to S300 for a large number of batches.
[0047] In this embodiment described above, the learning model 121 is a model for inferring the state of a region captured in the input endoscopic image IMscope from the input endoscopic image IMscope. The method for generating the learning model involves inputting an anchor image IM_1 and N-1 endoscopic images IM_2 to IM_N into the learning model 121. The anchor image IM_1 is an image selected from N endoscopic images IM_1 to IM_N, each of which is assigned a correct label for the state. The N-1 endoscopic images IM_2 to IM_N are the images from N endoscopic images IM_1 to IM_N excluding the anchor image IM_1. The learning model 121 outputs N prediction probability distributions p1 to pN by outputting the i-th prediction probability distribution pi from the i-th endoscopic image IM_i. The i-th prediction probability distribution pi is the prediction probability for each class of state corresponding to the i-th endoscopic image IM_i. The N prediction probability distributions p1~pN include the anchor prediction probability distribution p1, which corresponds to the anchor image IM_1, and N-1 prediction probability distributions p2~pN, which correspond to N-1 endoscopic images IM_2~IM_N. The method for generating the learning model updates the weight parameters of the learning model 121 using a loss function LSKD based on the error between the anchor prediction probability distribution p1 and the ground truth label TGp1 assigned to the anchor image IM_1, and the distance between the anchor prediction probability distribution p1 and the distributions calculated from the N-1 prediction probability distributions p2~pN.
[0048] According to this embodiment, the distance between the anchor prediction probability distribution p1 and the distribution calculated from N-1 prediction probability distributions p2 to pN can be considered as the soft target loss obtained from the output of the teacher model. This enables learning by self-knowledge distillation without separating the student model and the teacher model. By adopting this self-knowledge distillation method, it is possible to learn by associating the features of a large number of endoscopic images IM_1 to IM_N using the prediction probability distributions p1 to pN of a large number of endoscopic images IM_1 to IM_N. As a result, even when input images that look different from the overall diagnostic result of a case are input in the image diagnosis of Helicobacter pylori infection, inference is performed based on associations between various images, so that accurate image diagnostic results can be obtained from those input images.
[0049] In this embodiment, the correct label is a label assigned based on the diagnosis of a disease in the affected area. More specifically, the correct label is a label assigned based on the diagnosis of Helicobacter pylori infection status in the stomach.
[0050] According to this embodiment, the learning model 121 is fine-tuned based on the correct labels assigned based on the diagnostic results. The diagnostic results are, for example, the results of a physician's diagnosis of "disease in a specific area" or "Helicobacter pylori infection status in the stomach." This diagnosis is not limited to endoscopic images. The task of assigning correct labels to endoscopic images may be performed by a physician or by a non-physician based on the physician's diagnostic results. Multiple endoscopic images taken in a single case are assigned the diagnostic results for that case, i.e., the same diagnostic result, as the correct label. Since the batch input to the learning model 121 consists of images randomly selected from multiple cases, it may include images with multiple different correct labels.
[0051] In this embodiment, the endoscope system 200 also includes a memory 220 and a processor 210. The memory 220 stores the trained model 221 generated by the method for generating the learning model described above. The processor 210 performs inference processing on the endoscope image IMscope using the trained model 221 to infer the state of the area shown in the endoscope image IMscope. Specifically, in the method for generating the learning model, the correct label is a label assigned based on the diagnostic result of the Helicobacter pylori infection status in the stomach. At this time, the processor 210 performs the above inference processing to infer the diagnostic result of the Helicobacter pylori infection status in the stomach.
[0052] According to this embodiment, an endoscopic system 200 can be constructed that infers the status of Helicobacter pylori infection, etc., from endoscopic images using IMscope. Since the trained model 221 has been fine-tuned by associating features of a large number of endoscopic images, accurate image diagnostic results can be obtained even when input images that look different from the overall diagnostic result of the case are input.
[0053] 3. A more detailed flow of fine-tuning learning and inference. Figure 7 shows the detailed flow of the model-based calculation S100 in the fine-tuning learning S20. Note that even if the text in the mathematical formula is in bold, it will be written in non-bold font in the text. Also, the hat symbol "^" that is placed above the letters in the mathematical formula will be written as "^y" in the text. For example, if a hat symbol is placed above "y" in the mathematical formula, it will be written as "^y" in the text.
[0054] The learning model 121 includes an encoder Eθ, a multilayer perceptron G'θ, and a softmax function S. The encoder Eθ is a neural network, and is the AI model obtained by pre-training in step S10 of Figure 4. The multilayer perceptron G'θ is a type of neural network composed of three or more layers. The softmax function S is a function that transforms logits into a predicted probability distribution.
[0055] In step S110, the processor 110 of the learning system 100 inputs the N endoscopic images IM_1 to IM_N constituting the batch into the encoder Eθ. As shown in steps S121 and S122, the encoder Eθ calculates feature vectors yi from the endoscopic image IM_i. Feature vectors yi are visual features contained in the endoscopic image IM_i and are vectors composed of multiple numerical values. For example, the encoder Eθ is a CNN, and the training of the CNN determines what kind of feature vectors yi will be extracted.
[0056] In step S130, the processor 110 obtains a similarity matrix Y that shows the similarity information between features by comparing the features {y1, y2, ..., yN} as shown in equation (1) below. Y is an N x N matrix, and each element is a real number. The right-hand side of equation (1) below represents the inner product of ^yi and ^yj. The indices i and j are the indices of the images in the batch, i.e., integers between 1 and N, from 1 to 1. Yij represents the element in the i row and j column of matrix Y. ^yi represents the normalized representation of yi, as shown in equation (2) below. The denominator of the right-hand side of equation (2) below represents the Euclidean norm of yi.
[0057]
number
[0058]
number
[0059] Processor 110 discards the diagonal elements of matrix Y according to equation (3) below to avoid self-knowledge reinforcement. That is, the diagonal elements of the similarity matrix Y do not play a role in knowledge distillation, so they are removed. The "1" on the right side of equation (1) below represents an N x N matrix where each element is 1. I represents the identity matrix. The operation symbol with a dot inside a circle represents the Hadamard product. That is, when i=j, Yij=Yij×0=0, and when i≠j, Yij=Yij×1=Yij.
[0060]
number
[0061] Processor 110 normalizes matrix Y using equation (4) below. The summation symbol in the denominator on the right-hand side of equation (4) below represents summing over all j such that j≠i.
[0062]
number
[0063] In step S140, the processor 110 inputs the features y1 to yN into the multilayer perceptron G'θ. As shown in steps S151 and S152, the multilayer perceptron G'θ calculates the logit li from the features yi. The logit li is a distribution in which numerical values are shown for each of the multiple classes. The numerical value for each class quantifies the probability that the feature yi belongs to that class. When the number of classes is K, the logit li is represented as a vector (li(1), li(2), ..., li(K)). When the index of a class is k, that is, when k is an integer between 1 and K, the k-th element of the vector li(k) quantifies the probability that it belongs to class k.
[0064] In step S160, the processor 110 inputs the logits l1 to lN into the softmax function S. As shown in steps S171 and S172, the softmax function S transforms the logits li into a predicted probability distribution pi normalized to the sum of its elements to 1. Specifically, the predicted probability distribution pi is obtained by equation (5) below. pi is represented by a vector (pi(1), pi(2), ..., pi(K)), where the kth element is represented by pi(k). τ is the temperature hyperparameter for softness. Hyperparameters are parameters used to control the behavior of a machine learning algorithm.
[0065]
number
[0066] As shown in step S180, the predicted probability distributions p1 to pN are represented as the matrix P in equation (6) below. P is an N x K matrix, and each element is a real number.
[0067]
number
[0068] Figure 8 shows the detailed flow of the loss function calculation S200 in fine-tuning learning S20.
[0069] In step S210, the processor 110 of the learning system 100 calculates the soft target loss using the prediction probability matrix P and the normalized similarity matrix ^Y. The soft target loss is the KL divergence between the soft target Q and the prediction probability matrix P. The soft target Q is expressed by equation (7) below. Q is an N x K matrix, and each element is a real number. ω is the weight coefficient.
[0070]
number
[0071] Equation (7) above is derived as follows. First, as shown in equation (8) below, the soft target Q is generated as the sum of the initial prediction probability matrix P and the propagated prediction probability matrix ^YP. This is to avoid propagating and ensembling noisy predictions.
[0072]
number
[0073] As shown in equation (9) below, propagation and ensemble are performed multiple times to generate a better soft target Q for self-knowledge distillation. t represents the t-th iteration.
[0074]
number
[0075] As the number of iterations approaches infinity, equation (10) below holds. Equation (7) above can be obtained from equation (9) above and equation (10) below.
[0076]
number
[0077] In step S220, the processor 110 calculates the hard target loss from the anchor prediction probability distribution p1 and the ground truth label TGp1 assigned to the anchor image IM_1. The hard target loss is the cross-entropy loss.
[0078] Based on the above, processor 110 calculates the loss function LSKD in equation (11) below. LCE represents the cross-entropy loss. DKL represents the KL divergence. λ represents the balance hyperparameter. Both the cross-entropy loss and the KL divergence are measures of how similar two probability distributions are. That is, the cross-entropy loss in equation (11) below is the result of the relationship between the anchor prediction probability distribution p1 and the ground truth label TGp1. This is a measure of how similar the two distributions are. The more similar the two distributions are, the smaller the cross-entropy loss. The KL divergence in equation (11) below is a measure of how similar the probability distribution represented by the soft target loss Q and the probability distribution represented by the prediction probability matrix P are. The more similar the two distributions are, the smaller the KL divergence.
[0079]
number
[0080] As shown in Figure 7, in step S300, the processor 110 updates the weight parameters of the encoder Eθ using the loss function LSKD. For stable learning, the gradient is not backpropagated through the soft target. After fine tuning by this self-knowledge distillation, the encoder Eθ becomes available for diagnosis of Helicobacter pylori infection symptoms.
[0081] Here, we consider the soft target Q corresponding to the anchor prediction probability distribution p1. For simplicity, we use the soft target Q in equation (8) above, which does not have iterations t.
[0082] As shown in equation (6) above, the first row of the prediction probability matrix P is the anchor prediction probability distribution p1=(p1(1),p1(2),···,p1(K)). If we consider the first row of the soft target Q as the vector Q1=(Q11,Q12,···,Q1K), then the anchor prediction probability distribution p1 corresponds to the vector Q1 of the first row of the soft target Q. From equation (8) above, the vector Q1 is given by equation (12) below.
[0083]
number
[0084] In equation (12) above, the predicted probability distribution pj is weighted by the similarity ^Y1j of the features of endoscopic images IM_1 and IM_j, and the sum of these weighted predicted probability distributions pj is the vector Q1. That is, the KL divergence in equation (11) above is related to the distance between the anchor predicted probability distribution p1 and the vector Q1, i.e., the distance between the anchor predicted probability distribution p1 and the distribution calculated from N-1 predicted probability distributions p2~pN. In fine-tuning learning, the weight parameters are updated to reduce the hard target loss related to the anchor predicted probability distribution p1 and to reduce the above distance shown by the KL divergence. As a result, the features of a large number of endoscopic images IM_1~IM_N are learned in association, and the diagnostic accuracy of Helicobacter pylori infection status can be improved.
[0085] In this embodiment described above, the encoder Eθ included in the learning model 121 calculates N features y1~yn by calculating the i-th feature yi from the i-th endoscopic image IM_i. The learning model generation method calculates similarity information regarding the similarity Yij obtained by comparing the i-th feature yi and the j-th feature yj. In Figure 7, the similarity matrix Y corresponds to the similarity information. The learning model generation method calculates the soft target Q from the similarity information (Y) and N prediction probability distributions p1~pN. The learning model generation method updates the weight parameters of the encoder Eθ using the loss function LSKD. In the loss function LSKD, the hard target loss is defined by the anchor prediction probability distribution p1 and the ground truth label TGp1 assigned to the anchor image IM_1, and the soft target loss is defined by the N prediction probability distributions p1~pN and the soft target Q.
[0086] According to this embodiment, a soft target Q is calculated from similarity information (Y) and N predicted probability distributions p1 to pN, and the soft target loss is calculated using this soft target Q. As a result, the similarity information between images is reflected in the soft target loss, making it possible to learn by associating the features of a large number of endoscopic images IM_1 to IM_N. This enables robust image diagnosis in which an accurate diagnosis of a case can be obtained from the boundary data, even in situations where input images that appear to represent various Helicobacter pylori infection states in a given case are mixed, i.e., when boundary data at the boundary of classification classes are mixed.
[0087] Specifically, the soft target loss is related to the distance between the predicted probability distribution and the distribution represented by the soft target Q. In this embodiment, as shown in Q1 of equation (12) above, the soft target Q is calculated using the similarity information between images, and the predicted probability distribution is weighted according to the similarity information between images. As a result, the soft target loss is related to the distance between the predicted probability distribution and the predicted probability distribution weighted according to the similarity information between images. By using such a soft target loss, it becomes possible to learn by relating the features of a large number of endoscopic images IM_1 to IM_N.
[0088] In this embodiment, the hard target loss is the cross-entropy loss, and the soft target loss is the KL divergence.
[0089] The cross-entropy loss between the anchor prediction probability distribution p1 and the ground truth label TGp1 represents the error between the anchor prediction probability distribution p1 and the ground truth label TGp1. The KL divergence between the prediction probability distributions p1~pN and the soft target Q represents the loss related to the distance between the prediction probability distributions p1~pN and the distribution represented by the soft target Q. Using these losses, we can obtain the error between the anchor prediction probability distribution p1 and the ground truth label TGp1, and the loss function LSKD based on the distance between the anchor prediction probability distribution p1 and the distribution calculated from the N prediction probability distributions p1~pN.
[0090] In this embodiment, the i-th predicted probability distribution pi includes the predicted probabilities of K classes. The N endoscopic images IM_1 to IM_N input to the learning model include endoscopic images assigned a ground truth label for the first class out of K classes, and endoscopic images assigned a ground truth label for the second class out of K classes that is different from the first class.
[0091] According to this embodiment, fine-tuning learning is performed using N endoscopic images IM_1 to IM_N, which are a mixture of images from multiple cases with different diagnostic results, as input. As a result, inference is performed based on the association of various image features in multiple cases, so even if an endoscopic image with features seemingly different from the case's H. pylori infection status is input, it becomes possible to infer the case's H. pylori infection status from that endoscopic image.
[0092] Figure 9 shows the detailed flow of the inference process S40. The trained model 221 includes an encoder Eθ, a multilayer perceptron G'θ, and a softmax function S. The encoder Eθ is the encoder obtained by the fine-tuning learning in Figure 7. The multilayer perceptron G'θ and softmax function S are the same as the multilayer perceptron and softmax function used in the fine-tuning learning in Figure 7.
[0093] In step S410, the processor 210 of the endoscope system 200 inputs the endoscopic image IMscope, captured by the endoscope 260, to the encoder Eθ. As shown in step S420, the encoder Eθ calculates feature quantities y from the endoscopic image IMscope. The specific feature quantities y to be extracted are determined by fine-tuning learning.
[0094] In step S440, the processor 210 inputs the feature vector y into the multilayer perceptron G'θ. As shown in step S450, the multilayer perceptron G'θ calculates the logit l from the feature vector y.
[0095] In step S460, the processor 210 inputs the logit l to the softmax function S. As shown in step S470, the softmax function S converts the logit l into a predicted probability distribution p normalized to a sum of elements equal to 1. The processor 210 outputs a diagnostic result for Helicobacter pylori infection status based on the predicted probability distribution p. For example, the processor 210 displays the class with the highest predicted probability in the predicted probability distribution p as the diagnostic result for Helicobacter pylori infection status on the display 230. Alternatively, the processor 210 may display the predicted probabilities of multiple classes on the display 230 based on the predicted probability distribution p. However, the manner in which the diagnostic result is displayed is not limited to these.
[0096] 4. Pre-learning Figure 10 shows the detailed flow of pre-training S10. The pre-trained model 21 includes encoders Eθ and EΨ, multilayer perceptron projectors Gθ and GΨ, and multilayer perceptron predictor Pθ. The encoders Eθ and EΨ in pre-training are AI models similar to encoder Eθ in fine-tuning learning, and their parameters are updated based on an asymmetric relationship. The pre-trained encoder Eθ is used as the initial model of encoder Eθ in fine-tuning learning.
[0097] In steps S911 and S912, the processor 110 of the learning system 100 generates two images v1 and v2 by augmenting the endoscopic image IMP. Specifically, the processor 110 randomly selects two transformations t1 and t2 from a set of transformations T to generate two visually different images v1=t1(IMP) and v2=t2(IMP). These transformations combine standard data augmentation techniques such as cropping, resizing, image inversion, color jitter, or Gaussian blur.
[0098] Image v1 is processed by the online network encoder Eθ and projector Gθ. An online network is a network that is the target of weight parameter updates using gradient descent in self-supervised learning. Image v2 is processed by the target network encoder EΨ and projector GΨ. A target network is a network that generates the learning targets for the online network.
[0099] Specifically, in step S921, the processor 110 inputs image v1 to encoder Eθ. As shown in step S931, encoder Eθ calculates feature vector y1 from image v1. In step S941, the processor 110 inputs feature vector y1 to projector Gθ. As shown in step S951, projector Gθ converts feature vector y1 into a distribution of predicted values z1. The distribution of predicted values z1 is a distribution in which predicted values are shown for each of multiple classes. When the number of classes is K, the distribution of predicted values z1 is represented by a vector (z1(1), z1(2), ..., z1(K)). When the index of a class is k, that is, when k is an integer between 1 and K, the k-th element z1(k) of the vector is a numerical representation of the probability of belonging to class k.
[0100] In step S960, the processor 110 inputs the distribution of predicted values z1 to the predictor Pθ. As shown in step S970, the multilayer perceptron predictor Pθ transforms the distribution of predicted values z1 into the final output q1 of the online network. Here, q1 is a vector with a class number K, similar to z1. In other words, it plays the role of a nonlinear feature transformation via the multilayer perceptron. Furthermore, the predictor Pθ is used only in online networks and prevents learning collapse by creating structural asymmetry. Learning collapse occurs when different images v1 and v2 are mapped to the same vectors z1 and z2, resulting in a similarity loss LSIM of zero and preventing the network from learning further. By using the predictor Pθ, the network learns further and the image representation ability of the features improves, thus gaining versatility when applied to downstream tasks such as fine-tuning learning.
[0101] In step S922, the processor 110 inputs the image v2 to the encoder EΨ. As shown in step S932, the encoder EΨ calculates the feature vector y2 from the image v2. In step S942, the processor 110 inputs the feature vector y2 to the projector GΨ. As shown in step S952, the projector GΨ converts the feature vector y2 into a distribution of predicted values z2. The distribution of predicted values z2, like z1, is a distribution in which the predicted values are shown for each of the multiple classes.
[0102] The "SG" shown between steps S942 and S952 stands for Stop Gradient, meaning that the weight parameter updates for the target network using gradient descent are stopped.
[0103] In step S980, the processor 110 calculates the similarity loss LSIM using the following equation (13). The similarity loss LSIM compares the representation of image v1 calculated by the online network with the representation of image v2 calculated by the target network. That is, the similarity loss LSIM indicates the similarity between image v1 and image v2. ^q1 represents the normalized q1, i.e., vector q1 divided by the Euclidean norm of vector q1. ^z1 represents the normalized z1, i.e., vector z1 divided by the Euclidean norm of vector z1.
[0104]
number
[0105] In step S990, the processor 110 updates the weight parameters of the encoder Eθ and projector Gθ of the online network to minimize the similarity loss LSIM, as shown in equation (14) below. Opt represents the optimizer and α represents the learning rate.
[0106]
number
[0107] The weight parameters of the encoder EΨ and projector GΨ in the target network are the exponential moving average of the weight parameters of the encoder Eθ and projector Gθ in the online network. That is, the online network and the target network are networks with the same structure, and there is a one-to-one correspondence between the inter-node connections in the online network and the inter-node connections in the target network. The inter-node weight parameters in the target network are updated by the exponential moving average of the corresponding inter-node weight parameters in the online network.
[0108] The processor 110 updates the weight parameters of the target network's encoder EΨ and projector GΨ according to equation (15) below. "Ψ" represents the weight parameters of the target network. "θ" represents the weight parameters of the online network. ζ represents the moving average. Updating the network using a moving average has the effect of preventing learning collapse. In addition, since past parameters are included in the target network's calculations, it is expected to contribute to improving the model's expressive power by utilizing its past knowledge.
[0109]
number
[0110] The processor 110 grows the online network by repeating steps S911 to S990 for a large number of endoscopic images.
[0111] In this embodiment described above, the method for generating a learning model involves pre-training the pre-trained model 21 so that it infers the state of the area shown in the endoscopic image IMP based on the input endoscopic image IMP. The method for generating a learning model then uses the pre-trained pre-trained model 21 as the initial model for the learning model 121 and performs fine-tuning learning on the pre-trained model 21.
[0112] According to this embodiment, a pre-trained model 21, which has been pre-trained using some learning method, can be subjected to fine-tuning learning using self-knowledge distillation. In the field of endoscopy, even if the diagnostic results are the same, there is a large variation in image features depending on the site, imaging conditions, or individual differences, making it difficult to perform image diagnosis from endoscopic images. According to this embodiment, in the self-knowledge distillation of fine-tuning learning, learning is performed considering the similarity of image features, so that various image features are associated and learned, enabling highly accurate image diagnosis even in the field of endoscopy.
[0113] Furthermore, in this embodiment, pre-training is self-supervised learning. Self-supervised learning is Contrastive Learning.
[0114] Self-supervised learning is a method that uses unlabeled data to train a model using labels generated within the model itself. Training requires a large amount of data, and pre-labeling this data is extremely labor-intensive. Self-supervised learning allows training on large amounts of data without the need for labeling.
[0115] Contrastive Learning is a type of self-supervised learning that learns based on the similarity between data. In this method, learning progresses so that outputs of similar data move closer together, and outputs of dissimilar data move further apart. By applying such Contrastive Learning to the image diagnosis of H. pylori infection, learning is performed not only by associating the features of anchor endoscopic images with the features of endoscopic images similar to the anchor endoscopic image. As a result, even in situations where input images that appear to represent various H. pylori infection states are mixed in a given case, i.e., when boundary data at the boundary of classification classes is mixed, it becomes possible to obtain a robust image diagnosis in which an accurate diagnosis of the case can be obtained from that boundary data.
[0116] Although this embodiment and its variations have been described above, this disclosure is not limited to each embodiment or its variations as they are, and in the implementation stage, the components can be modified and made concrete without departing from the spirit of the invention. Furthermore, multiple components disclosed in each of the embodiments and variations described above can be combined as appropriate. For example, some components may be deleted from all the components described in each embodiment or variation. Furthermore, components described in different embodiments and variations may be combined as appropriate. In this way, various modifications and applications are possible without departing from the spirit of this disclosure. In addition, any term that is described at least once in the specification or drawings together with a different term that is broader or synonymous may be replaced with that different term at any point in the specification or drawings. [Explanation of symbols]
[0117] 21…Pre-trained model, 100…Learning system, 110…Processor, 120…Memory, 121…Learning model, 130…Display, 140…Operating device, 200…Endoscope system, 210…Processor, 220…Memory, 221…Trained model, 230…Display, 250…Controller, 260…Endoscope, EΨ,Eθ…Encoder, G'θ…Multilayer perceptron, GΨ,Gθ…Projector, IM_1~IM_N…Endoscopic image, LSKD…Loss function, Q…Soft target, S…Softmax function, TGp1…Ground truth label, Y…Similarity matrix, l1~lN…Logit, p1~pN…Prediction probability distribution, y1~yN…Features
Claims
1. A method for generating a learning model for inferring the Helicobacter pylori infection status of the stomach as seen in an input endoscopic image, The steps include inputting an anchor image, selected from N endoscopic images (where N is an integer greater than or equal to 3) and assigned a correct label based on the diagnostic result of the Helicobacter pylori infection status in the stomach, and N-1 endoscopic images from the N endoscopic images excluding the anchor image, into a learning model; The learning model outputs an i-th predicted probability distribution, which is the predicted probability of each class of Helicobacter pylori infection status corresponding to the i-th endoscopic image (where i is an integer between 1 and N) from the N endoscopic images, thereby outputting N predicted probability distributions, including an anchor predicted probability distribution corresponding to the anchor image and N-1 predicted probability distributions corresponding to the N-1 endoscopic images. The learning model includes the step of calculating N features by having the encoder calculate the i-th feature from the i-th endoscopic image, The steps include: calculating similarity information regarding the similarity between the i-th feature and the j-th feature (where j is an integer between 1 and N) from among the N features; A step of calculating a soft target from the similarity information and the N predicted probability distributions, The steps include updating the weight parameters of the encoder using a loss function in which the hard target loss is defined by the anchor prediction probability distribution and the ground truth label assigned to the anchor image, and the soft target loss is defined by the N prediction probability distributions and the soft target; Includes, A method for generating a learning model, characterized in that the N endoscopic images are composed of a combination of multiple cases, multiple locations of the stomach, multiple imaging conditions, and multiple ground truth labels.
2. In the method for generating a learning model described in claim 1, The hard target loss is the cross-entropy loss, A method for generating a learning model, characterized in that the soft target loss is KL divergence.
3. In the method for generating a learning model described in claim 1, The i-prediction probability distribution mentioned above includes the prediction probabilities of K classes (where K is an integer greater than or equal to 2). A method for generating a learning model, characterized in that the N endoscopic images input to the learning model include an endoscopic image to which a first class ground truth label is assigned from among the K classes, and an endoscopic image to which a second class ground truth label different from the first class is assigned from among the K classes.
4. In the method for generating a learning model described in claim 1, The steps include: pre-training a pre-trained model to learn how to represent endoscopic images based on input endoscopic images; The steps include performing fine-tuning learning on the pre-trained model by executing the method for generating a learning model described in claim 1, using the pre-trained model as the initial model of the learning model, A method for generating a learning model, characterized by including the following:
5. In the method for generating a learning model described in claim 4, A method for generating a learning model, characterized in that the pre-training is self-supervised learning.
6. In the method for generating a learning model described in claim 5, The method for generating a learning model is characterized in that the self-supervised learning described above is Contrastive Learning.