Image processing device, image processing method, and program

Abstract

An image processing device 1X comprises a first acquisition means 30X, a second acquisition means 31X, and an inference means 33X. The first acquisition means 30X acquires a set value of a first index indicating the accuracy of lesion analysis. The second acquisition means 31X acquires, for each of a plurality of models performing inference on lesions, a prediction value of a second index that is different from the first index and indicates the accuracy in a case where the model satisfies the set value of the first index. The inference means 33X performs inference on lesions in an endoscopic image capturing a subject on the basis of the prediction values of the second index and the plurality of models.

Description

Technical Field

[0001] The present disclosure relates to the technical field of an image processing apparatus, an image processing method, and a storage medium for processing an image acquired in an endoscopic examination.

Background Art

[0002] Conventionally, an endoscopic examination system for assisting in the diagnosis of an endoscopic examination has been known. For example, Patent Document 1 discloses a diagnostic support system that generates a plurality of detection results based on a plurality of discriminator candidates whose detection results are different from each other when detecting a target region from an input image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a system for automatically detecting a lesion using an endoscopic image in an endoscopic examination, requirements by a user for various indexes (for example, sensitivity, specificity, etc.) regarding the accuracy of lesion detection are different.

[0005] In view of the above problems, one object of the present disclosure is to provide an image processing apparatus, an image processing method, and a storage medium capable of suitably detecting a lesion in an endoscopic examination.

Means for Solving the Problems

[0006] One aspect of the image processing apparatus is a first acquisition means for acquiring a set value of a first index indicating the accuracy regarding lesion analysis, A second acquisition means for obtaining a predicted value of a second indicator, which is an indicator of accuracy different from the first indicator, when the set value of the first indicator is satisfied for each of the multiple models that perform inferences about the lesion, An inference means that performs inferences regarding lesions in endoscopic images taken of a subject, based on the predicted values ​​and the plurality of models, This is an image processing device that has the following features.

[0007] One aspect of the image processing method is: Computers The setting value for the first indicator, which shows the accuracy of lesion analysis, is obtained. For each of the multiple models that perform inferences about lesions, the predicted value of the second indicator, which is an indicator of accuracy different from the first indicator, is obtained when the set value of the first indicator is satisfied. Based on the predicted values ​​and the multiple models, inferences are made regarding lesions in endoscopic images taken of the subject. This is an image processing method.

[0008] program One aspect of this is, The setting value for the first indicator, which shows the accuracy of lesion analysis, is obtained. For each of the multiple models that perform inferences about lesions, the predicted value of the second indicator, which is an indicator of accuracy different from the first indicator, is obtained when the set value of the first indicator is satisfied. Based on the predicted values ​​and the multiple models, the computer is instructed to perform a process of inference regarding lesions in endoscopic images taken of the subject. program That is the case. [Effects of the Invention]

[0009] One example of the effects of this disclosure is that lesions can be suitably detected during endoscopic examinations. [Brief explanation of the drawing]

[0010] [Figure 1] This shows the general configuration of an endoscopic examination system. [Figure 2]Shows the hardware configuration of the image processing apparatus. [Figure 3] It is a functional block diagram of an image processing apparatus related to lesion analysis processing in the first embodiment. [Figure 4] It is a graph showing the correspondence between the first index and the second index indicated by the index correspondence information. [Figure 5] It is a diagram showing the predicted values of the second index of each model when the set value Vs1 of the first index is set. [Figure 6] It is a diagram showing the predicted values of the second index of each model when the set value Vs2 of the first index is set. [Figure 7] It is an example of a flowchart showing an outline of the processing executed by the image processing apparatus in the first embodiment. [Figure 8] It is a schematic configuration diagram of an endoscope inspection system in a modification of the first embodiment. [Figure 9] It is a functional block diagram of an image processing apparatus related to lesion analysis processing in the second embodiment. [Figure 10] It is an example of a flowchart showing an outline of the processing executed by the image processing apparatus in the second embodiment. [Figure 11] It is a block diagram of an image processing apparatus in the third embodiment. [Figure 12] It is an example of a flowchart executed by the image processing apparatus in the third embodiment.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of an image processing apparatus, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] <First Embodiment> (1-1) System Configuration Figure 1 shows a schematic configuration of the endoscopic examination system 100. As shown in Figure 1, the endoscopic examination system 100 is a system that presents an area on the image of a subject suspected of having a lesion (also called the "lesion area") to an examiner, such as a physician, who is performing an examination or treatment using an endoscope, and mainly comprises an image processing device 1, a display device 2, and an endoscope scope 3 connected to the image processing device 1.

[0013] The image processing device 1 acquires images (also called "endoscopic image Ia") taken by the endoscope scope 3 in a time series from the endoscope scope 3 and displays a screen based on the endoscopic image Ia on the display device 2. The endoscopic image Ia is an image taken at a predetermined frame period during at least one of the insertion or withdrawal process of the endoscope scope 3 into the patient. In this embodiment, the image processing device 1 performs an analysis of the endoscopic image Ia regarding at least the presence or absence of a lesion area and displays information regarding the analysis results on the display device 2. This allows the image processing device 1 to support the decision-making of examiners, such as physicians, in determining the operation method of the endoscope and determining the treatment plan for the patient being examined. Hereafter, the above-mentioned analysis of endoscopic image Ia will also be called "lesion analysis".

[0014] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

[0015] The endoscope scope 3 mainly consists of an operating section 36 for the examiner to make predetermined inputs, a flexible shaft 37 that is inserted into the organ to be imaged by the patient, a tip section 38 that incorporates an imaging unit such as a miniature image sensor, and a connecting section 39 for connecting to the image processing device 1.

[0016] The configuration of the endoscopic examination system 100 shown in Figure 1 is an example, and various modifications may be made. For example, the image processing device 1 may be configured integrally with the display device 2. In another example, the image processing device 1 may consist of multiple devices.

[0017] In this disclosure, the subject of endoscopic examination is not limited to the large intestine, but may be any organ that can be examined endoscopically, such as the esophagus, stomach, or pancreas. For example, the endoscopes covered in this disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenal endoscopes, small bowel endoscopes, colonoscopes, capsule endoscopes, thoracos

[0018] (a) Head and neck: Pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: Esophageal cancer, esophagitis, hiatal hernia, esophageal varices, achalasia, submucosal tumors of the esophagus, benign tumors of the esophagus (c) Stomach: Stomach cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor (d) Duodenum: Duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: Small intestinal cancer, small intestinal neoplastic diseases, small intestinal inflammatory diseases, small intestinal vascular diseases (f) Large intestine: Colon cancer, colon neoplastic disease, colon inflammatory disease, colon polyps, colon polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

[0019] (1-2) Hardware configuration Figure 2 shows the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, memory 12, interface 13, input unit 14, light source unit 15, and sound output unit 16. Each of these elements is connected via a data bus 19.

[0020] The processor 11 executes predetermined processes by running programs and other data stored in memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0021] Memory 12 consists of various volatile memories used as working memory, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memory that stores information necessary for processing by the image processing device 1. Memory 12 may also include external storage devices such as hard disks connected to or built into the image processing device 1, or it may include storage media such as removable flash memory. Memory 12 stores programs for the image processing device 1 to execute each of the processes in this embodiment.

[0022] Memory 12 also stores model information D1, which is information about the lesion analysis model used for lesion analysis in endoscopic examinations, and index correspondence information D2, which shows the correspondence between two different indices that indicate the accuracy of the lesion analysis model. Examples of indices that indicate the accuracy of the lesion analysis model include sensitivity (i.e., detection rate or true positive rate), specificity, false positive rate, and precision. Model information D1 and index correspondence information D2 will be described later.

[0023] Interface 13 performs interface operations between the image processing device 1 and an external device. For example, interface 13 supplies display information "Ib" generated by the processor 11 to the display device 2. Interface 13 also supplies light generated by the light source unit 15 to the endoscope scope 3. Interface 13 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope scope 3 to the processor 11. Interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or it may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc.

[0024] The input unit 14 generates input signals based on operations performed by the examiner. The input unit 14 can be, for example, a button, touch panel, remote controller, or voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope scope 3. The light source unit 15 may also incorporate a pump for supplying water or air to the endoscope scope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0025] Next, we will explain in detail the model information D1 and index correspondence information D2 stored in memory 12.

[0026] Model information D1 contains parameters for multiple candidate lesion analysis models to be used for lesion analysis.

[0027] Here, the lesion analysis model is a model (engine) that outputs inference results regarding the lesion region in an input endoscopic image when an endoscopic image is input to the model. Each lesion analysis model is a machine learning model (including statistical models, the same applies hereinafter) having an arbitrary architecture such as a neural network or a support vector machine. In this case, each lesion analysis model is trained using training data, and the parameters of the lesion analysis model obtained through training are pre-stored in memory 12 as model information D1. Here, the training data has multiple records, for example, consisting of a pair of an input endoscopic image and ground truth data showing the inference result that each lesion analysis model should output when that endoscopic image is input. Each lesion analysis model may have a different architecture from the others, or may be a model trained on different training data sets. Representative examples of neural networks used in the architecture of lesion analysis models include, for example, Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab. Furthermore, when the lesion analysis model is constructed using a neural network, the model information D1 includes various parameters (including hyperparameters) for each lesion analysis model, such as the layer structure, the neuronal structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter.

[0028] The lesion analysis model may be a lesion detection engine that detects the presence and extent of lesions in the input endoscopic image, a classification engine that classifies the presence or absence of lesions in the endoscopic image, or a segmentation engine that segments the regions of the input endoscopic image based on the presence or absence of lesions.

[0029] If the lesion analysis model is a lesion detection engine, it outputs inference results that show, for example, a bounding box indicating the extent of the lesion region in the input endoscopic image, and the degree of confidence (i.e., confidence score) that the bounding box indicates the extent of the lesion region. The confidence score corresponds to the confidence of each class (here, two classes: lesion present and lesion absent) output from the output layer of the neural network, for example, if the lesion analysis model is composed of a neural network. Hereafter, a higher confidence score indicates a higher probability.

[0030] Furthermore, if the lesion analysis model is a classification engine, it outputs inference results that indicate, for example, the presence or absence of a lesion area in the input endoscopic image, and the corresponding confidence level (classification score) for the presence or absence of the lesion area. If the lesion analysis model is a segmentation engine, it outputs inference results that indicate, for example, the presence or absence of a lesion area at the pixel level (or block level consisting of multiple pixels) in the input endoscopic image, and the pixel-level confidence level for indicating a lesion area. Note that the inference results output by the lesion analysis model described above are just examples, and various types of inference results may be output from the lesion analysis model.

[0031] The presence or absence of a lesion is then determined using the inference results output by the lesion analysis model and a threshold for determining the presence or absence of a lesion (also called the "lesion determination threshold"). For example, the lesion determination threshold is a threshold that can be set by the user and is compared with the confidence level mentioned above. For example, if the lesion analysis model is a classification engine, the image processing device 1 determines that a lesion exists when the confidence level corresponding to the presence of a lesion is greater than or equal to the lesion determination threshold, and determines that a lesion does not exist when the confidence level is less than the lesion determination threshold. Similarly, if the lesion analysis model is a lesion detection engine, the image processing device 1 determines that a lesion exists within a bounding box when the confidence level of the presence of a lesion in the bounding box is greater than or equal to the lesion determination threshold, and determines that a lesion does not exist when the confidence level is less than the lesion determination threshold.

[0032] Furthermore, the sensitivity and specificity of each lesion analysis model can be adjusted by changing the size of the lesion detection threshold used in the lesion analysis model. Additionally, the lesion analysis model may output a result indicating the presence or absence of a lesion area based on the lesion detection threshold. In this case, the lesion detection threshold is used as a hyperparameter of the lesion analysis model.

[0033] The index correspondence information D2 is information that shows the correspondence between two indices (referred to as the "first index" and the "second index") related to the accuracy of each lesion analysis model. As will be described later, the first index is an index in which the target value (also called the "set value") that should be met in the lesion analysis model used is set by the user. Preferably, the pair of the first index and the second index is selected to be in a trade-off relationship (for example, a pair of sensitivity and false positive rate).

[0034] The index correspondence information D2 is, for example, a table of information that shows the pairs of values ​​for the first index and the second index at each lesion detection threshold when the lesion detection threshold is changed for each lesion analysis model. For example, if the pair of the first and second indexes is a pair of sensitivity (true positive rate) and false positive rate, the above table information corresponds to an ROC (Receiver Operating Characteristic) curve, and if the pair of the first and second indexes is a pair of sensitivity and precision, the above table information corresponds to a PR (Precision Recall) curve. The above table information is not limited to information corresponding to an ROC curve or a PR curve, but may also be information corresponding to an LROC (ROC-type curve for task of detection and localization) curve or an FROC (Free-response receiver operating characteristic) curve, for example. Each record in the above table information is associated with a lesion detection threshold corresponding to the pair of values ​​for the first index and the second index.

[0035] The table information described above is generated using test data that has multiple records, each consisting of a pair of an input endoscopic image and ground truth data indicating the presence or absence of a lesion in the endoscopic image. In this case, the correctness (specifically true positive, false positive, false negative, true negative) is determined for each record of the test data while changing the lesion determination threshold for each lesion analysis model, and the table information described above is generated by aggregating the determination results for each lesion analysis model and each lesion determination threshold. The table information described above is then stored in memory 12 as index correspondence information D2. Therefore, index correspondence information D2 is table information that shows the relationship between the first indicator and the second indicator based on actual values ​​from the test data.

[0036] (1-3) Overview of lesion analysis processing This section describes the lesion analysis process, which is a process related to lesion analysis. In general, the image processing device 1 obtains the predicted value of the second indicator (also called the "predicted value") for each lesion analysis model, based on the indicator correspondence information D2, assuming that the setting value of the first indicator set by the user is met. The image processing device 1 then performs lesion analysis using the lesion analysis model with the best predicted value for the second indicator. This makes it possible for the image processing device 1 to perform lesion analysis using the lesion analysis model that is predicted to yield the best results for the second indicator (i.e., the best performance) while satisfying the setting value of the first indicator set by the user.

[0037] Figure 3 is a functional block diagram of the image processing device 1 related to lesion analysis processing. Functionally, the processor 11 of the image processing device 1 includes a set value acquisition unit 30, a predicted value acquisition unit 31, a model selection unit 32, and a lesion analysis unit 33. In Figure 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to this. The same applies to the diagrams of other functional blocks described later.

[0038] The setting value acquisition unit 30 accepts input of a setting value for the first indicator and determines the setting value for the first indicator based on the input signal received from the input unit 14 via the interface 13. In this case, the setting value acquisition unit 30 may generate a display signal for displaying a display screen that accepts input of a setting value for the first indicator, and supply the display signal to the display device 2 to display the above-mentioned display screen on the display device 2. In this case, the setting value acquisition unit 30 may display a user interface for accepting input specifying a specific numerical value for the first indicator, or it may display a user interface for accepting input specifying a level for the first indicator. In the latter example where the first indicator is sensitivity, the setting value acquisition unit 30 presents three options, for example, "high sensitivity mode" corresponding to sensitivity "0.95", "high specificity mode" corresponding to sensitivity "0.80", and "intermediate mode" corresponding to sensitivity "0.90", and accepts input specifying the option to be adopted. The setting value acquisition unit 30 notifies the prediction value acquisition unit 31 of the acquired setting value for the first indicator.

[0039] The prediction value acquisition unit 31 acquires, based on the index correspondence information D2, the predicted value of the second index that is predicted when the setting value of the first index notified by the setting value acquisition unit 30 is met, for each lesion analysis model. In this case, the prediction value acquisition unit 31 extracts a record corresponding to the setting value of the first index in the index correspondence information D2 for each lesion analysis model, and acquires the value of the second index recorded in the record as the predicted value of the second index for each lesion analysis model. In this case, if the value of the first index corresponding to the setting value of the first index is not recorded in the index correspondence information D2, the prediction value acquisition unit 31 may acquire the value of the second index corresponding to the first index value closest to the setting value as the predicted value described above, or it may calculate the value of the second index corresponding to the setting value of the first index by arbitrary interpolation processing.

[0040] Furthermore, the prediction value acquisition unit 31 acquires, for each lesion analysis model, a set value of the first indicator and a predicted value of the second indicator, along with a lesion determination threshold associated with the indicator correspondence information D2. This lesion determination threshold is the threshold to be used for the inference results output by each lesion analysis model. The prediction value acquisition unit 31 then supplies the predicted value of the second indicator and the lesion determination threshold acquired for each lesion analysis model to the model selection unit 32.

[0041] The model selection unit 32 compares the predicted values ​​of the second index for each lesion analysis model supplied from the predicted value acquisition unit 31, and selects the lesion analysis model corresponding to the predicted value of the second index that shows the best accuracy as the lesion analysis model to be used for lesion analysis. The model selection unit 32 then supplies the lesion analysis unit 33 with information indicating the selection result of the lesion analysis model and the lesion determination threshold corresponding to the lesion analysis model. Alternatively, the model selection unit 32 may read model information D1 corresponding to the selected lesion analysis model from the memory 12 and supply the read model information D1 to the lesion analysis unit 33.

[0042] The lesion analysis unit 33 performs lesion analysis based on the endoscopic image Ia supplied from the endoscope scope 3 after the start of the endoscopic examination, the lesion analysis model notified by the model selection unit 32, and the lesion determination threshold, and determines whether or not there is a lesion area in the endoscopic image Ia. In this case, the lesion analysis unit 33 constructs the lesion analysis model by referring to the model information D1 corresponding to the lesion analysis model selected by the model selection unit 32. Then, the lesion analysis unit 33 inputs the endoscopic image Ia into the constructed lesion analysis model and determines whether or not there is a lesion area in the input endoscopic image Ia based on the inference results output from the lesion analysis model and the lesion determination threshold. For example, the lesion analysis unit 33 determines that a lesion area exists if the confidence level of the presence of a lesion area included in the inference results output by the lesion analysis model is equal to or greater than the lesion determination threshold, and determines that there is no lesion area if the confidence level is less than the lesion determination threshold.

[0043] Furthermore, the lesion analysis unit 33 may function as an output control means that outputs via a display device or an audio output device. For example, the lesion analysis unit 33 displays the latest endoscopic image Ia and the results of the lesion analysis on the display device 2. In this case, the lesion analysis unit 33 generates display information Ib based on the latest endoscopic image Ia supplied from the endoscope scope 3 and the results of the lesion analysis based on the endoscopic image Ia, and supplies the generated display information Ib to the display device 2. In addition to the determination result of whether or not a lesion area exists, the results of the lesion analysis may also include information indicating the extent of existence of the lesion area output by the lesion analysis model used. In this case, the lesion analysis unit 33 displays on the display device 2 the latest endoscopic image Ia and the determination result of whether or not a lesion area exists, as well as a mask image indicating the extent of existence of the lesion area in the endoscopic image Ia, or a frame that outlines the area of ​​the lesion area on the endoscopic image Ia. If the lesion analysis unit 33 determines that a lesion area exists, it may control the sound output of the sound output unit 16 to output a warning sound or voice guidance to notify the user that a lesion area exists. Furthermore, the lesion analysis unit 33 may, for example, determine and output a treatment method based on a model generated by machine learning the correspondence between the results of lesion analysis and treatment methods, and the results of the lesion analysis of the subject. The method for determining the treatment method is not limited to the method described above. Outputting a treatment method can further support the examiner's decision-making.

[0044] The setting value acquisition unit 30, the predicted value acquisition unit 31, the model selection unit 32, and the lesion analysis unit 33 can be implemented, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to implement each component. At least a portion of these components may be implemented not only by software programs, but also by any combination of hardware, firmware, and software. At least a portion of these components may also be implemented using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program composed of the above components may be implemented using this integrated circuit. At least a portion of each component may also be implemented using an ASSP (Application Specific Standard Produce), ASIC (Application Specific Integrated Circuit), or quantum processor (quantum computer control chip). Thus, each component may be implemented using various hardware. The same applies to other embodiments described later. Furthermore, each of these components may be implemented by the collaboration of multiple computers, for example, using cloud computing technology.

[0045] (1-4) Specific examples of model selection Next, we will explain a specific example of the selection of a lesion analysis model by the model selection unit 32.

[0046] Figure 4 is a graph showing the correspondence between the first and second indicators as indicated by the indicator correspondence information D2. Here, three models (first model, second model, and third model) are registered in the model information D1 as lesion analysis models, and graphs G1, G2, and G3 are shown, corresponding to the first model, the second model, and the third model, respectively. Graphs G1 to G3 are curves on a two-dimensional coordinate system with the first and second indicators as the coordinate axes, and were obtained by aggregating the results of the inference results (specifically true positive, false positive, false negative, and true negative) using test data while varying the lesion judgment threshold for each of the three models mentioned above.

[0047] Here, the first indicator is one indicative of good accuracy (e.g., sensitivity), where a higher value indicates better accuracy, and the second indicator is one indicative of good accuracy (e.g., false positive rate), where a lower value indicates better accuracy. Note that the combination of the first and second indicators shown in Figure 4 is just one example; for example, the first indicator could be one indicative of good accuracy (e.g., false positive rate) where a lower value indicates better accuracy, and the second indicator could be one indicative of good accuracy (e.g., sensitivity) where a higher value indicates better accuracy. Furthermore, there is a trade-off relationship between the first and second indicators, where improvement in one indicator leads to deterioration in the other. When the first indicator is sensitivity and the second indicator is the false positive rate, graphs G1 to G3 correspond to ROC curves. In addition, any two graphs G1 to G3 intersect at least one point.

[0048] Figure 5 clearly shows the predicted values ​​"Vp11" to "Vp13" of the second indicator for each model when the setting value "Vs1" for the first indicator is set, in graphs G1 to G3. When the user sets the setting value Vs1 for the first indicator, the prediction value acquisition unit 31 acquires the predicted values ​​of the second indicator for the first model, second model, and third model when the setting value Vs1 for the first indicator is satisfied. For example, in the case of the first model, the prediction value acquisition unit 31 recognizes the corresponding point "P11" in graph G1 that corresponds to the setting value Vs1 for the first indicator, and acquires the value of the second indicator corresponding to the corresponding point P11 (here, Vp11) as the predicted value of the second indicator. Similarly, for the second model and third model, the prediction value acquisition unit 31 recognizes the corresponding points "P12" and "P13" that correspond to the setting value Vs1 for the first indicator, and acquires the values ​​of the second indicator corresponding to the corresponding points P12 and P13 (here, Vp12 and Vp13) as the predicted values ​​of the second indicator.

[0049] The model selection unit 32 then identifies the prediction value that shows the best accuracy among the prediction values ​​of the second indicator for each model, and selects a lesion analysis model corresponding to the identified prediction value of the second indicator. Here, the model selection unit 32 selects the second model corresponding to the prediction value Vp12 because the prediction value Vp12 is the lowest and best among the prediction values ​​Vp11 to Vp13 of the second indicator. The lesion analysis unit 33 then performs lesion analysis based on the endoscopic image Ia using the second model thus selected and the lesion determination threshold linked to the corresponding point P12. As a result, the image processing device 1 can perform lesion analysis using a lesion analysis model that is predicted to satisfy the user-set value Vs1 of the first indicator while also yielding the best results (i.e., having the best performance) for the second indicator.

[0050] Figure 6 clearly shows the predicted values ​​"Vp21" to "Vp23" of the second indicator for each model when the setting value "Vs2" for the first indicator is set, in graphs G1 to G3. When the user sets the setting value Vs2 for the first indicator, the prediction value acquisition unit 31 acquires the predicted values ​​of the second indicator for the first model, second model, and third model when the setting value Vs2 for the first indicator is satisfied. For example, in the case of the first model, the prediction value acquisition unit 31 recognizes the corresponding point "P21" in graph G1 that corresponds to the setting value Vs2 for the first indicator, and acquires the value of the second indicator corresponding to corresponding point P21 (here, Vp21) as the predicted value of the second indicator. Similarly, for the second model and third model, the prediction value acquisition unit 31 recognizes the corresponding points "P22" and "P23" that correspond to the setting value Vs2 for the first indicator, and acquires the values ​​of the second indicator corresponding to corresponding points P22 and P23 (here, Vp22 and Vp23) as the predicted values ​​of the second indicator.

[0051] The model selection unit 32 then identifies the prediction value that shows the best accuracy among the predicted values ​​of the second indicator for each model and selects the lesion analysis model corresponding to the identified prediction value of the second indicator. Here, the model selection unit 32 selects the corresponding first model because the prediction value Vp21 is the lowest among the predicted values ​​Vp21 to Vp23 of the second indicator, making it the best. Note that since graphs G1 to G3 intersect with each other, as the setting value of the first indicator changes from value Vs1 to value Vs2, the model that gives the best prediction value of the second indicator also changes from the second model to the first model. The lesion analysis unit 33 then performs lesion analysis based on the endoscopic image Ia using the first model thus selected and the lesion determination threshold linked to the corresponding point P21. As a result, the image processing device 1 can perform lesion analysis using a lesion analysis model that satisfies the user-set setting value Vs2 for the first indicator and is predicted to give the best result (i.e., the best performance) for the second indicator as well.

[0052] (1-5) Processing flow Figure 7 is an example of a flowchart showing an overview of the processes performed by the image processing device 1 in the first embodiment. The image processing device 1 starts the processes outlined in the flowchart before the start of the endoscopic examination.

[0053] First, the image processing device 1 receives an input specifying the setting value of the first indicator (step S11). For example, the image processing device 1 receives an input signal specifying the setting value of the first indicator from the input unit 14. Then, the image processing device 1 obtains the predicted value of the second indicator for each lesion analysis model corresponding to the setting value of the first indicator specified in step S11 (step S12). In this case, the image processing device 1 refers to the indicator correspondence information D2 and obtains the value of the second indicator corresponding to the setting value of the first indicator as the predicted value for each lesion analysis model.

[0054] Next, the image processing device 1 selects a lesion analysis model and a lesion determination threshold to be used for lesion analysis based on the predicted value of the second indicator of each lesion analysis model (step S13). In this case, the image processing device 1 identifies the lesion analysis model with the best predicted value of the second indicator and the lesion determination threshold associated with that predicted value in the indicator correspondence information D2.

[0055] The image processing device 1 then determines whether or not the endoscopic examination has started (step S14). For example, the image processing device 1 determines that the endoscopic examination has started when it receives an endoscopic image Ia from the endoscope scope 3 via the interface 13. If the endoscopic examination has not started (step S14; No), the image processing device 1 continues to execute step S14.

[0056] Then, if it is determined that the endoscopic examination has started (step S14; Yes), the image processing device 1 performs a lesion analysis based on the latest endoscopic image Ia received from the endoscope 3 via the interface 13, and the lesion analysis model and lesion determination threshold selected in step S13 (step S15). In this case, the image processing device 1 also displays the latest endoscopic image Ia and the lesion analysis results on the display device 2.

[0057] Then, after step S15, the image processing device 1 determines whether or not the endoscopic examination has been completed (step S16). For example, the image processing device 1 determines that the endoscopic examination has been completed when it detects a predetermined input to the input unit 14 or the operation unit 36. If the image processing device 1 determines that the endoscopic examination has been completed (step S16; Yes), it terminates the processing of the flowchart. On the other hand, if the image processing device 1 determines that the endoscopic examination has not been completed (step S16; No), it returns to step S15. Then, the image processing device 1 executes the processing of step S15 on the endoscopic image Ia newly generated by the endoscope scope 3.

[0058] (1-6) Variation Next, a preferred modification of the first embodiment described above will be explained. The following modifications may be applied in combination to the first embodiment described above.

[0059] (Extreme Variation 1-1) The setting value acquisition unit 30 may acquire the setting value of the first indicator from the memory 12 or an external device, instead of acquiring the setting value of the first indicator based on the input signal of the input unit 14.

[0060] In this case, for example, when an endoscopic examination is started, the image processing device 1 reads or receives the setting value of the first indicator from the memory 12 where the setting value of the first indicator is stored or from an external device. Subsequently, the image processing device 1 selects a lesion analysis model based on the acquired setting value of the first indicator, and performs lesion analysis based on the selected lesion analysis model and the endoscopic image Ia acquired during the endoscopic examination. In this embodiment as well, the image processing device 1 can acquire the setting value of the first indicator and perform lesion analysis based on the lesion analysis model that has the best performance to satisfy the setting value of the first indicator.

[0061] (Variations 1-2) Model information D1 and index correspondence information D2 may be stored in a storage device separate from the image processing device 1.

[0062] Figure 8 is a schematic diagram of the endoscopic examination system 100A in a modified example. For simplicity, the display device 2 and endoscope scope 3 are not shown. The endoscopic examination system 100A includes a server device 4 that stores model information D1 and indicator correspondence information D2. The endoscopic examination system 100A also includes a plurality of image processing devices 1 (1A, 1B, ...) that can communicate data with the server device 4 via a network.

[0063] In this case, each image processing device 1 accesses model information D1 and index correspondence information D2 via the network. In this case, the interface 13 of each image processing device 1 includes a communication interface such as a network adapter for communication. In this configuration, each image processing device 1 can access model information D1 and index correspondence information D2, similar to the embodiment described above, and suitably perform processing related to lesion detection.

[0064] <Second Embodiment> In the second embodiment, instead of using the predicted value of the second index to select a lesion analysis model, the image processing device 1 uses the predicted value of the second index to set the weights of the inference results of each lesion analysis model when integrating the inference results of multiple lesion analysis models to determine the presence or absence of a lesion site. The hardware configuration of the image processing device 1 in the second embodiment is as shown in Figure 2. Hereafter, the same reference numerals are used for components that are the same as in the first embodiment, and their descriptions are omitted as appropriate.

[0065] (2-1) Functional Blocks Figure 9 is a functional block diagram of the image processing device 1 in the second embodiment. Functionally, the processor 11 of the image processing device 1 includes a set value acquisition unit 30, a predicted value acquisition unit 31, a weight determination unit 32A, and a lesion analysis unit 33A. The processing performed by the set value acquisition unit 30 and the predicted value acquisition unit 31 is the same as in the first embodiment, so their explanation is omitted.

[0066] The weight determination unit 32A determines the weights to be used for integrating the inference results of each lesion analysis model based on the predicted values ​​of the second indicator corresponding to each lesion analysis model whose parameters are registered in the model information D1 supplied from the predicted value acquisition unit 31. For example, the weight determination unit 32A sets the weights when calculating the ensemble average of the confidence level of the presence of a lesion area based on the predicted values ​​of the second indicator. In this case, for example, if a lower value of the second indicator indicates better accuracy, a value that has a negative correlation with the predicted value of the second indicator (for example, the reciprocal of the predicted value or the value obtained by subtracting the predicted value from the maximum value of the second indicator) is set as the weight. On the other hand, if a higher value of the second indicator indicates better accuracy, a value that has a positive correlation with the predicted value of the second indicator (for example, the predicted value of the second indicator itself) is set as the weight. If an expression or table showing the correspondence between the predicted value of the second indicator and the weight is stored in memory 12, the weight determination unit 32A may refer to such expression or table and determine the weight from the predicted value of the second indicator. Preferably, the weight determination unit 32A normalizes the weights so that the sum of the weights used to integrate the inference results of each lesion analysis model is 1. The weight determination unit 32A supplies information indicating the weight for each lesion analysis model to the lesion analysis unit 33A.

[0067] The lesion analysis unit 33A inputs the endoscopic image Ia to each lesion analysis model whose parameters are registered in the model information D1, and obtains the inference results output by each lesion analysis model. The lesion analysis unit 33A then integrates the inference results output by each lesion analysis model using the weights determined by the weight determination unit 32A, and based on the integrated inference results, it makes a determination as to whether or not there is a lesion area in the input endoscopic image Ia.

[0068] A concrete example of integrating weight-based inference results will be explained. Here, we consider the case in Figure 5 where the prediction value acquisition unit 31 acquires the predicted values ​​Vp11 to Vp13 of the second indicator for the set value Vs1 of the first indicator. The confidence level of the existence of a lesion region output by the first model is "s1", the confidence level of the existence of a lesion region output by the second model is "s2", and the confidence level of the existence of a lesion region output by the third model is "s3".

[0069] In this case, the lesion analysis unit 33A calculates the final confidence level "s" to be compared with the lesion determination threshold by weighted averaging as follows. For example, the predicted values ​​Vp11 to Vp13 of the second indicator are assumed to have a range of minimum value 0 and maximum value 1. s={(s1×(1-Vp11))+(s2×(1-Vp12))+(s3×(1-Vp13))} / (Vp11+Vp12+Vp13)

[0070] In the above formula, for example, the lesion analysis unit 33A sets the weight to the value obtained by subtracting the predicted value from the maximum value of the second indicator. However, instead, any value that has a negative correlation with the predicted value of the second indicator, such as the reciprocal of the predicted value of the second indicator, may be set as the weight.

[0071] The lesion analysis unit 33A determines that a lesion region exists if the final confidence level s is equal to or greater than the lesion determination threshold, and determines that a lesion region does not exist if the final confidence level s is less than the lesion determination threshold. The lesion determination threshold used here may be a default value pre-stored in memory 12, etc., or it may be the average of the lesion determination thresholds corresponding to the predicted values ​​of the second indicator for each lesion analysis model (including a weighted average based on the predicted values ​​of the second indicator).

[0072] In this way, the image processing device 1 can perform more accurate lesion analysis by integrating the inference results of multiple lesion analysis models, whose parameters are recorded in the model information D1, using weighting based on the predicted value of the second indicator.

[0073] (2-2) Processing flow Figure 10 is an example of a flowchart showing an overview of the processes performed by the image processing device 1 in the second embodiment. The image processing device 1 starts the processes outlined in the flowchart before the start of the endoscopic examination.

[0074] First, the image processing device 1 receives an input specifying the setting value of the first indicator (step S21). Then, the image processing device 1 obtains the predicted value of the second indicator of each lesion analysis model corresponding to the setting value of the first indicator specified in step S11 (step S22).

[0075] Next, the image processing device 1 determines the weight of each lesion analysis model based on the predicted value of the second index of each lesion analysis model whose parameters are registered in the model information D1 (step S23). Then, the image processing device 1 determines whether or not the endoscopic examination has been started (step S24). If the endoscopic examination has not been started (step S24; No), the image processing device 1 continues to execute step S24.

[0076] Then, if it is determined that the endoscopic examination has started (Step S24; Yes), the image processing device 1 performs a lesion analysis based on the latest endoscopic image Ia received from the endoscope 3 via the interface 13, each lesion analysis model, and the weights of each lesion analysis model (Step S25). In this case, the image processing device 1 also displays the latest endoscopic image Ia and the lesion analysis results on the display device 2.

[0077] Then, after step S15, the image processing device 1 determines whether or not the endoscopic examination has been completed (step S26). If the image processing device 1 determines that the endoscopic examination has been completed (step S26; Yes), it terminates the processing in the flowchart. On the other hand, if the image processing device 1 determines that the endoscopic examination has not been completed (step S26; No), it returns to step S25. Then, the image processing device 1 executes the processing in step S25 on the endoscopic image Ia newly generated by the endoscope scope 3.

[0078] (2-3) Variation Next, a modified example suitable for the second embodiment described above will be explained. The following modified examples and "(1-6) Variation The modifications described above may be applied in combination to the second embodiment described above.

[0079] (Variation 2-1) The image processing device 1 may switch between performing lesion analysis based on the first embodiment (i.e., lesion analysis using a single selected lesion analysis model) and lesion analysis based on the second embodiment (i.e., lesion analysis using multiple lesion analysis models).

[0080] In the first example, the image processing device 1 performs lesion analysis based on the first embodiment at the start of an endoscopic examination, and if it detects a disease other than the target disease (also called a "background disease") in the latest endoscopic image Ia, it switches from lesion analysis based on the first embodiment to lesion analysis based on the second embodiment. Background diseases include, for example, inflammation, bleeding, Barrett's esophagus if the subject is the esophagus, and any other disease other than the target disease. In this case, for example, memory 12 stores information about a model that outputs information about background diseases when an endoscopic image is input, and the image processing device 1 determines the presence or absence of a background disease based on the information output by the model when the latest endoscopic image Ia is input to the model. For example, the above model is a classification model that outputs a classification result regarding the presence or absence (and degree) of a background disease when an endoscopic image is input. The classification result output by the above model may indicate the presence or absence (and degree) of a specific type of background disease, or it may indicate the type (and degree) of the corresponding background disease. The above model may be any machine learning model (including statistical models, the same applies hereinafter) such as a neural network or a support vector machine.

[0081] Thus, in the first example, the image processing device 1 performs lesion analysis based on the second embodiment, which is expected to yield more accurate inference results when an underlying disease is found. The image processing device 1 switches from lesion analysis based on the second embodiment to lesion analysis based on the first embodiment when it no longer detects an underlying disease. By doing so, the computationally intensive lesion analysis based on the second embodiment is performed only in critical situations where high-precision lesion analysis is desired, thereby obtaining highly accurate lesion analysis results in critical situations while suitably reducing the computational load in other situations.

[0082] In the second example, the image processing device 1 switches between lesion analysis based on the first embodiment and lesion analysis based on the second embodiment based on the examiner's input (i.e., external input) via the input unit 14 or the operation unit 36. In this case, for example, the image processing device 1 performs lesion analysis based on the first embodiment at the start of the endoscopic examination, and switches from lesion analysis based on the first embodiment to lesion analysis based on the second embodiment when it detects a predetermined external input. Furthermore, after switching to lesion analysis based on the second embodiment, the image processing device 1 switches back from lesion analysis based on the second embodiment to lesion analysis based on the first embodiment when it detects a predetermined external input.

[0083] Thus, in the second example, the lesion analysis based on the first embodiment and the lesion analysis based on the second embodiment are switched based on the examiner's input (i.e., external input) via the input unit 14 or the operation unit 36. In this example as well, the computationally intensive lesion analysis based on the second embodiment is executed only when the user wants to perform lesion analysis with high accuracy, allowing for highly accurate lesion analysis results when the user wants to perform lesion analysis with high accuracy, while suitably reducing the computational load in other situations.

[0084] In this modified example, the lesion analysis based on the first embodiment is an example of the "first mode," and the lesion analysis based on the second embodiment is an example of the "second mode." Furthermore, the presence or absence of detection of a background disease in the first example and the presence or absence of detection of an external input in the second example are examples of "predetermined conditions."

[0085] (Variation 2-2) The image processing device 1 may perform lesion analysis by combining the first embodiment and the second embodiment.

[0086] For example, the image processing device 1 selects a predetermined number (two or more) of lesion analysis models from each lesion analysis model whose parameters are recorded in the model information D1, the models that have superior predicted values ​​for the corresponding second indicator. The image processing device 1 then integrates the inference results of the selected predetermined number of lesion analysis models using weights set based on the predicted values ​​for the corresponding second indicator. According to this embodiment, the image processing device 1 can obtain highly accurate inference results by limiting the lesion analysis models used for lesion analysis to those predicted to have good performance and by integrating the inference results of the limited lesion analysis models.

[0087] <Third Embodiment> Figure 11 is a block diagram of the image processing apparatus 1X in the third embodiment. The image processing apparatus 1X comprises a first acquisition means 30X, a second acquisition means 31X, and an inference means 33X. The image processing apparatus 1X may be composed of multiple devices.

[0088] The first acquisition means 30X acquires a set value of a first indicator that shows the accuracy of the lesion analysis. The first acquisition means 30X can be, for example, the set value acquisition unit 30 in the first or second embodiment.

[0089] The second acquisition means 31X acquires a predicted value of a second indicator, which is an indicator with a different accuracy than the first indicator, for each of the multiple models that perform inferences about the lesion, when the set value of the first indicator is met. The second acquisition means 31X can be, for example, the predicted value acquisition unit 31 in the first or second embodiment.

[0090] The inference means 33X performs inferences regarding lesions in endoscopic images taken of the subject, based on the predicted value of the second indicator and a plurality of models. The inference means 33X can be, for example, the model selection unit 32 and lesion analysis unit 33 in the first embodiment, or the weight determination unit 32A and lesion analysis unit 33A in the second embodiment.

[0091] Figure 12 is an example of a flowchart showing the processing procedure in the third embodiment. The first acquisition means 30X acquires a set value of a first indicator that indicates the accuracy of lesion analysis (step S31). The second acquisition means 31X acquires a predicted value of a second indicator, which is an indicator of accuracy different from the first indicator, for each of the multiple models that perform inference about lesions, when the set value of the first indicator is satisfied (step S32). The inference means 33X performs inference about lesions in endoscopic images taken of the subject based on the predicted value of the second indicator and the multiple models (step S33).

[0092] According to the third embodiment, the image processing device 1X can suitably perform inferences regarding lesions in endoscopic images of a subject, taking into account the predicted values ​​of the second indicator of each model when the set value of the first indicator is satisfied.

[0093] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable medium and supplied to a computer, such as a processor. Non-transitory computer-readable mediums include various types of tangible storage mediums. Examples of non-transitory computer-readable mediums include magnetic storage mediums (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage mediums (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to the computer by various types of transient computer-readable mediums. Examples of transient computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable mediums can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0094] In addition, some or all of the above embodiments (including modifications, the same applies hereinafter) may also be described as follows, but are not limited to the following.

[0095] [Note 1] A first acquisition means for obtaining a set value of a first indicator that shows the accuracy of lesion analysis, A second acquisition means for obtaining a predicted value of a second indicator, which is an indicator of accuracy different from the first indicator, when the set value of the first indicator is satisfied for each of the multiple models that perform inferences about the lesion, An inference means that performs inferences regarding lesions in endoscopic images taken of a subject, based on the predicted values ​​and the plurality of models, An image processing device having [Note 2] The image processing apparatus according to Appendix 1, wherein the second acquisition means acquires correspondence information showing the correspondence between the first index and the second index for each of the plurality of models, and acquires the value of the second index corresponding to the set value in the correspondence information as the predicted value. [Note 3] The aforementioned correspondence information represents an ROC curve, LROC curve, FROC curve, or PR curve, as described in Appendix 2 of the image processing apparatus. [Note 4] The image processing apparatus according to Appendix 2, wherein each of the curves of the plurality of models on a two-dimensional coordinate system with the first index and the second index as coordinate axes, as shown by the correspondence information, has an intersection point where they intersect each other. [Note 5] The inference means selects the model with the best accuracy indicated by the predicted value from the plurality of models, and performs the inference based on the selected model and the endoscopic image, as described in Appendix 1. [Note 6] The inference means generates an inference result by weighting the inference results of each of the plurality of models based on the endoscopic image based on the predicted value, as described in Appendix 1. [Note 7] The aforementioned inference means is A first mode in which the model with the best accuracy shown by the predicted value is selected from the plurality of models, and the inference is performed based on the selected model and the endoscopic image, The image processing apparatus described in Appendix 1, wherein the inference means switches between and executes a second mode, which generates an inference result in which the inference results of each of the plurality of models based on the endoscopic image are weighted based on the predicted value, based on predetermined conditions. [Note 8] The image processing apparatus according to Appendix 7, wherein the inference means switches between the first mode and the second mode and executes based on whether or not a disease other than the target disease is detected in the lesion analysis. [Note 9] The inference means is an image processing apparatus according to Appendix 7, which switches between the first mode and the second mode based on an external input. [Note 10] The inference means selects a predetermined number of top models from the plurality of models that have superior accuracy as indicated by the predicted value, and generates an inference result by weighting the inference results of each of the predetermined number of top models based on the predicted value, as described in Appendix 1. [Note 11] The image processing apparatus described in Appendix 1 is a model that has been trained using a set of endoscopic images and ground truth data indicating the inference result that the model should output when the endoscopic image is input as training data. [Note 12] The image processing apparatus according to Appendix 1, further comprising output control means for outputting the inference result obtained by the inference means via a display device or audio output device to support the examiner's decision-making. [Note 13] Computers The setting value for the first indicator, which shows the accuracy of lesion analysis, is obtained. For each of the multiple models that perform inferences about lesions, the predicted value of the second indicator, which is an indicator of accuracy different from the first indicator, is obtained when the set value of the first indicator is satisfied. Based on the predicted values ​​and the multiple models, inferences are made regarding lesions in endoscopic images taken of the subject. Image processing methods. [Note 14] The setting value for the first indicator, which shows the accuracy of lesion analysis, is obtained. For each of the multiple models that perform inferences about lesions, the predicted value of the second indicator, which is an indicator of accuracy different from the first indicator, is obtained when the set value of the first indicator is satisfied. A storage medium containing a program that causes a computer to perform a process of inferring information about lesions in endoscopic images taken of a subject, based on the predicted values ​​and the plurality of models.

[0096] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of symbols]

[0097] 1, 1A, 1X Image Processing Device 2 Display device 3 Endoscope 4 Server devices 11 processors 12 memory 13 Interfaces 14 Input section 15 Light source section 16. Sound output section 100, 100A Endoscopy System

Claims

1. A first acquisition means for obtaining a set value of a first indicator that shows the accuracy of lesion analysis, A second acquisition means for obtaining a predicted value of a second indicator, which is an indicator of accuracy different from the first indicator, when the set value of the first indicator is satisfied for each of the multiple models that perform inferences about the lesion, An inference means that performs inferences regarding lesions in endoscopic images taken of a subject, based on the predicted values ​​and the plurality of models, An image processing device having

2. The image processing apparatus according to claim 1, wherein the second acquisition means acquires correspondence information showing the correspondence relationship between the first index and the second index for each of the plurality of models, and acquires the value of the second index corresponding to the set value in the correspondence information as the predicted value.

3. The image processing apparatus according to claim 2, wherein the correspondence information represents an ROC curve, an LROC curve, an FROC curve, or a PR curve.

4. The image processing apparatus according to claim 2, wherein each of the curves of the plurality of models on a two-dimensional coordinate system with the first index and the second index as coordinate axes, as shown by the correspondence information, has an intersection point where they intersect each other.

5. The image processing apparatus according to claim 1, wherein the inference means selects the model with the best accuracy indicated by the predicted value from the plurality of models, and performs the inference based on the selected model and the endoscopic image.

6. The image processing apparatus according to claim 1, wherein the inference means generates an inference result obtained by weighting the inference results of each of the plurality of models based on the endoscopic image based on the predicted value.

7. The aforementioned inference means is A first mode in which the model with the best accuracy indicated by the predicted value is selected from the plurality of models, and the inference is performed based on the selected model and the endoscopic image, The image processing apparatus according to claim 1, wherein the inference means switches between and executes a second mode, which generates an inference result in which the inference results of each of the plurality of models based on the endoscopic image are weighted based on the predicted value, based on predetermined conditions.

8. The image processing apparatus according to claim 7, wherein the inference means switches between the first mode and the second mode and executes based on whether or not a disease other than the target disease is detected in the lesion analysis.

9. The image processing apparatus according to claim 7, wherein the inference means switches between the first mode and the second mode based on an external input.

10. The image processing apparatus according to claim 1, wherein the inference means selects a predetermined number of top models from the plurality of models that have excellent accuracy as indicated by the predicted value, and generates an inference result by weighting the inference result of each of the predetermined number of top models based on the predicted value.

11. The image processing apparatus according to claim 1, wherein the model is a machine learning model that uses a pair of an endoscopic image and ground truth data indicating the inference result that the model should output when the endoscopic image is input as training data.

12. The image processing apparatus according to claim 1, further comprising output control means for outputting the inference result obtained by the inference means via a display device or audio output device to support the examiner's decision-making.

13. Computers We obtain the setting value for the first indicator that shows the accuracy of lesion analysis, For each of the multiple models that perform inferences about lesions, a predicted value of a second indicator, which is an indicator of accuracy different from the first indicator, is obtained when the set value of the first indicator is satisfied. Based on the predicted values ​​and the multiple models, inferences are made regarding lesions in endoscopic images taken of the subject. Image processing methods.

14. We obtain the setting value for the first indicator that shows the accuracy of lesion analysis, For each of the multiple models that perform inferences about lesions, a predicted value of a second indicator, which is an indicator of accuracy different from the first indicator, is obtained when the set value of the first indicator is satisfied. A program that causes a computer to perform a process of inference regarding lesions in endoscopic images taken of a subject, based on the predicted values ​​and the multiple models.

Images