Image processing system, operation method of image processing system, and information storage medium

Abstract

This image processing system (100) includes a memory (120) for storing a learned model (130) and a processor (110). The processor (110) acquires a plurality of estimation results by inputting a plurality of endoscopic images associated with a single case into the learned model (130). Position information associated with the position of a digestive tract is added to each of the endoscopic images. The processor (110) tallies a plurality of estimation results using the obtained plurality of estimation results and the position information, and integrates the plurality of estimation results on the basis of the result of the tally, thereby determining the state of infection by Helicobacter pylori bacterium with respect to the individual cases.

Description

[Technical Field]

[0001] The present invention relates to an image processing system, Method of operating an image processing system and information storage media, etc. [Background technology]

[0002] In recent years, artificial intelligence (hereinafter also referred to as AI) has also been utilized in the medical field. Patent Document 1 discloses a technology in which AI assists in the diagnosis of diseases from endoscopic images of the digestive tract. In this technology, the AI ​​outputs a diagnosis result for a single input image.

[0003] An example of a digestive tract disease is gastric cancer, which is an upper gastrointestinal tract disease. Although gastric cancer is considered curable if detected early, it still has a very high incidence and mortality rate. For this reason, attention is being paid to the importance and accuracy of endoscopic examinations of the upper gastrointestinal tract. Furthermore, recent research has revealed that Helicobacter pylori infection can cause gastric cancer, and it has also become clear that the risk of gastric cancer varies depending on the presence or absence of infection and the degree of atrophy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 225448 Summary of the Invention [Problem to be solved by the invention]

[0005] For diseases that are diagnosed comprehensively based on information about the entire affected area, using the AI ​​technology of Patent Document 1, which determines the disease based only on local information, may not provide a high level of accuracy. For example, when a doctor determines the infection status of Helicobacter pylori, the doctor diagnoses the disease comprehensively based on information about the entire patient's digestive tract. If the AI ​​technology of Patent Document 1 were used to diagnose Helicobacter pylori infection, the diagnosis would be output for each individual image, resulting in a diagnosis based only on information about a localized area in the patient's digestive tract. For this reason, even if the AI ​​technology of Patent Document 1 were used to diagnose Helicobacter pylori infection, there is a high possibility that the accuracy of the diagnosis would be lower than that of a skilled doctor. [Means for solving the problem]

[0006] One aspect of the present disclosure relates to an image processing system that includes a memory that stores a trained model that identifies the Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the infection state for each endoscopic image, and a processor that performs an integration process that obtains multiple inference results by inputting multiple endoscopic images associated with a single case into the trained model and integrating the obtained multiple inference results to determine the Helicobacter pylori infection state for each case, wherein the processor obtains the multiple inference results by inputting the multiple endoscopic images, each of which has site information related to a site of the digestive tract attached to it, into the trained model, aggregates the multiple inference results using the obtained multiple inference results and the site information, and integrates the multiple inference results based on the aggregated results to determine the Helicobacter pylori infection state for each case.

[0007] Another aspect of the present disclosure relates to an image processing method for determining the Helicobacter pylori infection state using a trained model that identifies the Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the infection state for each endoscopic image, the image processing method including: obtaining multiple inference results by inputting multiple endoscopic images, each of which has site information related to a site of the digestive tract attached to it and is linked to a single case, into the trained model; aggregating the multiple inference results using the acquired multiple inference results and the site information; and integrating the multiple inference results based on the aggregation result, thereby performing an integration process for determining the Helicobacter pylori infection state on a case-by-case basis.

[0008] Yet another aspect of the present disclosure relates to a non-transitory information storage medium readable by a computer that stores a program that causes a computer to execute the following steps: determine a Helicobacter pylori infection state using a trained model that identifies the Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the infection state for each endoscopic image; obtain multiple inference results by inputting multiple endoscopic images, each of which is assigned site information regarding a site of the digestive tract and is linked to a single case, into the trained model; aggregate the multiple inference results using the acquired multiple inference results and the site information; and perform an integration process to determine the Helicobacter pylori infection state on a case-by-case basis. [Brief explanation of the drawings]

[0009] [Figure 1] An example of the configuration of an endoscope system. [Figure 2] Examples of the locations of parts of the stomach and their names. [Figure 3] FIG. 2 is an explanatory diagram of processing performed by the image processing system. [Figure 4] FIG. 2 is an explanatory diagram of processing performed by the image processing system. [Figure 5]Example image showing the inferred results of H. pylori infection status. [Figure 6] An example of a clinical perspective. [Figure 7] An endoscopic image called RAC appears on the mucosa that has not been affected by inflammation caused by Helicobacter pylori infection. [Figure 8] An example of an integrated processing flow that emphasizes clinical perspectives. [Figure 9] An example of an integration process flow that emphasizes the accuracy of model judgment. [Figure 10] An illustration of several examples of models and their learning methods. [Figure 11] An example flow of the process of integrating the outputs of the main model and specialized models. [Figure 12] A list of processes that can be incorporated into the integrated process flow. [Figure 13] A list of processes that can be incorporated into the integrated process flow. [Figure 14] A list of processes that can be incorporated into the integrated process flow. [Figure 15] A list of processes that can be incorporated into the integrated process flow. [Figure 16] List of selection methods. [Figure 17] The Kyoto Classification is an example of a clinical perspective. [Figure 18] Specific examples of "related modules". [Figure 19] Specific examples of "related modules". [Figure 20] An example of an integration process flow using modules. [Figure 21] 10 shows a first detailed flow example of step S63. [Figure 22] 10 shows a second detailed flow example of step S63. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0011] 1 shows an example of the configuration of an endoscopic system. The endoscopic system 1 includes an endoscope 200, a video processor 250, a display 290, an image processing system 100, and a display 190. The following describes an example of diagnosing Helicobacter pylori infection using an endoscope for the upper gastrointestinal tract. However, the method of diagnosing disease on a case-by-case basis from multiple images according to the present disclosure is applicable to diagnoses using various endoscopes, not limited to endoscopes for the upper gastrointestinal tract.

[0012] The endoscope 200 is a flexible endoscope that is inserted into the digestive tract and captures images of the inside of the digestive tract. The endoscope 200 includes an insertion section that is inserted into a body cavity, an operation section that is connected to the base end of the insertion section, a universal cord that is connected to the base end of the operation section, and a connector section that is connected to the base end of the universal cord. An imaging device for capturing images of the inside of the body cavity and an illumination optical system for illuminating the inside of the body cavity are provided at the tip of the insertion section. The imaging device includes an objective optical system and an image sensor that captures an image of a subject formed by the objective optical system. The connector section detachably connects a transmission cable to the video processor 250. An image captured by the endoscope 200 will be referred to as an endoscopic image.

[0013] The video processor 250 is a processing device that controls the endoscope and performs image processing and display processing of the endoscopic image. The video processor 250 is composed of a processor such as a CPU, performs image processing on the image signal transmitted from the endoscope 200 to generate an endoscopic image, and outputs the endoscopic image to the display 290 and the image processing system 100. The endoscope system 1 includes a light source device (not shown) that generates and controls illumination light. The light source device may be housed in the same housing as the video processor 250 or in a separate housing. The illumination light emitted from the light source device is guided by a light guide to the illumination optical system of the endoscope 200 and is emitted from the illumination optical system into the body cavity.

[0014] The image processing system 100 determines the infection state of Helicobacter pylori from an endoscopic image and provides diagnostic support by presenting the determination result to a doctor. The image processing system 100 includes a processor 110 and a memory 120.

[0015] The processor 110 includes hardware. The processor 110 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microcomputer, or a digital signal processor (DSP). Alternatively, the processor 110 may be an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor 110 may be composed of one or more of a CPU, a GPU, a microcomputer, a DSP, an ASIC, and an FPGA. The memory 120 is, for example, a semiconductor memory such as a volatile memory or a nonvolatile memory. Alternatively, the memory 120 may be a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive.

[0016] The memory 120 includes a trained model 130 obtained by machine learning. The trained model 130 is, for example, a neural network trained by deep learning. In this case, the trained model 130 includes a program describing the neural network algorithm and weight parameters between the nodes of the neural network. The neural network includes an input layer to which input data is input, an intermediate layer that performs arithmetic processing on the data input through the input layer, and an output layer that outputs an inference result based on the arithmetic result output from the intermediate layer. In the learning stage, a learning system configured by an information processing device or a cloud system performs the machine learning processing. The learning system includes a processor and a memory that stores the model and training data. The processor of the learning system generates the trained model 130 by training the model using the training data. Note that the image processing system 100 may also function as the learning system.

[0017] The memory 120 stores a program 140 in which the details of the process for determining the H. pylori infection status are written. The processor 110 determines the H. pylori infection status by executing the program 140. As will be described in detail later, the processor 110 performs a process of acquiring an inference result for the H. pylori infection status of each image from the trained model 130, and an integration process of determining the H. pylori infection status for each case by integrating the inference results for each image. The program 140 includes program modules in which each process is written, and the processor 110 executes each process by executing the program modules.

[0018] The trained model 130 or the program 140 may be stored in a non-transitory information storage medium that is a computer-readable medium. The information storage medium may be, for example, an optical disk, a memory card, a hard disk drive, or a semiconductor memory. The semiconductor memory may be, for example, a ROM or a non-volatile memory.

[0019] The image processing system 100 may be an information processing device in a separate housing from the video processor 250, or may be incorporated in the same housing as the video processor 250. The image processing system 100 may be configured with multiple information processing devices. That is, multiple processors and multiple memories may be used. For example, the image processing system 100 may use multiple AI models, with each model assigned to an information processing device, and these multiple information processing devices may be connected to each other for communication. Alternatively, the image processing system 100 may be realized by a cloud system in which multiple information processing devices are connected via a network.

[0020] Figure 2 shows examples of the locations of stomach regions and the names of those regions. The names correspond to the locations marked with the same circled numbers. Although the locations of the regions are shown on the outside of the stomach, the endoscopic images are images of each region photographed from the inside of the stomach. The locations of the regions may include not only the numbered locations but also their surroundings, and may extend to a certain extent. Note that Figure 2 is an example of a region, and further regions may be added for Helicobacter pylori infection diagnosis, or some of the regions shown in Figure 2 may be omitted for Helicobacter pylori infection diagnosis.

[0021] 3 and 4 are explanatory diagrams of the processing performed by the image processing system. The trained model 130 includes an image check model 410, a body part recognition model 420, and a Helicobacter pylori infection identification model 430. The program 140 includes a program describing the processing shown in FIGS. 3 and 4 and a program describing integration processing 440.

[0022] As shown in FIG. 3, the processor 110 acquires an image group 12 captured by the endoscope 200. The image group 12 is a plurality of endoscopic images IM1 to IM9 linked to one case. Although FIG. 3 illustrates nine images in the image group 12 linked to one case, as an example, the image group 12 may include approximately 50 images. However, the number of endoscopic images constituting the image group 12 may be any number. Each endoscopic image is a so-called release image, and is an image captured by pressing a release button provided on the endoscope 200. Note that each endoscopic image is not limited to a release image, and may be a frame image of a video.

[0023] The processor 110 inputs the image group 12 into the image check model 410. The image check model 410 excludes endoscopic images that are unsuitable for diagnosing the H. pylori infection state from the image group 12. Images that are unsuitable for diagnosis are images in which the visibility of the gastric mucosa is poor due to blur, mucus, etc. Blur and defocus are caused by the image being out of focus or the subject moving and blurring. The image group 14 includes multiple endoscopic images after exclusion. Figure 3 shows an example in which endoscopic images IM2, IM5, and IM9 have been excluded.

[0024] The image check model 410 is a trained model based on machine learning, and is trained to determine whether an input endoscopic image is suitable for diagnosing the H. pylori infection state. The image check model 410 may be a program using a rule-based algorithm, or the image check may be performed manually by a person.

[0025] The processor 110 inputs the image group 12 to the part recognition model 420. The part recognition model 420 assigns information about the part shown in each endoscopic image included in the image group 12 to that endoscopic image. The part information is the name of the part, a tag indicating the part, or the like. Each endoscopic image included in the image group 14 is assigned part information. FIG. 3 shows an example in which endoscopic images IM1 and IM3 are assigned "part A," endoscopic images IM4 and IM6 are assigned "part B," and endoscopic images IM7 and IM8 are assigned "part C." Each of parts A to C is one of the seven parts shown in FIG. 2. Although FIG. 3 only shows three parts, endoscopic images are taken of each of the seven parts shown in FIG. 2.

[0026] The part recognition model 420 is a trained model based on machine learning, and is trained to output information about the part that appears in the input endoscopic image. The part recognition model 420 may be a program that uses a rule-based algorithm, or part recognition may be performed manually by a person.

[0027] As shown in FIG. 4, the processor 110 inputs the image group 14 to the H. pylori infection identification model 430. The H. pylori infection identification model 430 identifies the H. pylori infection state from each endoscopic image included in the image group 14 and outputs the result. The identification result is classified into three categories, for example, uninfected, previously infected, and currently infected. Uninfected means a state in which the patient has never been infected. Previously infected means a state in which the patient was infected in the past but was eradicated and is not currently infected. Currently infected means a state in which the patient is currently infected. Alternatively, the identification result may be whether or not the patient is uninfected, previously infected, or currently infected. Alternatively, the identification result may be whether or not there are findings specific to H. pylori infection.

[0028] The classification result group 16 includes the site information assigned to each endoscopic image by the site recognition model 420 and the classification result of H. pylori infection for each endoscopic image. The classification result group 16 need only be a group of information in which the ID, site information, and classification result of the endoscopic image are associated, and does not need to include the endoscopic image itself. In Figure 4, for example, site A and result a are associated with the ID of endoscopic image IM1.

[0029] The processor 110 executes an integration process 440 that integrates the inference results of the multiple images included in the identification result group 16, and outputs the Helicobacter pylori infection status for each case 450. The processor 110 weights the inference results of the multiple images based on the sites or findings that are important for identifying the infection status, and then integrates the inference results. The processor 110 displays the inference results on the display 190. The inference results may also be displayed on the display 290. In that case, the display 190 may be omitted.

[0030] Inflammation caused by H. pylori infection, for example, begins in the lower stomach and gradually spreads to other areas. Therefore, findings from endoscopic images vary depending on the region. Even if AI identifies the H. pylori infection status from a single endoscopic image, it is difficult to determine the H. pylori infection status for the entire case. In this regard, according to this embodiment, the H. pylori infection status is inferred from each of multiple endoscopic images associated with a single case, and the multiple inference results are integrated to determine the H. pylori infection status for each case. This enables highly accurate estimation of the H. pylori infection status by comprehensively considering the overall information within the patient's gastrointestinal tract. Furthermore, since physicians no longer need to check multiple images one by one, the burden of viewing images on physicians can be reduced. Furthermore, the inferred H. pylori infection status can be used for training physicians and other professionals.

[0031] 5 is an example of an image displaying the inference result of the H. pylori infection state. The image 30 includes an area 31 displaying an image of the integrated processing flow and an area 32 displaying the inference result.

[0032] A flow diagram such as that described later with reference to Fig. 11 is displayed in area 31. By looking at the flow diagram, a doctor can understand what algorithm was used to obtain the inference result.

[0033] Area 32 displays inference results 33 and an image list 34. An example of the inference result 33 is "The patient was determined to be uninfected due to factor XXX when integrating the inference results." XXX may be, for example, the findings that were emphasized in the integration process, or the step in the flow at which the determination was made. The image list 34 is a list of endoscopic images from the image group 14 input to the H. pylori infection identification model 430 that were determined to have the same infection state as the uninfected state, which is the inference result for each case. In FIG. 5, the image list 34 is a list of endoscopic images that were determined to be uninfected on an image-by-image basis.

[0034] According to this embodiment, a doctor can reconfirm the diagnosis result or expect an educational effect by comparing the diagnosis result he or she judged with the image 30. For example, by looking at the image 30, the doctor can think, "The infection state I judged and the infection state inferred by the system were different. I would like to use this experience in my next diagnosis."

[0035] There are two methods for integrating the inference results for each endoscopic image: one that emphasizes a clinical perspective, and one that emphasizes the accuracy of the model's judgment. First, we will explain the integration method that emphasizes a clinical perspective.

[0036] Figure 6 shows an example of a clinical perspective. Here, we show an example of combining the Kyoto Classification, a doctor's opinion, and book information, but the clinical perspective reflected in the integration method is not limited to that shown in Figure 6.

[0037] The "No." column in Figure 6 indicates the order in which a doctor looks at the areas when diagnosing H. pylori infection. The "Importance of H. pylori infection status" column indicates which areas the doctor places importance on when determining the infection status. For example, a doctor will place importance on the findings in the EG junction, vestibule, and lesser curvature of the gastric angle to determine whether or not the patient is uninfected. The "Determination of H. pylori infection status based on findings" column indicates how a doctor can determine the infection status based on the findings of the areas.

[0038] Thus, for each infection state—uninfected, previously infected, and currently infected—there are mucosal conditions or regions that are important for clinical identification. Take the lesser curvature of the stomach shown in No. 5 as an example. Figure 7 shows an endoscopic image of what is called RAC, which appears in mucosa that has not been affected by inflammation caused by H. pylori infection. As shown within the dotted line 50, the RAC has a granular texture in the mucosa. Doctors use the presence or absence of RAC in the lesser curvature of the stomach as one of the criteria for determining whether a patient is uninfected.

[0039] 8 shows an example of the flow of integration processing that emphasizes a clinical perspective. The processor 110 inputs the identification result group 16 output by the H. pylori infection identification model 430 to the integration processing 440.

[0040] In step S1, the processor 110 determines whether or not a site important for determining whether or not an infection is present in the identification result group 16. Referring to Fig. 6, for example, the processor 110 determines whether or not an endoscopic image of the EG junction, the vestibule, or the lesser curvature of the gastric angle is present.

[0041] If the result in step S1 is true, in step S2, processor 110 determines that the case is uninfected. If the result in step S1 is false, in step S3, processor 110 performs a majority vote using the results of the parts in identification result group 16 that are important for determining current infection, and determines whether there are many current infections. Referring to Figure 6, for example, processor 110 determines whether the number of endoscopic images determined to be current infections is more than half of the number of endoscopic images of the upper body, middle body, and upper body taken from a view down of the greater curvature of the body.

[0042] If step S3 is true, then in step S4, processor 110 determines that the case is currently infected. If step S3 is false, then in step S5, processor 110 determines that the case is previously infected.

[0043] 9 shows an example of the flow of integration processing that places importance on the judgment accuracy of the model. The processor 110 inputs the identification result group 16 output by the H. pylori infection identification model 430 to the integration processing 440.

[0044] In step S11, the processor 110 selects an identification result for a predetermined region from the identification result group 16. The number of selected regions may be one or more. FIG. 9 shows an example in which identification results for regions A and B from regions A to C are retained. The predetermined region is a region for which the model has high determination accuracy, and is determined by evaluating the model in advance. For example, the determination accuracy when an endoscopic image of regions A and B is input into model X is compared with the determination accuracy when an endoscopic image of region C is input into model X, and regions A and B are selected if the determination accuracy of regions A and B is high. Alternatively, the determination accuracy when an endoscopic image of regions A and B is input into model X is compared with the determination accuracy when an endoscopic image of region C is input into model Y, which is different from model X, and if the determination accuracy of model X is high, model X may be adopted and regions A and B may be selected.

[0045] In step S12, the processor 110 determines whether the number of uninfected sheets in the selected identification result group is equal to or greater than a predetermined threshold (here, 60%). Note that the predetermined threshold is not limited to 60%, and may be determined by the user as appropriate, taking into consideration the characteristics of the H. pylori infection identification model 430, etc.

[0046] If the result in step S12 is true, in step S13, processor 110 determines that the case is uninfected. If the result in step S12 is false, in step S14, processor 110 performs a majority vote on the selected group of identification results to determine whether there are many current infections.

[0047] If step S14 is true, then in step S15 processor 110 determines that the case is currently infected. If step S14 is false, then in step S16 processor 110 determines that the case is previously infected.

[0048] Note that the selection of the region may be performed for each of steps S12 and S14. That is, the first selection may be performed before step S12, and the second selection may be performed before step S14, and the region selected in the second selection may be different from the region selected in the first selection.

[0049] According to this embodiment, by integrating multiple inference results based on a clinical perspective or model judgment accuracy, it is possible to automate comprehensive judgments similar to those made by doctors. By using a clinical perspective or model judgment accuracy, it is possible to accurately determine the H. pylori infection state. Furthermore, by using an integration method that emphasizes model judgment accuracy, the integration process is based on an evaluation of the actual judgment accuracy, and further improvement in accuracy can be expected.

[0050] The integration process flow is not limited to that shown in Figures 8 and 9. For example, an integration process flow may be configured by combining an integration method that emphasizes a clinical perspective with an integration method that emphasizes model judgment accuracy. Alternatively, as will be described later with reference to Figures 12 to 15, various processes or judgments may be prepared and combined to configure an integration process flow.

[0051] 4 shows an example in which the output results of the Helicobacter pylori infection identification model 430 are integrated, but the outputs of multiple models may be integrated. FIG. 10 is an explanatory diagram of an example of multiple models and their learning methods.

[0052] The multiple models include a main model that identifies the infection state of various regions and specialized models that identify the infection state of a specific region or specific findings related to the infection state. The main model is a model that can identify the H. pylori infection state without specializing in the infection state, region, or findings. The main model is the H. pylori infection identification model 430, which classifies each endoscopic image into three categories: current infection, previous infection, and non-infection. The specialized models are models that make judgments specialized in a specific infection state, specific region, or specific findings. Examples of specialized models include an antrum specialized model 520, a RAC specialized model 530, and a current infection specialized model 540. The antrum specialized model 520 determines the presence or absence of atrophy from an endoscopic image of the vestibular region. The RAC specialized model 530 determines the presence or absence of RAC from an endoscopic image of the lesser curvature of the stomach. The current infection specific model 540 determines the presence or absence of diffuse redness or fold swelling from endoscopic images of the middle and upper parts of the greater curvature.

[0053] In the learning process, the learning system generates a self-supervised model 550 through self-supervised learning. In self-supervised learning, endoscopic images of various parts of the stomach are input to the model, and the representation of the endoscopic images is learned. Next, the learning system generates a main model and specialized models by fine-tuning using the self-supervised model 550. An example of fine-tuning learning is so-called knowledge distillation. In knowledge distillation, the self-supervised model 550 is the teacher model, and the main model and specialized model are student models. Specifically, the teacher data includes endoscopic images and correct labels attached to the endoscopic images. The learning system inputs the endoscopic images of the teacher data into the self-supervised model 550, and the output is used as the soft target, and the correct labels of the teacher data are used as the hard target. The learning system inputs the endoscopic images of the teacher data into the student model and trains the student model based on an evaluation function calculated from the output, the soft target, and the hard target.

[0054] The learning methods for the main model and specialized models are not limited to those described above, and each model may be generated by so-called supervised learning. The current infection specialized model 540 may determine atrophy and diffuse redness of the lower part of the greater curvature. The main model targets images observed under illumination with white light (WLI), for example. However, the main model may also target images observed under illumination other than WLI, or images of stained tissue.

[0055] 11 shows an example of a process flow for integrating the outputs of the main model and the specialized model. In step S21, the processor 110 inputs multiple endoscopic images to the main model, and the main model outputs an inference result for each endoscopic image. Here, all parts may be used, or only some parts with a high accuracy of determining whether they are uninfected may be used.

[0056] In step S22, processor 110 determines whether the number of images determined to be uninfected by the main model is equal to or greater than a threshold value. Processor 110 may also determine whether the ratio of the number of images determined to be uninfected by the main model to the number of images input to the main model is equal to or greater than a threshold value.

[0057] If the number is greater than or equal to the threshold in step S22, then in step S23, processor 110 determines whether the difference between the number of images determined to be infected by the main model and the number of images determined to be uninfected by the main model is less than or equal to the threshold.

[0058] If the difference is greater than the threshold in step S23, the processor 110 determines the Helicobacter pylori infection state of the case to be uninfected in step S24. If the difference is equal to or less than the threshold in step S23, the processor 110 selects, in step S25, endoscopic images of the regions to be judged by the specialized model 1. The specialized model 1 is the vestibule specialized model 520 and the RAC specialized model 530. That is, the regions selected are the vestibule and the lesser curvature of the stomach.

[0059] In step S26, processor 110 inputs endoscopic images of the selected region into specialized model 1, which outputs inference results for each endoscopic image. In steps S27 and S28, processor 110 determines the Helicobacter pylori infection status of the case as uninfected or infected based on the output of specialized model 1. Specifically, processor 110 inputs endoscopic images of the anterior region into antrum specialized model 520, which outputs the presence or absence of atrophy for each endoscopic image. Processor 110 inputs endoscopic images of the lesser curvature of the stomach into RAC specialized model 530, which outputs the presence or absence of RAC for each endoscopic image. Processor 110 determines whether the case is uninfected or infected based on these identification results.

[0060] If the number is less than the threshold in step S22, in step S29, processor 110 determines whether the difference between the number of images determined to be already infected by the main model and the number of images determined to be currently infected by the main model is greater than or equal to the threshold.

[0061] If the difference is less than the threshold in step S29, then in step S30, processor 110 determines whether the difference between the number of images determined to be infected by the main model and the number of images determined to be uninfected by the main model is less than or equal to the threshold.

[0062] If the difference is greater than the threshold in step S30, then in step S31, processor 110 determines the Helicobacter pylori infection state of the case to be pre-infected. If the difference is equal to or less than the threshold in step S30, then in step S32, processor 110 selects an endoscopic image of the region to be judged by specialized model 1. Steps S32 to S35 are the same as steps S25 to S28.

[0063] If the difference is equal to or greater than the threshold in step S29, in step S36, processor 110 selects an endoscopic image of the region to be judged by specialized model 2. Specialized model 2 is current infection specialized model 540. That is, the regions selected are the middle greater curvature and the upper greater curvature.

[0064] In step S37, processor 110 inputs endoscopic images of the selected region into specialized model 2, which outputs inference results for each endoscopic image. In steps S38 and S39, processor 110 determines the H. pylori infection status of the case as either pre-existing or current infection based on the output of specialized model 2. Specifically, processor 110 inputs endoscopic images of the middle and upper parts of the greater curvature into current infection specialized model 540, which outputs the presence or absence of diffuse redness or fold swelling for each endoscopic image. Processor 110 determines whether the case is pre-existing or current infection based on the identification results.

[0065] The integrated processing flow described in Figures 8, 9, and 11 is an example, and an integrated processing flow may be configured by combining various processes as described below. Figures 12 to 15 are a list of processes that can be incorporated into an integrated processing flow. An integrated processing flow is configured by combining processes 1 to 11 shown in Figures 12 to 15. It is not necessary to use all of processes 1 to 11. Furthermore, the same process may be used multiple times within a flow.

[0066] 12 to 15 show the details of each process, including "related modules," "input to the process," "processing content," and "output from the process." "Related modules" mean that the output of the modules is input to the integrated process. The modules related to processes 1 to 10 are trained models using machine learning, and the module related to process 11 is a program using a rule-based algorithm. "Input to the process" is the content of the data input to the integrated process, that is, the output of the "related modules." "Processing content" means what kind of process is performed using the "input to the process." "Output from the process" means what kind of output is obtained as a result of the process.

[0067] Taking process 2 as an example, processor 110 inputs the output of the main model to process 2. The input is the classification results of uninfected, previously infected, and currently infected for each endoscopic image of the region selected based on accuracy. Processor 110 determines whether the proportion of images determined to be in specific infection state A among the total number of images in the input classification results is equal to or greater than threshold th a. Here, the total number of images in the input classification results is the number of endoscopic images of the region selected based on accuracy. As an example, assume that specific infection state A is uninfected and threshold th a is 60%. In this case, processor 110 determines whether the proportion of images determined to be uninfected among the total number of images in the input classification results is equal to or greater than 60%. If the proportion is equal to or greater than threshold th a, processor 110 determines that the case is in specific infection state A. Processor 110 may then determine that the case is in specific infection state A, or may proceed to the next process based on the determination result. In the latter case, for example, processor 110 may determine the infection state using the result of determining that the case is in specific infection state A in process 2 and the determination result obtained in subsequent processes. There are various processes when the ratio is smaller than threshold th a. Processor 110 may determine that the case is not in specific infection state A, but may proceed to the next process as there are two possibilities for the case to be in a category other than specific infection state A. Alternatively, processor 110 may determine the infection state using the result of determining that the case is in specific infection state A in process 2 and the determination result obtained in subsequent processes, while leaving open the possibilities for any of previously infected, uninfected, and currently infected.

[0068] Each of the specific infection states A to D shown in Figures 12 to 15 is either uninfected, previously infected, or currently infected. The thresholds tha to thd may be different values ​​from one another, or any two or more of them may be the same value. The order of execution of processes 1 to 11 may be arbitrary, but as an example, process 5 or 6 is executed after process 1, 2, 3, or 4. Or process 7 or 8 is executed after process 1, 2, 3, or 4. Or process 8 or 10 is executed after process 7 or 8.

[0069] In Figures 12 to 15, "Input to processing" includes an item "Internal stomach region." This item refers to the region to be selected. Specifically, the "Internal stomach region" item refers to which region's endoscopic image is to be input to the "related module," or which region's identification result is to be used from among the identification results output by the "related module." Figure 16 is a list of selection methods. Method 2, which selects a region based on the accuracy of the model, was also described above in Figure 9. Method 3, which selects a region based on a clinical perspective, was also described above in Figures 6 to 9.

[0070] Figure 17 shows the Kyoto Classification, an example of a clinical perspective used in Method 3. "Localization" corresponds to a location inside the stomach. "Infected" corresponds to current infection, and "Post-eradication" corresponds to existing infection. The circle, cross, or triangle shown for "infected," "uninfected," and "post-eradication" indicates whether "endoscopic findings" are likely to be observed in the location indicated by "localization" in that infection state. For example, diffuse redness throughout the entire gastric mucosa is often observed in "infected," but not in "uninfected" or "post-eradication."

[0071] As a clinical perspective, not only the Kyoto Classification in Fig. 17 but also a doctor's opinion, book information, etc. may be used. For example, as described in Fig. 6, a clinical perspective that combines the Kyoto Classification, a doctor's opinion, and book information may be used.

[0072] Figures 18 and 19 are specific examples of the "related modules" in Figures 12 to 15. "Characteristics" and "Useful for" indicate the purpose or nature of the module. "Learning" indicates the "teaching data" used in the learning stage of the module. "Inference" indicates the "input" to the module and the "output" from the module in the inference stage.

[0073] The main model is used as the main model for processes 1, 2, 5, 6, or 7 in Figures 12 to 15. The main model in Figure 18 can distinguish between current infection, previous infection, and non-infection using endoscopic images of all regions as input. In this case, processes 2, 4, 6, and 7 in Figures 12 to 15 can be realized by inputting endoscopic images of some regions into the main model, or by inputting the discrimination results of some regions from the output of the main model into the integration process. The current infection-specialized model, vestibule-specialized model, and RAC-specialized model are used as process 4 or 8 in Figures 12 to 15. The atrophy model is used as process 10 in Figures 12 to 15.

[0074] "Teacher data" refers to the input to a model in the learning stage and the correct labels attached to the input. This "teacher data" is used as teacher data in supervised learning, for example. Alternatively, in the fine-tuning of knowledge distillation described above, the output of the teacher model is the soft target, and the correct labels of the "teacher data" are the hard target.

[0075] Fig. 20 is an example of a flow of integration processing using the modules of Fig. 18 and Fig. 19. The parts indicated by circled numbers in the flow correspond to the parts indicated by circled numbers in Fig. 2.

[0076] In step S51, the processor 110 inputs endoscopic images of the lower body, the greater curvature of the stomach, looking down at the greater curvature of the body (middle body), and looking down at the greater curvature of the body (upper body) into the current infection specialization model, and the current infection specialization model outputs an inference result as to whether or not each endoscopic image indicates a current infection.

[0077] In step S61, the processor 110 determines whether the ratio of the number of images determined to be current infections to the total number of images in the inference results output by the current infection specific model is equal to or greater than a threshold value th1.

[0078] If the ratio is equal to or greater than the threshold value th1 in step S61, the processor 110 determines the H. pylori infection state of the case to be current infection in step S81.

[0079] If the ratio is smaller than the threshold value th1 in step S61, the processor 110 acquires past medical interview information in step S52. For example, the memory 120 stores past medical interview information related to the case, and the processor 110 reads the past medical interview information from the memory 120. Alternatively, the processor 110 may acquire the past medical interview information related to the case from an external database of the endoscopy system 1.

[0080] In step S62, the processor 110 determines whether or not there is a record of "sterilization success" in the past medical interview information.

[0081] If "eradication successful" is recorded in step S62, the processor 110 determines the H. pylori infection state of the case to be already infected in step S82.

[0082] If there is no record of "eradication success" in step S62, in step S63, processor 110 determines whether the case's Helicobacter pylori infection status is already infected or not, and processor 110 determines that the case is already infected in step S83 or not infected in step S84.

[0083] 21 is a first detailed flow example of step S63. In step S53, processor 110 inputs endoscopic images of the gastric angle greater curvature, vestibule, gastric angle lesser curvature, and corpus lesser curvature viewed from the lower body into the main model, and the main model classifies each endoscopic image as currently infected, previously infected, or not infected.

[0084] In step S71, processor 110 determines whether the ratio of the number of images determined to be uninfected to the total number of images in the inference results output by the main model is equal to or greater than threshold th2.

[0085] If the ratio is smaller than the threshold value th2 in step S71, the processor 110 determines the H. pylori infection state of the case to be already infected in step S83a.

[0086] If the ratio is equal to or greater than the threshold value th2 in step S71, the processor 110 inputs the endoscopic images of the lesser curvature of the gastric angle to the atrophy model in step S54, and the atrophy model determines the presence or absence of atrophy from each endoscopic image.

[0087] In step S72, processor 110 determines whether the ratio of the number of images determined to have atrophy to the total number of images in the inference result output by the atrophy model is equal to or greater than threshold th3.

[0088] If the ratio is equal to or greater than the threshold value th3 in step S72, the processor 110 determines the H. pylori infection state of the case to be already infected in step S83b.

[0089] If the ratio is smaller than the threshold value th3 in step S73, the processor 110 determines the H. pylori infection state of the case to be uninfected in step S84.

[0090] 22 is a second detailed flow example of step S63. In step S53, processor 110 inputs endoscopic images of the gastric angle greater curvature, vestibule, gastric angle lesser curvature, and corpus lesser curvature viewed from the lower body into the main model, and the main model classifies each endoscopic image as currently infected, previously infected, or not infected.

[0091] In step S73, processor 110 determines whether the ratio of the number of images determined to be uninfected to the total number of images in the inference results output by the main model is equal to or greater than threshold th4.

[0092] If the ratio is smaller than the threshold value th4 in step S73, the processor 110 determines the H. pylori infection state of the case to be already infected in step S83c.

[0093] If the ratio is equal to or greater than the threshold value th4 in step S73, the processor 110 inputs the endoscopic images of the lesser curvature of the stomach into the RAC specialization model in step S55, and the RAC specialization model determines the presence or absence of RAC from each endoscopic image.

[0094] In step S74, the processor 110 determines whether the ratio of the number of images determined to have atrophy to the total number of images in the inference results output by the RAC specialization model is equal to or greater than a threshold value th5.

[0095] If the ratio is equal to or greater than the threshold value th5 in step S74, the processor 110 determines the H. pylori infection state of the case to be already infected in step S83d.

[0096] If the ratio is smaller than the threshold value th4 in step S74, in step S56, the processor 110 inputs the endoscopic images of the vestibule into the vestibule specialized model, and the vestibule specialized model determines the presence or absence of atrophy in the vestibule from each endoscopic image.

[0097] In step S75, processor 110 determines whether the ratio of the number of images determined to have atrophy, i.e., to the total number of images in the inference results output by the vestibular specialized model, is equal to or greater than threshold th6.

[0098] If the ratio is equal to or greater than the threshold value th6 in step S75, the processor 110 determines the H. pylori infection state of the case to be already infected in step S83e.

[0099] If the ratio is smaller than the threshold value th6 in step S75, the processor 110 determines the H. pylori infection state of the case to be uninfected in step S84.

[0100] The image processing system 100 of the present embodiment described above includes a memory 120 and a processor 110. The memory 120 stores a trained model 130 that identifies the H. pylori infection state for each input endoscopic image and outputs an inference result indicating the infection state for each endoscopic image. The processor 110 performs an integration process in which multiple endoscopic images associated with a single case are input to the trained model 130 to obtain multiple inference results, and the acquired multiple inference results are integrated to determine the H. pylori infection state for each case. The processor 110 inputs multiple endoscopic images, each of which has site information related to a site in the digestive tract attached to it, to the trained model 130 to obtain multiple inference results. The processor 110 aggregates the multiple inference results using the acquired multiple inference results and the site information, and integrates the multiple inference results based on the aggregation result to determine the H. pylori infection state for each case.

[0101] According to this embodiment, the trained model 130 infers the H. pylori infection state for each endoscopic image from multiple endoscopic images associated with a single case, thereby obtaining multiple inference results. The processor 110 then integrates the multiple inference results to determine the H. pylori infection state of the case. Although it is expected that the inference results for the H. pylori infection state will differ for each endoscopic image, integrating these results makes it possible to comprehensively determine the H. pylori infection state for the entire case, rather than for each image. Furthermore, compared to determining the H. pylori infection state for each image, this enables a judgment closer to the comprehensive judgment made by a doctor, thereby enabling the H. pylori infection state for the entire case to be determined with high accuracy.

[0102] The trained model 130 may be a main model, a specialized model, or may include both a main model and a specialized model. The "inference result indicating the infection status for each endoscopic image" may be either the inference result of the main model or the inference result of the specialized model. That is, the "inference result indicating the infection status for each endoscopic image" may be one of three categories: pre-infected, uninfected, and current infection; pre-infected or uninfected; current infection or not; or the presence or absence of specific findings. "Counting multiple inference results" refers to calculating the number of images based on the inference results. For example, the counting may include summing the number of images determined to be uninfected as the inference result, summing the number of images determined to have specific findings as the inference result, summing the number of images used in inference regarding a certain region, etc.

[0103] In this embodiment, the integration process may include at least one of a first process and a second process. The first process may be a process of determining that the Helicobacter pylori infection state of the specific site is a first infection state when the ratio of the number of second images to the number of first images is equal to or greater than a specific threshold. The second process may be a process of determining that the Helicobacter pylori infection state of the specific site is one of current infection, previous infection, and uninfection other than the first infection state when the ratio of the number of second images to the number of first images is equal to or less than a specific threshold. The first number of images may be the number of endoscopic images to which site information of the specific site is attached among the multiple endoscopic images. The second number of images may be the number of endoscopic images determined to be in the first infection state by the trained model 130. The first infection state may be one of current infection, previous infection, and uninfection.

[0104] According to this embodiment, by at least one of the first process and the second process, it is possible to determine whether the H. pylori infection state of a specific site is the first infection state. That is, it is possible to determine whether the H. pylori infection state of a specific site is uninfected, previously infected, or currently infected. Then, by incorporating such a determination into the integration process, it becomes possible to comprehensively determine the H. pylori infection state for each case.

[0105] The "integration process" here corresponds to processes 2, 4, 6, 7, 8, or 10 in Figures 12 to 15. Taking process 2 as an example, the "specific region" is the region selected based on accuracy, and the "first number of images" is the number of endoscopic images that capture the region selected based on accuracy among multiple endoscopic images associated with one case. Furthermore, the "first infection state" is specific infection state A, and the "second number of images" is the number of endoscopic images that are determined to be in specific infection state A among endoscopic images that capture the region selected based on accuracy. Furthermore, the "specific threshold" is a threshold value tha.

[0106] In this embodiment, the specific site may be a site in the digestive tract where an important finding for the first infection state occurs.

[0107] According to this embodiment, by selecting a region where important findings for the first infection state appear as a specific region, the inference results for the endoscopic image of the specific region can be used as information for determining whether or not the case is in the first infection state. By incorporating such a determination into the integration process, it becomes possible to accurately integrate and determine the Helicobacter pylori infection state for each case.

[0108] The "specific site" here corresponds to the "clinically selected site" in Figure 14. That is, the "integration process" here corresponds to process 7 or 8 in Figure 14. A specific finding is a finding that is specific (characteristic) to the state of H. pylori infection that appears in a specific site from a clinical perspective. For example, in the example in Figure 6, atrophy of the antrum or RAC of the lesser curvature of the gastric angle are specific findings.

[0109] In this embodiment, the specific site may include a site in the digestive tract where the trained model 130 identifies the first infection state with the highest accuracy.

[0110] According to this embodiment, by selecting a specific site at which the trained model 130 can identify the first infection state with the highest accuracy, the inference results for the endoscopic image of the specific site can be used to determine whether the case is in the first infection state. By incorporating such a determination into the integration process, it becomes possible to accurately integrate and determine the Helicobacter pylori infection state for each case.

[0111] The "specific portion" here corresponds to the "portion selected based on accuracy" in Figures 12, 13, and 15. That is, the "integration process" here corresponds to process 2, 4, 6, or 10 in Figure 14.

[0112] In addition, in this embodiment, the trained model 130 may classify the input endoscopic image as being currently infected, previously infected, or not infected, and output the classification result as an inference result.

[0113] According to this embodiment, the H. pylori infection state of a case can be determined by integrating the classification results of current infection, previous infection, and non-infection inferred for each endoscopic image. Although it is assumed that the classification results of current infection, previous infection, and non-infection will differ for each endoscopic image, integrating these results makes it possible to comprehensively determine the H. pylori infection state of the case as a whole, rather than on an image-by-image basis.

[0114] Note that the "trained model" here corresponds to the main model in Figures 12 to 14. That is, the "integration process" here corresponds to process 1, 2, 5, 6, or 7 in Figures 12 to 14. However, the trained model 130 may further include models other than the main model.

[0115] In addition, in this embodiment, the trained model 130 determines whether the input endoscopic image corresponds to a specific infection state among current infection, previous infection, and non-infection, and outputs the determination result as an inference result.

[0116] According to this embodiment, the H. pylori infection state of a case can be determined by integrating the results of the determination of whether each endoscopic image corresponds to a specific infection state inferred for that image. Although it is expected that the results of the determination of whether each endoscopic image corresponds to a specific infection state will differ, integrating these results makes it possible to comprehensively determine the H. pylori infection state of the case as a whole, rather than on an image-by-image basis.

[0117] Note that the "trained model" here corresponds to the specialized model in Figures 12 and 14. That is, the "integration process" here corresponds to process 3, 4, or 8 in Figures 12 and 14. However, the trained model 130 may further include models other than the specialized model. "Whether or not it corresponds to a particular infection state" refers to whether or not it is uninfected, whether or not it is currently infected, or whether or not it is previously infected.

[0118] In addition, in this embodiment, the trained model 130 may determine whether or not atrophy is present in the input endoscopic image, and output the determination result as an inference result.

[0119] According to this embodiment, the H. pylori infection state of a case can be determined by integrating the determination results of the presence or absence of atrophy inferred for each endoscopic image. Although it is assumed that the determination results of the presence or absence of atrophy will differ for each endoscopic image, integrating them makes it possible to comprehensively determine the H. pylori infection state of the case as a whole, rather than for each image.

[0120] Note that "atrophy" refers to a state in which the gastric mucosa becomes thin due to chronic inflammation. "Atrophy" may be accompanied by inflammation, or may be atrophy that remains after inflammation has disappeared. "Atrophy" may include, for example, atrophic gastritis or intestinal metaplasia. The "trained model" here corresponds to the atrophy model in FIG. 15. That is, the "integration process" here corresponds to process 9 or 10 in FIG. 15. However, the trained model 130 may further include models other than the atrophy model.

[0121] In addition, in this embodiment, the trained model 130 may determine the presence or absence of specific findings linked to a specific infection state, among current infection, previous infection, and non-infection, from the input endoscopic image, and output the determination result as an inference result.

[0122] According to this embodiment, the H. pylori infection state of a case can be determined by integrating the judgment results of the presence or absence of specific findings inferred for each endoscopic image. Although it is expected that the judgment results of the presence or absence of specific findings will differ for each endoscopic image, integrating them makes it possible to comprehensively judge the H. pylori infection state of the case as a whole, rather than for each image.

[0123] The "trained model" here corresponds to the current infection-specific model, the antrum-specific model, the RAC-specific model, or the atrophy model in Figures 18 and 19. That is, the "integration process" here is based on the aggregated results of the outputs of the above models. However, the trained model 130 may further include models other than the above models.

[0124] Furthermore, in this embodiment, the trained model 130 may include a first trained model that outputs an inference result from an endoscopic image showing a region belonging to a first region group, and a second trained model that outputs an inference result from an endoscopic image showing a region belonging to a second region group. The processor 110 may acquire a first plurality of inference results by inputting a plurality of endoscopic images showing a region belonging to the first region group into the first trained model. The processor 110 may acquire a second plurality of inference results by inputting a plurality of endoscopic images showing a region belonging to the second region group into the second trained model. The processor 110 may determine the Helicobacter pylori infection state of each case based on the acquired first plurality of inference results and second plurality of inference results.

[0125] According to this embodiment, the H. pylori infection state of each case can be determined by combining various trained models and integrating the inference results output by each trained model. For example, it is possible to prepare models according to the site or findings and combine these models, which makes it possible to accurately determine the H. pylori infection state of each case.

[0126] Each of the first site group and the second site group is all or part of the stomach. The second site group may be the same as or different from the first site group. The sites included in the second site group may overlap with the sites included in the first site group. Each of the first trained model and the second trained model is the main model or specialized model of Figures 12 to 15. Alternatively, each of the "first trained model" and the "second trained model" is one of the main model, current infection specialized model, antrum specialized model, RAC specialized model, and atrophy model shown in Figures 18 and 19. The "second trained model" may be a model different from or the same as the "first trained model." When the "second trained model" is the same model as the "first trained model," the second site group may be different from the first site group.

[0127] In addition, in this embodiment, the processor 110 may determine the Helicobacter pylori infection state for each case based on the aggregation result of the first plurality of inference results and the aggregation result of the second plurality of inference results.

[0128] According to this embodiment, the Helicobacter pylori infection status can be determined on a case-by-case basis by integrating the Helicobacter pylori infection status determined based on the result of aggregating the first plurality of inference results and the Helicobacter pylori infection status determined based on the result of aggregating the second plurality of inference results.

[0129] 20 and 21, in step S61, a determination is made as to whether or not there is a current infection based on the inference results of the current infection-specific model, and in step S71, a determination is made as to whether or not there is an uninfected state based on the inference results of the main model, and these results are combined to determine whether the state is uninfected, previously infected, or currently infected. In this example, one of the current infection-specific model and the main model corresponds to the first trained model, and the other corresponds to the second trained model.

[0130] In addition, in this embodiment, the first trained model may classify the input endoscopic image as being currently infected, previously infected, or not infected, and output the classification result as the inference result.

[0131] In addition, in this embodiment, the second trained model may determine whether the input endoscopic image corresponds to a specific infection state among current infection, previous infection, and non-infection, and output the determination result as an inference result.

[0132] In addition, in this embodiment, the second trained model may determine whether or not atrophy is present in the input endoscopic image, and output the determination result as the inference result.

[0133] In addition, in this embodiment, the second trained model may determine the presence or absence of specific findings linked to a specific infection state, among current infection, previous infection, and non-infection, from the input endoscopic image, and output the determination result as an inference result.

[0134] The effects of using these trained models, the meanings of terms, and correspondence with the examples are as described above.

[0135] In this embodiment, as described in Figure 5, the processor 110 performs processing to display on the display 190 the judgment result of the Helicobacter pylori infection state obtained by the integration processing and a list of endoscopic images among multiple endoscopic images that have the same infection state as the judgment result.

[0136] According to this embodiment, the H. pylori infection status of a case automatically determined by the trained model and program and a list of endoscopic images with the same infection status as the determined result can be presented to the doctor. By comparing the presented content with the diagnosis result determined by the doctor, the doctor can reconfirm the diagnosis result or expect an educational effect.

[0137] The above-described embodiment may be implemented as an image processing method. That is, the image processing method determines the Helicobacter pylori infection state using the trained model 130. The trained model 130 identifies the Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the infection state for each endoscopic image. The image processing method includes inputting multiple endoscopic images to the trained model 130 to obtain multiple inference results. Each of the multiple endoscopic images is assigned site information related to a site in the digestive tract, and the multiple endoscopic images are linked to one case. The image processing method also includes performing an integration process. The integration process aggregates the multiple inference results using the acquired multiple inference results and the site information. The integration process aggregates the multiple inference results based on the aggregation result to determine the Helicobacter pylori infection state for each case.

[0138] This embodiment may also be implemented as a program or an information storage medium. That is, the program causes a computer to determine a Helicobacter pylori infection state using the trained model 130. The trained model 130 identifies a Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the infection state for each endoscopic image. The program causes a computer to obtain multiple inference results by inputting multiple endoscopic images to the trained model 130. Each of the multiple endoscopic images is assigned site information related to a site in the digestive tract, and the multiple endoscopic images are linked to one case. The program causes a computer to perform an integration process. The integration process aggregates the multiple inference results using the acquired multiple inference results and the site information. The integration process aggregates the multiple inference results based on the aggregation result to determine the Helicobacter pylori infection state for each case. The program may be stored in a computer-readable non-transitory information storage medium.

[0139] Although the present disclosure has been described above with reference to the present embodiment and its modifications, it is not limited to the embodiments and modifications as they are. In the implementation stage, the components may be modified and embodied without departing from the spirit of the present disclosure. Furthermore, multiple components disclosed in the above-described embodiments and modifications may be combined as appropriate. For example, some components may be omitted from all components described in the embodiments and modifications. Furthermore, components described in different embodiments and modifications may be combined as appropriate. In this manner, various modifications and applications are possible without departing from the spirit of the present disclosure. Furthermore, a term described at least once in the specification or drawings together with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. [Explanation of symbols]

[0140] 1...endoscopic system, 100...image processing system, 110...processor, 120...memory, 130...trained model, 140...program, 190...display, 200...endoscope, 250...video processor, 290...display, 410...image check model, 420...area recognition model, 430...H. pylori infection identification model, 440...integration processing, 450...H. pylori infection status, 520...vestibule specialized model, 530...RAC specialized model, 540...current infection specialized model, IM1 to IM9...endoscopic image

Claims

1. a memory that stores a trained model that identifies the Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the Helicobacter pylori infection state for each input endoscopic image; a processor that performs an integration process to input multiple endoscopic images associated with one case into the trained model to obtain multiple inference results, and to integrate the obtained multiple inference results to determine the Helicobacter pylori infection status for each case; Including, The processor: acquiring the plurality of inference results by inputting the plurality of endoscopic images, each of which has site information relating to a site of the digestive tract attached thereto, into the trained model; aggregating the acquired inference results using the plurality of inference results and the site information, and integrating the plurality of inference results based on the aggregation result to determine the Helicobacter pylori infection state of each case; The integration process includes: a first process for determining that the Helicobacter pylori infection state of the specific site is the first infection state when a ratio of a second number of images, which is the number of endoscopic images determined by the trained model to be in a first infection state, which is either currently infected, previously infected, or not infected, to a first number of images, which is the number of endoscopic images to which the site information of the specific site is attached, among the plurality of endoscopic images, is equal to or greater than a specific threshold value; a second process of determining that the H. pylori infection state of the specific site is one of the current infection, the previous infection, and the non-infection state other than the first infection state when the ratio of the second number of images to the first number of images is equal to or less than the specific threshold value; An image processing system comprising at least one of the above.

2. 2. The image processing system according to claim 1, The specific site is An image processing system, characterized in that the region of the digestive tract is a region where important findings regarding the first infection state occur.

3. 2. The image processing system according to claim 1, The specific site is An image processing system characterized in that the parts of the digestive tract include a part in which the first infection state is identified with the highest accuracy by the trained model.

4. 2. The image processing system according to claim 1, The trained model is An image processing system characterized by classifying an input endoscopic image into whether it corresponds to the currently infected, previously infected, or non-infected, and outputting the classification result as the inference result.

5. 2. The image processing system according to claim 1, The trained model is An image processing system characterized by determining whether an input endoscopic image corresponds to a specific infection state among the currently infected, previously infected, and uninfected, and outputting the determination result as the inference result.

6. 2. The image processing system according to claim 1, The trained model is An image processing system that determines whether or not atrophy is present in an input endoscopic image, and outputs the determination result as the inference result.

7. 2. The image processing system according to claim 1, The trained model is An image processing system characterized by determining the presence or absence of specific findings linked to a specific infection state among the current infection, the previous infection, and the non-infection from an input endoscopic image, and outputting the determination result as the inference result.

8. 2. The image processing system according to claim 1, The trained model is a first trained model that outputs the inference result from an endoscopic image that shows a part belonging to a first part group; a second trained model that outputs the inference result from an endoscopic image that shows a part belonging to a second part group; Including, The processor: A first plurality of inference results are obtained by inputting a plurality of endoscopic images showing parts belonging to the first part group into the first trained model, and a second plurality of inference results are obtained by inputting a plurality of endoscopic images showing parts belonging to the second part group into the second trained model; An image processing system characterized by determining the Helicobacter pylori infection state of each case based on the first plurality of inference results and the second plurality of inference results obtained.

9. 9. The image processing system according to claim 8, The processor: An image processing system characterized by determining the Helicobacter pylori infection status of each case based on a compilation result of the first plurality of inference results and a compilation result of the second plurality of inference results.

10. 9. The image processing system according to claim 8, The first trained model is An image processing system characterized by classifying an input endoscopic image into whether it corresponds to the currently infected, previously infected, or non-infected, and outputting the classification result as the inference result.

11. 9. The image processing system according to claim 8, The second trained model is An image processing system characterized by determining whether an input endoscopic image corresponds to a specific infection state among the currently infected, previously infected, and uninfected, and outputting the determination result as the inference result.

12. 9. The image processing system according to claim 8, The second trained model is An image processing system that determines whether or not atrophy is present in an input endoscopic image, and outputs the determination result as the inference result.

13. 9. The image processing system according to claim 8, The second trained model is An image processing system characterized by determining the presence or absence of specific findings linked to a specific infection state among the current infection, the previous infection, and the non-infection from an input endoscopic image, and outputting the determination result as the inference result.

14. 2. The image processing system according to claim 1, The processor: An image processing system characterized by performing a process of displaying on a display the judgment result of the Helicobacter pylori infection status obtained by the integration process and a list of endoscopic images from the multiple endoscopic images that have the same infection status as the judgment result.

15. A method for operating an image processing system using a trained model that identifies a Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the Helicobacter pylori infection state for each input endoscopic image, comprising: The image processing system inputs a plurality of endoscopic images, each of which is associated with one case and in which site information relating to a site of the digestive tract is assigned to the trained model, to the trained model, thereby obtaining a plurality of inference results; the image processing system aggregates the acquired inference results using the plurality of inference results and the site information, and integrates the plurality of inference results based on the aggregation result, thereby performing an integration process to determine the Helicobacter pylori infection state on a case-by-case basis; Including, The integration process includes: a first process for determining that the Helicobacter pylori infection state of the specific site is the first infection state when a ratio of a second number of images, which is the number of endoscopic images determined by the trained model to be in a first infection state, which is either currently infected, previously infected, or not infected, to a first number of images, which is the number of endoscopic images to which the site information of the specific site is attached, among the plurality of endoscopic images, is equal to or greater than a specific threshold value; a second process of determining that the H. pylori infection state of the specific site is one of the current infection, the previous infection, and the non-infection state other than the first infection state when the ratio of the second number of images to the first number of images is equal to or less than the specific threshold value; 2. A method for operating an image processing system, comprising:

16. A non-transitory information storage medium readable by a computer stores a program for causing a computer to determine a Helicobacter pylori infection state using a trained model that identifies a Helicobacter pylori infection state for each input endoscopic image and outputs an inference result indicating the Helicobacter pylori infection state for each of the endoscopic images, acquiring a plurality of inference results by inputting a plurality of endoscopic images, each of which is associated with a single case, into the trained model, wherein the endoscopic images each include site information relating to a site of the digestive tract; aggregating the acquired inference results using the plurality of inference results and the site information, and integrating the plurality of inference results based on the aggregation result, thereby performing an integration process for determining the Helicobacter pylori infection state for each case; a program for causing a computer to execute the above; The integration process includes: a first process for determining that the Helicobacter pylori infection state of the specific site is the first infection state when a ratio of a second number of images, which is the number of endoscopic images determined by the trained model to be in a first infection state, which is either currently infected, previously infected, or not infected, to a first number of images, which is the number of endoscopic images to which the site information of the specific site is attached, among the plurality of endoscopic images, is equal to or greater than a specific threshold value; a second process of determining that the H. pylori infection state of the specific site is one of the current infection, the previous infection, and the non-infection state other than the first infection state when the ratio of the second number of images to the first number of images is equal to or less than the specific threshold value; An information storage medium including at least one of the above.

Images