Endoscopic image diagnosis support processor, operation method for endoscopic image diagnosis support processor, and program for endoscopic image diagnosis support processor

Abstract

[Problem] To provide an endoscopic image diagnosis support processor 2 that stably and efficiently identifies an observed organ. [Solution] An endoscopic image diagnosis support processor 2 comprises: a site inference unit 13 that infers a certainty degree that an endoscopic image is a site image, an inappropriate image, or an indeterminable image; a site identification unit 14 that identifies the endoscopic image as a site image, an inappropriate image, or an indeterminable image; and an organ identification unit 15 that identifies an organ of the identified site as an observed organ. The organ identification unit 15: identifies an organ of a first site as the observed organ when the certainty degree of the first site is the highest; identifies an organ of a second site as the observed organ when the certainty factor of the inappropriate image is the highest and the certainty factor of the second site is the second highest; and identifies the organ identified immediately before as the observed organ when the certainty degree of the indeterminable image is the highest.

Description

Endoscopic Image Diagnosis Support Processor, Operating Method of Endoscopic Image Diagnosis Support Processor, and Program of Endoscopic Image Diagnosis Support Processor

[0001] Embodiments of the present invention relate to an endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed, an operating method of the endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed, and a program of the endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed.

[0002] For an image captured using an endoscope, computer-aided detection (CADe: Computer-Aided Detection) indicating the position of a lesion candidate and computer-aided diagnosis (CADx: Computer-Aided Diagnosis) indicating discrimination information of the lesion candidate are known (hereinafter, CADe and CADx are collectively referred to as "CAD"). In CAD, for example, a machine learning model obtained by performing deep learning or the like using a plurality of images as teacher data is used.

[0003] International Publication WO2022 / 18,1748 discloses a technique for generating a plurality of models respectively adapted to a plurality of organs by learning using images for each of the plurality of organs. For example, a plurality of types of models such as a pharyngeal model learned using only pharyngeal images and an esophageal model learned using only esophageal images are created. Then, by using a model adapted to the organ being observed with the endoscope from among the plurality of types of models, the accuracy as CAD can be increased. [[ID=Eleven]]

[0004] Japanese Patent Application Laid-Open No. 2007-151,809 discloses a receiving device that detects that the organ being photographed by a capsule endoscope has changed and receives data from the capsule endoscope.

[0005] However, it is not easy to identify the organ being observed with an endoscope when the imaging conditions are not good, and there are cases where the organ being observed cannot be identified.

[0006] International Publication WO2022 / 18,1748, Japanese Patent Application Laid-Open No. 2007-151,809

[0007] The present invention aims to provide an endoscopic image diagnostic support processor capable of reliably identifying an organ being observed with an endoscope, a method for operating the endoscopic image diagnostic support processor capable of reliably identifying an organ being observed with an endoscope, and a program for the endoscopic image diagnostic support processor that enables a computer to reliably identify an organ being observed with an endoscope.

[0008] The endoscopic image diagnostic support processor according to an embodiment of the present invention comprises: a site inference unit that, for a plurality of endoscopic images input in succession, uses a model trained on a plurality of images of a plurality of sites as training data to infer the confidence level of each endoscopic image being an image of one of the plurality of sites, an inappropriate image, or an undeterminable image; a site identification unit that identifies whether the endoscopic image is an image of one of the plurality of sites, an inappropriate image, or an undeterminable image based on predetermined conditions using the confidence levels; and an organ identification unit that identifies the organ corresponding to the observation site identified by the site identification unit as an observed organ. The organ identification unit identifies the organ corresponding to the first site as the observed organ if the highest confidence level is the confidence level of the first site; identifies the organ corresponding to the second site as the observed organ if the highest confidence level is the confidence level of the inappropriate image and the second highest confidence level is the confidence level of the second site; and identifies the organ identified immediately before as the observed organ if the highest confidence level is the confidence level of the undeterminable image.

[0009] The operation method of the endoscopic image diagnostic support processor according to an embodiment of the present invention involves, for a plurality of endoscopic images that are input sequentially, using a model that has learned multiple images of multiple body parts as training data, inferring the confidence level of each endoscopic image to be an image of one of the multiple body parts, an inappropriate image, or an undeterminable image, and, based on predetermined conditions using the confidence levels, identifying that the endoscopic image is an image of one of the multiple body parts, an inappropriate image, or an undeterminable image, identifying the organ corresponding to the identified body part as the organ to be observed, if the highest confidence level is the confidence level of the first body part, identifying the organ corresponding to the first body part as the organ to be observed, if the highest confidence level is the confidence level of the inappropriate image and the second highest confidence level is the confidence level of the second body part, identifying the organ corresponding to the second body part as the organ to be observed, and if the highest confidence level is the confidence level of the undeterminable image, identifying the organ identified immediately before as the organ to be observed.

[0010] The program for the endoscopic image diagnostic support processor according to an embodiment of the present invention, for a plurality of endoscopic images that are input sequentially, uses a model that has learned multiple images of multiple body parts as training data to infer the confidence level of each endoscopic image as an image of one of the multiple body parts, an inappropriate image, or an undeterminable image. Based on predetermined conditions using the confidence levels, it identifies whether the endoscopic image is an image of one of the multiple body parts, an inappropriate image, or an undeterminable image. It identifies the organ corresponding to the identified body part as the organ to be observed. If the highest confidence level is for the first body part, it identifies the organ corresponding to the first body part as the organ to be observed. If the highest confidence level is for the inappropriate image and the second highest confidence level is for the second body part, it identifies the organ corresponding to the second body part as the organ to be observed. If the highest confidence level is for the undeterminable image, it identifies the organ identified immediately before as the organ to be observed. The program causes the computer to perform these processes.

[0011] According to embodiments of the present invention, it is possible to provide an endoscopic image diagnostic support processor that can reliably identify an organ being observed with an endoscope, a method for operating the endoscopic image diagnostic support processor that can reliably identify an organ being observed with an endoscope, and a program for the endoscopic image diagnostic support processor that enables a computer to reliably identify an organ being observed with an endoscope.

[0012] Figure 1 is a configuration diagram of the endoscopic image diagnostic support processor of the embodiment. Figure 2 is a flowchart of the operation method of the endoscopic image diagnostic support processor of the first embodiment. Figure 3 is a diagram of the display screen of the display device of the endoscopic image diagnostic support processor of the embodiment. Figure 4 is a diagram showing an example of operation of the endoscopic image diagnostic support processor of the first embodiment. Figure 5 is a flowchart of the operation method of the endoscopic image diagnostic support processor of the second embodiment. Figure 6 is a flowchart of the operation method of the endoscopic image diagnostic support processor of the third embodiment.

[0013] Embodiments of the present invention will be described below with reference to the drawings. The drawings based on the embodiments are schematic. The illustration and reference numerals of some components have been omitted.

[0014] <First Embodiment> As shown in Figure 1, the endoscopic image diagnostic support processor 2 (hereinafter referred to as "processor 2") of this embodiment, together with the endoscope 9, display device 30, notification unit 31, and storage device 32, constitutes the endoscopic system 1.

[0015] The camera unit 8 of the endoscope 9 captures images of the inside of the subject's body. The endoscope 9 outputs image data at a predetermined frame rate. The user can capture still images, for example, by operating the operation buttons 7 on the control panel (not shown) of the endoscope 9. The endoscope 9 may be either a flexible or rigid endoscope.

[0016] The processor 2 includes an observed organ determination device 10 (hereinafter referred to as "determination device 10") and a lesion detection device 20.

[0017] The determination device 10 includes a CPU 11, an image input unit 12, a site inference unit 13, a site identification unit 14, an organ identification unit 15, and a memory 16. The processor 2 is also connected to the server 40.

[0018] The CPU 11 is a system controller that controls the entire processor 2. The image input unit 12 processes images taken by the endoscope 9 at a predetermined frame rate. Alternatively, the image input unit 12 may receive endoscopic images processed by an image processing processor (not shown) or a server 40 from the image data of the endoscope 9.

[0019] The site inference unit 13 infers the confidence level for each of the multiple sites in the endoscopic image, whether it is an inappropriate image, or whether it is an undeterminable image. For example, the site inference unit 13 is a first AI processing unit that infers the confidence level for each of the multiple sites in the input endoscopic image using a model that has been trained on multiple images of multiple sites as training data.

[0020] The site inference unit 13 also infers the confidence level PA of whether the endoscopic image is an inappropriate image. An inappropriate image is a so-called out-of-focus image, a blurred image, etc. For example, a model trained on inappropriate / appropriate images as training data outputs a confidence level PA, and if the confidence level PA is above a predetermined value, the endoscopic image is inferred to be an inappropriate image.

[0021] Furthermore, the site inference unit 13 estimates a confidence level PB for "undeterminable" if the endoscopic image is not any of the n types of sites. For example, a model trained on undeterminable / determinable images as training data outputs a confidence level PB, and if the confidence level PB is greater than or equal to a predetermined value, the endoscopic image is inferred to be an undeterminable image. The site inference unit 13 may also infer "undeterminable" if the confidence levels P for all n types of sites output by the first model are less than or equal to a predetermined value.

[0022] The site inference unit 13 may be configured to infer (n+2) confidence levels for each of the endoscopic images, which are classified as images of n different sites, inappropriate images, and undeterminable images, using a single first model.

[0023] The site identification unit 14 identifies an observation site, which is a part of the endoscopic image, from among multiple sites based on predetermined conditions using a confidence level P. The site identification unit 14 also identifies if the endoscopic image is an inappropriate image or an undeterminable image.

[0024] The organ identification unit 15 identifies the organ to be observed corresponding to the observation site. The memory 16 stores various data, such as the correspondence between the observation site and the observed organ.

[0025] The lesion detection device 20 is a second AI processing unit that infers lesions in images using multiple second models (organ CADs) that have been trained using multiple lesion images for multiple organs as training data.

[0026] The lesion detection device 20 includes a nasal cavity CAD (second model 1) 21, an oral cavity CAD (second model 2) 22, a pharyngeal CAD (second model 3) 23, an esophageal CAD (second model 4) 24, a gastric CAD (second model 5) 25, a duodenal CAD (second model 6) 26, and an ON / OFF control unit 29 for controlling the activation / deactivation of each model. Each model may also have its own ON / OFF control unit.

[0027] Furthermore, at least one of the multiple configurations of the processor 2 may consist of internal circuits of semiconductor elements processed by software, or dedicated hardware circuits, or it may include both internal circuits of semiconductor elements and dedicated hardware circuits. At least one of the functional parts of the processor 2 may also be a component of the server 40. For example, the lesion detection device 20 may be an operating part of the server 40.

[0028] For example, the body part inference unit 13 may include an FPGA (Field Programmable Gate Array) with hardware circuits and a GPU (Graphics Processing Unit) with software circuits. Also, the body part identification unit 14, organ identification unit 15, etc., may be part of a CPU 11 that processes data using software.

[0029] The display device 30, which displays the endoscopic image, is a monitor such as a liquid crystal display. The notification unit 31 informs the user of information from the processor 2 using sound, light, etc. The notification unit 31 may be part of the display device 30.

[0030] The storage device 32 is a non-temporary storage device (e.g., a magnetic disk, optical disk, or HDD) that stores the program of the processor 2. The storage device 32 may also be configured as a server 40 connected to the processor 2 via an internet connection or the like.

[0031] As described later, the organ identification unit 15 identifies the organ corresponding to the first site as the observed organ if the highest confidence level is the confidence level for the first site, and identifies the organ corresponding to the second site as the observed organ if the highest confidence level is the confidence level for an inappropriate image and the second highest confidence level is the confidence level for the second site.

[0032] In processor 2, the determination device 10 can efficiently and stably identify the organ being observed in the endoscopic image. The second CAD model of the lesion detection device 20 is created for each of the multiple organs. As a result, processor 2 has good model learning efficiency and high inference reliability.

[0033] <Processor Operation Method> The operation method of processor 2 will be explained according to the flowchart in Figure 2.

[0034] <Step S10> Image Input The endoscopic system 1 of this embodiment performs an examination of the upper gastrointestinal tract. The insertion part of the endoscope 9 is inserted through the oral cavity or nasal cavity of the subject and enters the pharynx. For example, multiple endoscopic images from the endoscope 9 or the image processing processor are input continuously to the image input unit 12 of the processor 2 at a predetermined frame rate (e.g., 120 fps). Processing is not performed if no images are input.

[0035] The image input unit 12 performs image processing on images directly input from the endoscope 9 (for example, RAW images) for display on the display device 30. If an image processed by an image processing processor is input, no image processing is required.

[0036] <Step S20> The inference unit 13 infers the confidence level PN (N=1 to n) that the input endoscopic image is an image of one of n different regions. The value of the confidence level P is in the range of (0 to 1). A confidence level P=0 means a probability of 0%, and a confidence level P=1 means a probability of 100%.

[0037] The machine learning model of the AI processing unit is a deep neural network (DNN: Deep Neural Network) that has multiple hidden layers and performs deep learning. The model may be a convolutional neural network (CNN: Convolution Neural Network), an R-CNN (Regions with CNN features) using a CNN, or a fully convolutional network (FCN), etc.

[0038] The first model is trained using images of n types of sites in the upper digestive tract as teacher data. As shown in Table 1 below, in this embodiment, n = 15.

[0039]

[0040] Each of the 15 types of sites corresponds to one of m types of organs. As shown in Table 2 below, in this embodiment, m = 6.

[0041]

[0042] The number of types of sites n is, for example, 2 or more and 100 or less. The number of types of organs m is (m < n) and is, for example, 2 or more and 20 or less. And the lesion detection device 20 has m types of organ CADs corresponding to m types of organs.

[0043] <Step S30> When the maximum value of the maximum confidence P of the confidence PX of the site X is the confidence PX of the site X (YES), the process from step S40 is performed.

[0044] <Step S40> The site identification unit 14 identifies the endoscopic image as the image of the site X. In other words, the site X is identified as the observation site during observation.

[0045] <Step S50> The organ identification unit 15 identifies the organ Y corresponding to the site X as the observed organ during observation.

[0046] <Step S60> Is the corresponding second model running? It is determined by, for example, the CPU 11 whether the second model Y (organ CAD) corresponding to organ Y is running. For example, when the site X is site 9 (midbody), the corresponding organ is (organ 5: stomach). The second model Y corresponding to (organ 5: stomach) is the second model 5 (stomach CAD). If the running second model is the second model 5 (YES), the process proceeds to step S80. That is, the running second model 5 is continuously used. If the running second model is not the second model 5 (NO), the process proceeds to step S70.

[0047] <Step S70> Start the corresponding second model (ON) The ON / OFF control unit 29 of the lesion detection device 20 starts the CAD (second model Y) corresponding to the observed organ Y. Also, the ON / OFF control unit 29 stops (turns off) the CAD other than (second model Y).

[0048] <Step S80> Lesion detection process The activated CAD of the lesion detection device 20 infers the lesion in the endoscopic image. The inference result is output as the confidence PP (PP = 0 to 1) inferred by the CAD corresponding to the observed organ Y. For example, when the observed organ is the stomach, it is inferred by the corresponding stomach CAD (second model 5).

[0049] For example, the confidence PP1 of normal tissue is 0.66, and the confidence PP2 of lesion tissue is 0.34 and output. The lesion detection device 20 may output the confidence PP1 or the confidence PP2.

[0050] At the time of startup of the processor 2, that is, at the start of the inspection, since there is no organ specified as the observed organ, there is no activated CAD. Therefore, the lesion detection process is not performed.

[0051] Note that the "organ CAD" may include a "boundary CAD" that performs a lesion detection process on an image including a plurality of organs. For example, when the confidence of site 4 (esophagus) and the confidence of site 5 (esophagogastric junction) are both above a predetermined confidence P, a boundary organ including the esophagus and the stomach is specified, and the corresponding boundary CAD may infer the lesion in the input image.

[0052] <Step S90> The display device 30 displays the observation site identified by the site identification unit 14, and / or the observation organ identified by the organ identification unit 15, as numerical values, illustrations, colors, etc. The display device 30 may also serve as a notification unit that notifies the observed organ.

[0053] Figure 3 shows an example of the screen of the display device 30. The screen in Figure 3 displays the endoscopic image 30A, the status of the judgment device 10 (AUTO mode) 30B, and the name of the detected observation site (antrum) as "Detected Site" 30C. Furthermore, the fact that the CAD currently in operation is (stomach) is graphically displayed using a figure 30D representing a stomach.

[0054] Although not shown in the diagram, if the confidence level PP2 for lesion tissue is greater than or equal to a predetermined threshold TPP, the lesion candidate may be superimposed on the endoscopic image 35A, or the confidence level PP2 may be displayed numerically or in color. For example, lesion candidates with a confidence level PP2 of 0.7 or higher may be superimposed on the endoscopic image 35A in red. Conversely, if the confidence level PP2 for lesion tissue is less than the predetermined threshold, no display is made by the lesion detection device 20. Then, the process from step S10 is repeated.

[0055] <Step S100> If the confidence level PA is at its maximum, and the confidence level PX for site X is not at its maximum in step S30 (S30: NO), if the confidence level PA that the endoscopic image is an inappropriate image is at its maximum (S100: YES), then the process in step S110 is performed.

[0056] <Step S110> If the second highest site confidence is PX2 of site X2 (S110: YES), then in step S40, site X2 is identified as the observation site. Then, the processing in steps S50-S90 is performed. That is, lesion detection processing is performed using the second model of the organ corresponding to site X2.

[0057] If the second highest site confidence score PN is not PX2 of site X2 (S110: NO), the organ identified immediately beforehand is used as the organ to be observed, and the lesion detection process is performed using the second model that is currently running.

[0058] <Step S120> If the confidence level PB is the highest and the endoscopic image is an undeterminable image (S100: NO), the organ identified immediately before is used as the organ to be observed, and the lesion detection process is performed using the second model that is currently running.

[0059] In the operation method of processor 2, the determination device 10 can efficiently and stably identify the observed organ in the endoscopic image being observed. The second CAD model of the lesion detection device 20 is created for each of multiple organs. Therefore, the operation method of processor 2 has good model learning efficiency and high inference reliability.

[0060] An example of processor 2 operation will be explained using Figure 4. (1) The second model being started is "Organ 3 (pharynx) CAD". The highest level of confidence is "Location 3: Pharynx". Therefore, "Organ 3 (pharynx)" is identified as the organ to be observed, and processing of "Organ 3 (pharynx) CAD" continues.

[0061] (2) The second model currently running is "Organ 3 (Pharynx) CAD". The highest level of confidence is "Location 4: Esophagus". Therefore, "Organ 4 (Esophagus)" is identified as the organ to be observed, "Organ 4 (Esophagus) CAD" is started, and "Organ 3 (Pharynx) CAD" is stopped.

[0062] (3) The second model currently running is "Organ 3 (Pharynx) CAD". The highest level of confidence is "Location 6: Dome". Therefore, "Organ 5 (Stomach)" is identified as the organ to be observed, "Organ 5 (Stomach) CAD" is started, and "Organ 3 (Pharynx) CAD" is stopped.

[0063] (4) The second model currently running is "Organ 3 (Pharynx) CAD". The highest confidence level is "Inappropriate". The second highest confidence level is "Site 3: Pharynx", therefore "Organ 3 (Pharynx)" is identified as the organ to be observed, and processing of "Organ 3 (Pharynx) CAD" continues.

[0064] (5) The second model currently running is "Organ 3 (Pharynx) CAD". The highest confidence level is "Inappropriate". The second highest confidence level is "Site 4: Esophagus", therefore "Organ 4 (Esophagus)" is identified as the organ to be observed, "Organ 4 (Esophagus) CAD" is started, and "Organ 3 (Pharynx) CAD" is stopped.

[0065] (6) The second model currently running is "Organ 3 (Pharynx) CAD". The highest confidence level is "Inappropriate". The second highest confidence level is "Location 6: Dome", therefore "Organ 5 (Stomach)" is identified as the organ to be observed, "Organ 5 (Stomach) CAD" is started, and "Organ 3 (Pharynx) CAD" is stopped.

[0066] (7) The second model currently running is "Organ 3 (Pharynx) CAD". The highest level of confidence is "Inappropriate". The second highest level of confidence is "Undeterminable". Therefore, the organ that was identified immediately before, "Organ 3 (Pharynx)", will be used as the organ to observe, and processing of "Organ 3 (Pharynx) CAD" will continue.

[0067] (8) The second model currently running is "Organ 3 (Pharynx) CAD". The highest level of confidence is "Undetermined". Therefore, the organ identified immediately before, "Organ 3 (Pharynx)", will be used as the organ to observe, and processing of "Organ 3 (Pharynx) CAD" will continue.

[0068] As described above, the operation method of the endoscopic image diagnostic support processor of this embodiment involves, for multiple endoscopic images that are input in succession, using a model that has learned multiple images of multiple body parts as training data, inferring the confidence level of each endoscopic image to be an image of one of the multiple body parts, an inappropriate image, or an undeterminable image, and, based on predetermined conditions using the confidence levels, identifying that the endoscopic image is an image of one of the multiple body parts, an inappropriate image, or an undeterminable image, identifying the organ corresponding to the identified body part as the organ to be observed, if the highest confidence level is the confidence level of the first body part, identifying the organ corresponding to the first body part as the organ to be observed, if the highest confidence level is the confidence level of the inappropriate image and the second highest confidence level is the confidence level of the second body part, identifying the organ corresponding to the second body part as the organ to be observed, and if the highest confidence level is the confidence level of the undeterminable image, identifying the organ identified immediately before as the organ to be observed.

[0069] The program for the endoscopic image diagnostic support processor according to an embodiment of the present invention, for a plurality of endoscopic images that are input sequentially, uses a model that has learned multiple images of multiple body parts as training data to infer the confidence level of each endoscopic image as an image of one of the multiple body parts, an inappropriate image, or an undeterminable image. Based on predetermined conditions using the confidence levels, it identifies whether the endoscopic image is an image of one of the multiple body parts, an inappropriate image, or an undeterminable image. It identifies the organ corresponding to the identified body part as the organ to be observed. If the highest confidence level is the confidence level for the first body part, it identifies the organ corresponding to the first body part as the organ to be observed. If the highest confidence level is the confidence level for the inappropriate image and the second highest confidence level is the confidence level for the second body part, it identifies the organ corresponding to the second body part as the organ to be observed. If the highest confidence level is the confidence level for the undeterminable image, it identifies the organ identified immediately before as the organ to be observed. The program causes the computer to perform these processes.

[0070] The non-temporary storage medium, the storage device 32, may store only a part of the program. Furthermore, the program may be distributed or provided via a communication network. Users can execute all or part of the processing, and thus perform the processing of the processor described above, by installing the program from the storage medium to their computer, or by downloading the program via the communication network and installing it to their computer.

[0071] Furthermore, the site identification unit 14 may identify a site as the observation site if the site with the highest confidence level P is the same site for a predetermined number of consecutive times or more. Also, even if the observation site inferred by the site inference unit 13 is different, the organ identification unit 15 may identify the corresponding organ as the observation organ if the organ corresponding to the observation site is the same for a consecutive number of times. In other words, after the organ identification unit 15 has provisionally determined an observation organ, it may finalize it as the observation organ if an organ different from the organ corresponding to the currently running CAD is provisionally observed for a certain number of consecutive times or more.

[0072] In other words, the organ identification unit 15 may make a final decision to change the organ after it has provisionally determined an organ different from the one in the previous endoscopic image, and after the same organ has been provisionally determined as the organ to be observed a certain number of times in a row.

[0073] <Second Embodiment> The endoscopic image diagnostic support processors 2A and 2B of the embodiments described below are similar to the endoscopic image diagnostic support processor 2 of the first embodiment and have the same effects as the endoscopic image diagnostic support processor 2. For this reason, components with the same functions as the endoscopic image diagnostic support processor 2 are denoted by the same reference numerals as the endoscopic image diagnostic support processor 2, and their descriptions are omitted.

[0074] As shown in the flowchart of Figure 5, in the processor 2A of this embodiment, in step S110, if the second highest confidence level for a location is the confidence level PX2 for location X2 (S110: YES), then in step S112, the location identification unit 14 of the determination device 10A determines whether the confidence level PX2 is equal to or greater than a predetermined first determination value TP.

[0075] In step S112, if the confidence level PX2 is less than the first decision value TP (S112: NO), the organ identified immediately before is used as the organ to be observed. That is, the lesion detection process is performed using the second model (CAD) that is currently running. If the confidence level PX2 is equal to or greater than the first decision value TP (S112: YES), in step S40, site X2 is identified as the site to be observed.

[0076] Furthermore, if the confidence level PX for site X is at its maximum (S30: YES), it is permissible to determine whether the confidence level PX is equal to or greater than a predetermined judgment value. If the confidence level PX is less than the first judgment value, the organ identified immediately beforehand is used as the organ to be observed.

[0077] In other words, the organ identification unit 15 will consider the organ identified immediately before as the organ to be observed, even if the confidence level PX or PX2 is less than a predetermined first judgment value.

[0078] For example, if the second model currently running is "Organ 3 (Pharynx) CAD," and the highest confidence level is "Inappropriate," and the second highest site confidence level, PX2, is less than the predetermined first judgment value TP, then the previously identified organ, "Organ 3 (Pharynx)," is used as the organ to be observed, and processing of "Organ 3 (Pharynx) CAD" continues.

[0079] Processor 2A can more efficiently and stably identify the organ being observed in the endoscopic image than processor 2.

[0080] <Third Embodiment> As shown in the flowchart of Figure 6, in the processor 2B of this embodiment, in step S110, if the second highest confidence level for a site is PX2 for the second site (site X2), the site identification unit 14 of the determination device 10B identifies the organ corresponding to site X2 in step S114.

[0081] Then, in step S116, the positional relationship between the observed organ identified in step S114 and the observed organ identified in the previous frame is determined. If there are no other organs between the two (S116: NO), the organ identified in step S114 is used, and the process proceeds to step S60. If there are other organs between the two (S116: YES), the organ identified immediately before is used as the observed organ, the second model is not changed, and the process proceeds to step S80. In other words, if there are other organs between the position of the organ corresponding to the second part X2 and the position of the organ identified immediately before by the organ identification unit 15, the organ identified immediately before is used as the observed organ.

[0082] The relative positions of multiple organs are stored, for example, in memory 16. The relative positions of multiple organs follow the insertion route of the endoscope 9, in the order of nasal cavity or oral cavity, pharynx, esophagus, stomach, and duodenum.

[0083] For example, if the organ identified immediately before was organ 3 (pharynx), and the organ being observed this time is organ 5 (stomach), then organ 4 (esophagus) is in between. Therefore, organ 3 (pharynx) is determined to be the organ being observed, and the lesion detection process is performed using the second model (CAD) that is currently running.

[0084] Furthermore, if the confidence level PX for site X is at its maximum (S30: YES), and there are other organs between the location of the identified observed organ and the location of the previously observed organ, the second model that is currently running may be used continuously.

[0085] If there are no other organs between the identified organ being observed and the organ observed immediately before it, the identified organ is determined to be the organ being observed.

[0086] Processor 2B can more efficiently and stably identify the organ being observed in the endoscopic image than processor 2.

[0087] The range of values ​​described above is not limited to that range and can be increased or decreased as appropriate. Furthermore, the present invention is not limited to the embodiments described above, and various changes, modifications, and combinations can be made without altering the essence of the present invention.

[0088] 1... Endoscopy system 2... Endoscopy image diagnostic support processor (processor) 9... Endoscope 10... Observed organ determination device (determination device) 11... CPU 12... Image input unit 13... Site inference unit 14... Site identification unit 15... Organ identification unit 16... Memory 20... Lesion detection device 30... Display device 31... Notification unit 32... Storage device 40... Server

Claims

1. Endoscopic image diagnostic support processor comprising: a site inference unit that, for multiple endoscopic images input in succession, uses a model trained on multiple images of multiple sites as training data to infer the confidence level of each endoscopic image being an image of one of the multiple sites, an inappropriate image, or an undeterminable image; a site identification unit that identifies whether the endoscopic image is an image of one of the multiple sites, an inappropriate image, or an undeterminable image based on predetermined conditions using the confidence levels; and an organ identification unit that identifies the organ corresponding to the observation site identified by the site identification unit as an observation organ, wherein the organ identification unit identifies the organ corresponding to the first site as the observation organ if the highest confidence level is the confidence level of the first site; identifies the organ corresponding to the second site as the observation organ if the highest confidence level is the confidence level of the inappropriate image and the second highest confidence level is the confidence level of the second site; and identifies the organ identified immediately before as the observation organ if the highest confidence level is the confidence level of the undeterminable image.

2. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit determines the organ identified immediately beforehand to be the organ to be observed when the highest confidence level is the confidence level of the inappropriate image and the second highest confidence level is the confidence level of the undeterminable image.

3. The endoscopic image diagnostic support processor according to claim 2, wherein the organ identification unit determines the organ identified immediately beforehand to be the organ to be observed if the highest confidence level is the confidence level of the inappropriate image and the confidence level of the second area is less than a predetermined first determination value.

4. The endoscopic image diagnostic support processor according to claim 1, wherein, in the case where there is a positional relationship between the identified organ and the organ identified immediately before, the organ identified immediately before is used as the organ to be observed.

5. The endoscopic image diagnostic support processor according to claim 1, further comprising a lesion detection device that infers lesions in an image using a plurality of second models that have been trained using a plurality of lesion images for a plurality of organs as training data, wherein the lesion detection device uses a second model from the plurality of second models that corresponds to the observed organ identified by the organ identification unit.

6. The endoscopic image diagnostic support processor according to claim 1, further comprising a notification unit that notifies the observed organ identified by the organ identification unit.

7. A method for operating an endoscopic image diagnostic support processor, which, in response to multiple endoscopic images input in succession, uses a model trained on multiple images of multiple body parts as training data to infer the confidence level of each endoscopic image being an image of one of the multiple body parts, an inappropriate image, or an undeterminable image; identifies the endoscopic image as an image of one of the multiple body parts, an inappropriate image, or an undeterminable image based on predetermined conditions using the confidence levels; identifies the organ corresponding to the identified body part as the organ to be observed; identifies the organ corresponding to the first body part as the organ to be observed if the highest confidence level is the confidence level of the first body part; identifies the organ corresponding to the second body part as the organ to be observed if the highest confidence level is the confidence level of the inappropriate image and the second highest confidence level is the confidence level of the second body part; and identifies the organ identified immediately before as the organ to be observed if the highest confidence level is the confidence level of the undeterminable image.

8. A program for an endoscopic image diagnostic support processor that causes a computer to perform the following processes: for multiple endoscopic images that are input in succession, using a model trained on multiple images of multiple body parts as training data, inferring the confidence level for each endoscopic image to be an image of one of the multiple body parts, an inappropriate image, or an undeterminable image; identifying the endoscopic image as an image of one of the multiple body parts, an inappropriate image, or an undeterminable image based on predetermined conditions using the confidence levels; identifying the organ corresponding to the identified body part as the organ to be observed; identifying the organ corresponding to the first body part as the organ to be observed if the highest confidence level is the confidence level for the first body part; identifying the organ corresponding to the second body part as the organ to be observed if the highest confidence level is the confidence level for the inappropriate image and the second highest confidence level is the confidence level for the second body part; and identifying the organ identified immediately before as the organ to be observed if the highest confidence level is the undeterminable image.

Images