Endoscopic image diagnostic support processor, operation method for endoscopic image diagnostic support processor, and program for endoscopic image diagnostic support processor

Abstract

[Problem] To provide an endoscopic image diagnostic support processor (2) for stably identifying an organ under observation during observation using an endoscope (9). [Solution] An endoscopic image diagnostic support processor (2) comprises: a lesion estimation unit (20) that uses a plurality of models (21-26), each of which was trained with a plurality of lesion images and a plurality of normal images for each of organs as training data, to estimate a normality confidence score PP1 and a lesion confidence score PP2 for each of a plurality of continuously input endoscopic images; and an organ identification unit (15) that identifies the organ under observation in the endoscopic images on the basis of a prescribed condition using the normality confidence score PP1 and the lesion confidence score PP2 for each of the plurality of endoscopic images.

Description

Endoscopic image diagnosis support processor, method of operating an endoscopic image diagnosis support processor, and program for an endoscopic image diagnosis support processor

[0001] Embodiments of the present invention relate to an endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed, a method of operating an endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed, and a program for an endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed.

[0002] For an image captured using an endoscope, computer-aided detection (CADe: Computer-Aided Detection) indicating the position of a lesion candidate and computer-aided diagnosis (CADx: Computer-Aided Diagnosis) indicating discrimination information of the lesion candidate are known (hereinafter, CADe and CADx are collectively referred to as "CAD"). In CAD, for example, a machine learning model that has performed deep learning or the like using a plurality of images as teacher data is used.

[0003] International Publication WO2022 / 181748 discloses a technique for generating a plurality of models respectively adapted to a plurality of organs by learning using images of each of the plurality of organs. For example, a plurality of types of models such as a pharyngeal model learned using only pharyngeal images and an esophageal model learned using only esophageal images are created. Then, by using a model adapted to the organ being observed with the endoscope from among the plurality of types of models, the accuracy as CAD can be increased.

[0004] Japanese Patent Application Laid-Open No. 2007-151809 discloses a receiving device that detects that the organ being imaged by a capsule endoscope has changed and receives data from the capsule endoscope.

[0005] However, it is not easy to identify the organ being observed with an endoscope when the imaging conditions are not good, and in some cases, there is a risk that an incorrect organ may be identified. Also, even when the imaging conditions are good, there are cases where the organ being observed cannot be correctly identified 100%.

[0006] International Publication WO2022 / 181748, Japanese Patent Publication No. 2007-151809

[0007] Embodiments of the present invention aim to provide an endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, a method for operating the endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, and a program for the endoscopic image diagnostic support processor that allows a computer to reliably identify an organ being observed with an endoscope.

[0008] An endoscopic image diagnostic support processor according to an embodiment of the present invention comprises: a lesion estimation unit that estimates the normal confidence level and lesion confidence level for each of the multiple endoscopic images by using multiple models, each of which has been learned using multiple lesion images and multiple normal images as training data for each of the multiple organs, for multiple endoscopic images that are input sequentially; and an organ identification unit that identifies the organ being observed in the endoscopic image based on predetermined conditions using the normal confidence level and lesion confidence level for each of the multiple endoscopic images.

[0009] The operation method of the endoscopic image diagnostic support processor according to an embodiment of the present invention involves, for multiple endoscopic images that are input sequentially, estimating the normal confidence level and the lesion confidence level for each of the multiple endoscopic images using multiple models, each model having learned from multiple lesion images and multiple normal images for each of the multiple organs as training data, and identifying the organ being observed in the endoscopic image based on predetermined conditions using the normal confidence level and the lesion confidence level for each of the multiple endoscopic images.

[0010] The program for the endoscopic image diagnostic support processor according to an embodiment of the present invention estimates the normal confidence level and lesion confidence level for each of the multiple endoscopic images that are input sequentially, using multiple models each that have learned using multiple lesion images and multiple normal images for each of the multiple organs as training data, and causes the computer to perform a process to identify the observed organ in the endoscopic image based on predetermined conditions using the normal confidence level and lesion confidence level for each of the multiple endoscopic images.

[0011] According to embodiments of the present invention, an endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, a method for operating the endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, and a program for the endoscopic image diagnostic support processor that allows a computer to reliably identify an organ being observed with an endoscope are provided.

[0012] Figure 1 is a configuration diagram of the endoscopic image diagnostic support processor of the embodiment. Figure 2 is a flowchart of the endoscopic image diagnostic support processor of the first embodiment. Figure 3 is an example of the screen of the display device of the embodiment. Figure 4 is a flowchart of the endoscopic image diagnostic support processor of modification 1 of the first embodiment. Figure 5 is a flowchart of the endoscopic image diagnostic support processor of the second embodiment. Figure 6 is a flowchart of the endoscopic image diagnostic support processor of the third embodiment.

[0013] Embodiments of the present invention will be described below with reference to the drawings. The drawings based on the embodiments are schematic. The illustration and reference numerals of some components have been omitted.

[0014] <First Embodiment> As shown in Figure 1, the endoscopic image diagnostic support processor 2 (hereinafter referred to as "processor 2") of this embodiment, together with the endoscope 9, display device 30, notification unit 31, and storage device 32, constitutes the endoscopic system 1.

[0015] The camera unit 8 of the endoscope 9 captures images of the inside of the subject's body. The endoscope 9 outputs image data at a predetermined frame rate. The user can capture still images, for example, by operating the operation buttons 7 on the control panel (not shown) of the endoscope 9. The endoscope 9 may be either a flexible or rigid endoscope.

[0016] Processor 2 includes a CPU 11, an image input unit 12, an organ identification unit 15, a memory 16, and a lesion estimation unit 20. Processor 2 is also connected to server 40.

[0017] The CPU 11 is a system controller that controls the entire processor 2. The image input unit 12 processes images taken by the endoscope 9 at a predetermined frame rate. Alternatively, the image input unit 12 may receive endoscope images processed by an image processing processor (not shown) or a server 40 from the image data from the endoscope 9.

[0018] The lesion estimation unit 20 is an AI processing unit that infers lesions from images using multiple models (organ CAD) that have been trained using multiple lesion images for multiple organs as training data.

[0019] The lesion estimation unit 20 includes a nasal cavity CAD (second model 1) 21, an oral cavity CAD (second model 2) 22, a pharyngeal CAD (second model 3) 23, an esophageal CAD (second model 4) 24, a gastric CAD (second model 5) 25, a duodenal CAD (second model 6) 26, and a display control unit 29 that controls the display of each model. Each model may also have its own display control unit.

[0020] The lesion estimation unit 20 estimates the normality confidence PP1 and lesion confidence PP2 for each of the multiple endoscopic images, using multiple models that have been trained on multiple lesion images and multiple normal images for each of the multiple organs as training data, for multiple endoscopic images that are input sequentially.

[0021] The machine learning model in the AI ​​processing unit is a deep neural network (DNN) that performs deep learning with multiple hidden layers. The model can also be a convolutional neural network (CNN), R-CNN (Regions with CNN features) which utilizes CNN, or FCN (Fully Convolutional Networks), etc.

[0022] The organ identification unit 15 identifies the organ being observed by the endoscope based on predetermined conditions using the normal confidence level and the lesion confidence level estimated by the lesion estimation unit 20. The memory 16 stores various data and programs.

[0023] Furthermore, at least one of the multiple configurations of the processor 2 may consist of internal circuits of semiconductor elements processed by software, or dedicated hardware circuits, or it may include both internal circuits of semiconductor elements and dedicated hardware circuits. At least one of the functional parts of the processor 2 may also be a component of the server 40. For example, the lesion estimation unit 20 may be an operating unit of the server 40.

[0024] For example, the lesion estimation unit 20 may include an FPGA (Field Programmable Gate Array) with hardware circuits and a GPU (Graphics Processing Unit) with software circuits. Also, the organ identification unit 15, etc., may be part of a CPU 11 that processes data using software.

[0025] The display device 30, which displays the endoscopic image, is a monitor such as a liquid crystal display. The notification unit 31 informs the user of information from the processor 2 using sound, light, etc. The notification unit 31 may be part of the display device 30.

[0026] The storage device 32 is a non-temporary storage device (e.g., a magnetic disk, optical disk, or HDD) that stores the program of the processor 2. The storage device 32 may also be configured as a server 40 connected to the processor 2 via an internet connection or the like.

[0027] As described later, the processor 2 identifies the organ being observed in the endoscopic image based on predetermined conditions using the normal confidence level PP1 and the lesion confidence level PP2 for each of the multiple endoscopic images.

[0028] Therefore, processor 2 can efficiently and reliably identify the organ being observed in the endoscopic image. Organ CAD models are created for each of multiple organs. As a result, processor 2 has high model learning efficiency and high inference reliability.

[0029] <Processor Operation Method> The operation method of processor 2 will be explained according to the flowchart in Figure 2.

[0030] <Step S10> Image Input The endoscopic system 1 of this embodiment performs an examination of the upper gastrointestinal tract. The insertion part of the endoscope 9 is inserted through the oral cavity or nasal cavity of the subject and enters the pharynx. For example, multiple endoscopic images from the endoscope 9 or the image processing processor are input to the image input unit 12 of the processor 2 in succession at a predetermined frame rate. Processing is not performed if no images are input.

[0031] The image input unit 12 performs image processing on images directly input from the endoscope 9 (for example, RAW images) for display on the display device 30. If an image processed by an image processing processor is input, no image processing is required.

[0032] <Step S20> Each of the m organ CADs of the inferred lesion estimation unit 20 infers the lesion from the endoscopic image. In this embodiment, as shown in Table 1 below, the number of organ types m = 6. The number of organ types m is, for example, 2 or more and 20 or less.

[0033]

[0034] The inference result is output as a confidence score PP. The value of the confidence score PP is in the range of (0 to 1). A confidence score P = 0 means a 0% probability, and a confidence score P = 1 means a 100% probability. For example, a confidence score PP1 of 0.66 for normal tissue and a confidence score PP2 of 0.34 for diseased tissue are output (PP1 + PP2 = 1.0). The lesion estimation unit 20 may output either a confidence score PP1 or a confidence score PP2.

[0035] Note that not all organ CADs need to be running at all times. For example, if the insertion device is inserted through the nose, organ CAD2 (oral cavity) may be stopped. At the start of the examination, for example, organ CAD5 (stomach) and organ CAD6 (duodenum) may be stopped.

[0036] <Step S30> Organ X has the highest confidence level. In the processor 2 of this embodiment, the organ identification unit 15 identifies the organ with the highest first confidence level P1 among the confidence levels PP (PP1 or PP2) as the organ to be observed.

[0037] For example, in the case of Table 2 below, since the first confidence level P1 with the highest confidence level in the organ identification unit 15 is the confidence level PP1(6) of the normal tissue of the duodenal CAD (Model 6), the "duodenum" is identified as the observed organ.

[0038]

[0039] <Step S40> Display

[0040] On the display device 30, the observed organ etc. specified by the organ identification unit 15 is displayed as a numerical value, illustration, color, etc. The display device 30 may also serve as a notification unit for notifying the observed organ.

[0041] FIG. 3 shows an example of the screen of the display device 30. On the screen of FIG. 3, an endoscope image 30A, the state (AUTO mode) 30B of the processor 2, and the detected name of the observed part (vestibular part) are displayed as "detected part" 30C. Further, it is graphically displayed using a figure 30D indicating the stomach that the CAD during startup is the gastric CAD. Note that in the case of the manual mode where the state of the processor 2 has the AUTO mode "off", the CAD corresponding to the organ specified by the user is activated.

[0042] Although not shown, when the confidence level PP2 of the lesion tissue is the first confidence level P1, a lesion candidate may be superimposed on the endoscope image 35A, or the confidence level PP2 may be displayed as a numerical value or color. For example, a lesion candidate with a confidence level PP2 of 0.7 or more may be superimposed on the endoscope image 35A in red. Conversely, when the confidence level PP2 of the lesion tissue is less than a predetermined threshold, the display by the lesion estimation unit 20 is not performed.

[0043] The endoscope image diagnosis support processor 2 can stably identify the observed organ during endoscope observation.

[0044] Note that "organ CAD" may include "boundary CAD" that performs lesion detection processing on images including a plurality of organs. For example, when the confidence levels of site 4 (esophagus) and site 5 (esophagogastric junction) are both equal to or higher than a predetermined confidence level, a boundary organ including the esophagus and the stomach is specified, and the corresponding boundary CAD may infer the presence or absence of a lesion in the endoscopic image.

[0045] As described above, the operation method of the endoscopic image diagnosis support processor according to the embodiment of the present invention is to estimate the normal confidence level and the lesion confidence level of each of the plurality of endoscopic images for each of the plurality of organs by using a plurality of models each learned with a plurality of lesion images and a plurality of normal images as teacher data for the plurality of continuously input endoscopic images, and to identify the observed organ of the endoscopic image based on a predetermined condition using the normal confidence level and the lesion confidence level of each of the plurality of endoscopic images.

[0046] The program of the endoscopic image diagnosis support processor according to the embodiment of the present invention causes a computer to execute a process of estimating the normal confidence level and the lesion confidence level of each of the plurality of endoscopic images by using a plurality of models each learned with a plurality of lesion images and a plurality of normal images as teacher data for each of the plurality of organs for the plurality of continuously input endoscopic images, and identifying the observed organ of the endoscopic image based on a predetermined condition using the normal confidence level and the lesion confidence level of each of the plurality of endoscopic images.

[0047] It may be a part of the program that is stored in the storage device 32 which is a non-temporary storage medium. Also, the program may be distributed or provided via a communication network. The user can execute all or part of the processing and execute the processing of the above-described processor by installing the program from the storage medium into the computer or by downloading the program via the communication network and installing it into the computer.

[0048] <Modifications of the First Embodiment> The processors 2A-2D of the modified embodiments and embodiments described below are similar to processor 2 and have the same effects. For this reason, in the following, components with the same function as processor 2 are denoted by the same reference numerals, and their illustration and description are omitted.

[0049] <Modification 1 of the First Embodiment> As shown in the flowchart of Figure 4, in the processor 2A of this modification, the organ identification unit 15 provisionally identifies the organ with the highest first confidence level P1 among the normal confidence level PP1 and the pathological confidence level PP2 as a provisional observation organ (step S31). Then, if the first confidence level P1 is greater than or equal to a predetermined first judgment value TP1 (step S32: YES), the organ with the first confidence level P1 is identified as an observation organ (step S40).

[0050]

[0051] For example, in the example shown in Table 3 above, the first judgment value TP1 is 0.6. The first judgment value TP1 may be the same for all organs, or it may be different for each organ.

[0052] In frame 2, the highest confidence level P1 is the normal confidence level PP1(5) for gastric CAD. Therefore, in step S31, the stomach is identified as a provisional observation organ. However, the normal confidence level (PP1(5) = 0.5) is less than the first judgment value TP1(0.6) (step S32: NO). Therefore, no new organ identification for observation is performed, and the display device 30 displays the esophagus, which was identified as the observation organ by the organ identification unit 15 in frame 1, as the observation organ.

[0053] In frame 3, the normality confidence score PP1(5) for the stomach, which has been identified as a provisional observation organ, is 0.7. The normality confidence score (PP1(5) = 0.7) is greater than or equal to the first judgment value TP1(0.6) (Step S32: YES). Therefore, the stomach is identified as an observation organ.

[0054] The endoscopic image diagnostic support processor 2A can identify the organ being observed with the endoscope more reliably than the endoscopic image diagnostic support processor 2.

[0055] <Modification 2 of the First Embodiment> As shown in the flowchart of Figure 5, in the processor 2B of this modification, the organ identification unit 15 identifies the organ with the first confidence level P1 as the organ to be observed when the organ with the highest first confidence level P1, where either the normal confidence level PP1 or the pathological confidence level PP2 is the highest, is observed consecutively for a number of times equal to or greater than a predetermined second determination value TC2 (step S36: YES).

[0056]

[0057] For example, in the example shown in Table 4 above, the second judgment value TC2 is 2. The second judgment value TC2 may be the same for all organs, or it may be different for each organ. The count C is initialized (C=0) at startup.

[0058] In frame 2, the highest confidence level P1 is the normal confidence level PP1 (5) for gastric CAD. Therefore, in step S31, the stomach is identified as a provisional observation organ. If the organ determined to be a provisional observation organ is different from the organ determined to be a provisional observation organ in the previous frame (step S33: NO), the count C is initialized. For example, since the organ determined to be a provisional observation organ in frame 1 is the duodenum (step S33: NO), the count C is initialized in step S34.

[0059] In step S35, 1 is added to the count C. For example, in frame 2, the count C is 1.

[0060] In frame 2, in step S36, the count C (C=1) is less than the second determination value TC2 (TC2=2) (NO). Therefore, no new organ identification of the observed organ is performed, and the display device 30 displays the duodenum, which was identified by the organ identification unit 15 in frame 1, as the observed organ.

[0061] In frame 3, the highest confidence level P1 is the normal confidence level PP1 (5) (0.6) for stomach CAD. Therefore, in step S31, the stomach is again identified as a provisional observation organ. In step S32, the identified organ (stomach) is the same as the organ (stomach) that was determined to be a provisional observation organ in frame 2 (YES) in step S33. In step S35, 1 is added to the count C, so the count C becomes 2.

[0062] In frame 3, in step S36, the count C (C=2) is greater than or equal to the second determination value TC2 (TC2=2) (YES). Therefore, in step S40, the provisional observation organ (stomach) is identified as the observation organ, and in step S50, the display device 30 displays the stomach as the observation organ.

[0063] The endoscopic image diagnostic support processor 2B can identify the organ being observed with the endoscope more reliably than the endoscopic image diagnostic support processor 2.

[0064] <Modification 3 of the First Embodiment> In this modification, the endoscopic image diagnostic support processor 2C identifies an organ with a first confidence level P1 as an observed organ when a provisionally observed organ with a maximum first confidence level P1 equal to or greater than the first judgment value TP1 occurs consecutively for a number of times equal to or greater than the second judgment value TC2.

[0065]

[0066] For example, in the example shown in Table 5 above, the first judgment value TP1 is 0.6, and the second judgment value TC2 is 2. The first judgment value TP1 and the second judgment value TC2 may be the same for all organs, or they may be different for each organ.

[0067] In frame 2, the highest first confidence level P1 is the normal confidence level PP1 (5) for the stomach, but the normal confidence level PP1 is less than the first judgment value TP1, and 1 is not added to the count C. For this reason, in frame 3 as well, the stomach is not identified as an observed organ. In frame 4, the stomach is identified as an observed organ.

[0068] The endoscopic image diagnostic support processor 2C can identify the organ being observed during endoscopy more reliably than the endoscopic image diagnostic support processors 2A and 2B.

[0069] <Third Embodiment> As shown in the flowchart of Figure 6, the endoscopic image diagnostic support processor 2D of this embodiment infers the normal confidence level PP1 and the lesion confidence level PP2 for each organ (step S20). If the organ with the highest confidence level is the same organ as in the previous frame (step 22 YES), the cumulative normal confidence level SPP1, which is the accumulation of the normal confidence level PP1, and the cumulative lesion confidence level SPP2, which is the accumulation of the lesion confidence level PP2, are obtained (step S24). If the organ with the highest confidence level is a different organ from the previous frame (step 22 NO), after all the cumulative confidence level values ​​have been initialized (step S23), SPP1 and SPP2 are obtained in step S24. Then, if either of the cumulative confidence level values ​​SPP (cumulative normal confidence level SPP1 and cumulative lesion confidence level SPP2) is greater than or equal to a predetermined third judgment value TSPP (step S25: YES), the corresponding organ is identified as the organ to be observed (step S40).

[0070]

[0071] For example, in the example shown in Table 6 above, the third judgment value TSPP is 2.0. The third judgment value TSPP may be the same for all organs, or it may be different for each organ. The cumulative confidence value SPP is initialized at startup (SPP = 0). In frames 1 and 2, it is assumed that the duodenal CAD is in the running state.

[0072] In frame 3, the organ with the highest confidence level changes from the duodenum to the stomach, so all SPP1 and SPP2 are reset (Figure 6: step S23), and the accumulation of PP1 and PP2 restarts (step S24). In frame 5, the accumulated confidence level for the stomach becomes equal to or greater than the third judgment value TSPP (step S25: YES), so the stomach is identified as the organ to be observed (step S40).

[0073] Endoscopic image diagnostic support processor 2D can identify the observed organ during endoscopy more reliably than endoscopic image diagnostic support processor 2-2C.

[0074] The range of values ​​described above is not limited to that range and can be increased or decreased as appropriate. Furthermore, the present invention is not limited to the embodiments described above, and various changes and modifications can be made as long as they do not alter the essence of the invention.

[0075] 1... Endoscope system 2, 2A-2D... Endoscopic image diagnostic support processor (processor) 7... Operation buttons 8... Camera unit 9... Endoscope 11... CPU 12... Image input unit 15... Organ identification unit 16... Memory 21-26... Second model 29... Display control unit 30... Display device 31... Notification unit 32... Storage device

Claims

1. An endoscopic image diagnostic support processor comprising: a lesion estimation unit that estimates the normal confidence level and lesion confidence level for each of the multiple endoscopic images that are input sequentially, using multiple models each that have been trained using multiple lesion images and multiple normal images for each of the multiple organs as training data; and an organ identification unit that identifies the organ being observed in the endoscopic image based on predetermined conditions using the normal confidence level and lesion confidence level for each of the multiple endoscopic images.

2. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit identifies the organ with the highest first confidence level among the normal confidence level and the lesion confidence level as the organ to be observed.

3. The endoscopic image diagnostic support processor according to claim 1, further comprising a notification unit that notifies the observed organ identified by the organ identification unit.

4. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit identifies the organ with the first confidence level as the observed organ when the maximum first confidence level among the normal confidence level and the lesion confidence level is equal to or greater than a predetermined first determination value.

5. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit identifies the organ with the first confidence level as the observed organ when the organ with the highest first confidence level, which is either the normal confidence level or the lesion confidence level, occurs consecutively for a predetermined number of times equal to or greater than a second determination value.

6. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit identifies the organ with the first confidence level as the observed organ when the organ with the highest first confidence level, where either the normal confidence level or the lesion confidence level is the highest, is equal to or greater than a predetermined first judgment value and occurs consecutively for a number of times equal to or greater than a predetermined second judgment value.

7. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit identifies an organ as the observed organ if either the cumulative normal confidence value obtained by accumulating the normal confidence values ​​of each of the plurality of organs, or the cumulative lesion confidence value obtained by accumulating the lesion confidence values ​​of each of the organs, is equal to or greater than a predetermined third determination value.

8. A method for operating an endoscopic image diagnostic support processor that, for multiple endoscopic images input consecutively, estimates the normal confidence level and the lesion confidence level for each of the multiple endoscopic images using multiple models, each of which has been trained using multiple lesion images and multiple normal images for each of the multiple organs as training data, and identifies the organ being observed in the endoscopic image based on predetermined conditions using the normal confidence level and the lesion confidence level for each of the multiple endoscopic images.

9. A program for an endoscopic image diagnostic support processor that, for multiple endoscopic images input in succession, estimates the normal confidence level and the pathological confidence level for each of the multiple endoscopic images using multiple models, each of which has been trained using multiple pathological images and multiple normal images for each of the multiple organs as training data, and causes a computer to perform a process to identify the observed organ in the endoscopic image based on predetermined conditions using the normal confidence level and the pathological confidence level for each of the multiple endoscopic images.

Images