Endoscopic image diagnosis support processor, operation method for endoscopic image diagnosis support processor, and program for endoscopic image diagnosis support processor

Abstract

[Problem] To provide an endoscopic image diagnosis support processor 2 for stably identifying an observation organ under observation by an endoscope 9. [Solution] The endoscopic image diagnosis support processor 2 includes an observation organ determination device 10, the observation organ determination device 10 comprising: a site inference section 13 for inferring, for each endoscopic image, a confidence level P indicating that the endoscopic image is an image of each of a plurality of sites, using a first model trained using a plurality of images of the plurality of sites as training data; a site identification unit 14 for identifying, from among the plurality of sites, an observation site that is a site of the endoscopic image, on the basis of a predetermined condition using the confidence level P; and an organ identification unit 15 for identifying an observation organ corresponding to the observation site.

Description

Endoscopic Image Diagnosis Support Processor, Operating Method of Endoscopic Image Diagnosis Support Processor, and Program of Endoscopic Image Diagnosis Support Processor

[0001] Embodiments of the present invention relate to an endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed, an operating method of the endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed, and a program of the endoscopic image diagnosis support processor capable of efficiently identifying an organ being observed.

[0002] Computer-aided detection (CADe: Computer-Aided Detection) indicating the position of a lesion candidate and computer-aided diagnosis (CADx: Computer-Aided Diagnosis) indicating discrimination information of the lesion candidate are known for an image captured using an endoscope (hereinafter, CADe and CADx are collectively referred to as "CAD"). In CAD, for example, a machine learning model obtained by performing deep learning or the like using a plurality of images as teacher data is used.

[0003] International Publication No. WO2022 / 181748 discloses a technique for generating a plurality of models each adapted to a plurality of organs by learning using images of each of the plurality of organs. For example, a plurality of types of models such as a pharyngeal model learned using only pharyngeal images and an esophageal model learned using only esophageal images are created. Then, by using a model adapted to the organ being observed by the endoscope from among the plurality of types of models, the accuracy as CAD can be increased.

[0004] Japanese Patent Application Laid-Open No. 2007-151809 discloses a receiving device that detects that the organ being imaged by a capsule endoscope has changed and receives data from the capsule endoscope.

[0005] However, it is not easy to identify the organ being observed by the endoscope when the imaging conditions are not good, and in some cases, there is a risk that an incorrect organ may be identified. Also, even when the imaging conditions are good, there are cases where the organ being observed cannot be correctly identified 100%.

[0006] International Publication WO2022 / 181748, Japanese Patent Publication No. 2007-151809

[0007] Embodiments of the present invention aim to provide an endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, a method for operating the endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, and a program for the endoscopic image diagnostic support processor that allows a computer to reliably identify an organ being observed with an endoscope.

[0008] The endoscopic image diagnostic support processor according to an embodiment of the present invention comprises: a site inference unit that, with respect to a plurality of endoscopic images that are input sequentially, uses a first model trained on a plurality of images of a plurality of sites as training data to infer the confidence level of each endoscopic image being an image of the plurality of sites; a site identification unit that identifies the observation site of the endoscopic image based on predetermined conditions using the confidence level; and an organ identification unit that identifies the observation organ corresponding to the observation site.

[0009] The operation method of the endoscopic image diagnostic support processor according to an embodiment of the present invention involves using a first model, which has been trained on multiple images of multiple body parts as training data, to infer the confidence level of each endoscopic image being an image of one of the multiple body parts, and then, from among the multiple body parts, identifying the observation site that is the body part of the endoscopic image based on predetermined conditions using the confidence level, and identifying the observation organ corresponding to the observation site.

[0010] The program for the endoscopic image diagnostic support processor according to an embodiment of the present invention uses a first model, which has been trained on multiple images of multiple body parts as training data, to infer the confidence level of each endoscopic image being an image of one of the multiple body parts from a plurality of endoscopic images that are input sequentially. The program then causes the computer to perform a process to identify the observation site that is the body part of the endoscopic image from among the multiple body parts, based on predetermined conditions using the confidence level, and to identify the observation organ corresponding to the observation site.

[0011] According to embodiments of the present invention, an endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, a method for operating the endoscopic image diagnostic support processor that reliably identifies an organ being observed with an endoscope, and a program for the endoscopic image diagnostic support processor that allows a computer to reliably identify an organ being observed with an endoscope are provided.

[0012] Figure 1 is a configuration diagram of the endoscopic image diagnostic support processor of the embodiment. Figure 2 is a flowchart of the operation method of the endoscopic image diagnostic support processor of the first embodiment. Figure 3A is an example of the screen of the display device of the embodiment. Figure 3B is an example of the screen of the display device of the embodiment. Figure 4 is a diagram for explaining the operation method of the endoscopic image diagnostic support processor of the first embodiment. Figure 5 is a flowchart of the operation method of the endoscopic image diagnostic support processor of modification 1 of the first embodiment. Figure 6 is a flowchart of the operation method of the endoscopic image diagnostic support processor of modification 1 of the first embodiment. Figure 7 is a flowchart of the operation method of the endoscopic image diagnostic support processor of the second embodiment. Figure 8 is an example of the screen of the display device of the second embodiment. Figure 9 is a flowchart of the operation method of the endoscopic image diagnostic support processor of the third embodiment. Figure 10 is a diagram for explaining the operation method of the endoscopic image diagnostic support processor of the fourth embodiment.

[0013] Embodiments of the present invention will be described below with reference to the drawings. The drawings based on the embodiments are schematic. The illustration and reference numerals of some components have been omitted.

[0014] <First Embodiment> As shown in Figure 1, the endoscopic image diagnostic support processor 2 (hereinafter referred to as "processor 2") of this embodiment, together with the endoscope 9, display device 30, notification unit 31, and storage device 32, constitutes the endoscopic system 1.

[0015] The camera unit 8 of the endoscope 9 captures images of the inside of the subject's body. The endoscope 9 outputs image data at a predetermined frame rate. The user can capture still images, for example, by operating the operation buttons 7 on the control panel (not shown) of the endoscope 9. The endoscope 9 may be either a flexible or rigid endoscope.

[0016] The processor 2 includes an observed organ determination device 10 (hereinafter referred to as "determination device 10") and a lesion detection device 20.

[0017] The determination device 10 includes a CPU 11, an image input unit 12, a site inference unit 13, a site identification unit 14, an organ identification unit 15, and a memory 16. The processor 2 is also connected to the server 40.

[0018] The CPU 11 is a system controller that controls the entire processor 2. The image input unit 12 processes images taken by the endoscope 9 at a predetermined frame rate. Alternatively, the image input unit 12 may receive endoscope images processed by an image processing processor (not shown) or a server 40 from the image data from the endoscope 9.

[0019] The site inference unit 13 is a first AI processing unit that uses a first model, which has been trained using multiple images of multiple sites as training data, to infer the confidence level of each of the multiple sites in the endoscopic image input from the endoscope. The site identification unit 14 identifies the observation site, which is a site in the endoscopic image, from among the multiple sites based on predetermined conditions using the confidence level P.

[0020] The organ identification unit 15 identifies the organ to be observed corresponding to the observation site. The memory 16 stores various data, such as the correspondence between the observation site and the observed organ.

[0021] The lesion detection device 20 is a second AI processing unit that infers lesions in images using multiple second models that have been trained using multiple lesion images for multiple organs as training data.

[0022] The lesion detection device 20 includes a nasal cavity CAD (second model 1) 21, an oral cavity CAD (second model 2) 22, a pharyngeal CAD (second model 3) 23, an esophageal CAD (second model 4) 24, a gastric CAD (second model 5) 25, a duodenal CAD (second model 6) 26, and an ON / OFF control unit 29 for controlling the activation / deactivation of each model. Each model may also have its own ON / OFF control unit.

[0023] Furthermore, at least one of the multiple configurations of the processor 2 may consist of internal circuits of semiconductor elements processed by software, or dedicated hardware circuits, or it may include both internal circuits of semiconductor elements and dedicated hardware circuits. At least one of the functional parts of the processor 2 may also be a component of the server 40. For example, the lesion detection device 20 may be an operating part of the server 40.

[0024] For example, the body part inference unit 13 may include an FPGA (Field Programmable Gate Array) with hardware circuits and a GPU (Graphics Processing Unit) with software circuits. Also, the body part identification unit 14, organ identification unit 15, etc., may be part of a CPU 11 that processes data using software.

[0025] The display device 30, which displays the endoscopic image, is a monitor such as a liquid crystal display. The notification unit 31 informs the user of information from the processor 2 using sound, light, etc. The notification unit 31 may be part of the display device 30.

[0026] The storage device 32 is a non-temporary storage device (e.g., a magnetic disk, optical disk, or HDD) that stores the program of the processor 2. The storage device 32 may also be configured as a server 40 connected to the processor 2 via an internet connection or the like.

[0027] In processor 2, the determination device 10 can efficiently and stably identify the organ being observed in the endoscopic image. The second CAD model of the lesion detection device 20 is created for each of the multiple organs. As a result, processor 2 has good model learning efficiency and high inference reliability.

[0028] <Processor Operation Method> The operation method of processor 2 will be explained according to the flowchart in Figure 2.

[0029] <Step S10> Image Input The endoscopic system 1 of this embodiment performs an examination of the upper gastrointestinal tract. The insertion part of the endoscope 9 is inserted through the oral cavity or nasal cavity of the subject and enters the pharynx. For example, multiple endoscopic images from the endoscope 9 or the image processing processor are input to the image input unit 12 of the processor 2 in succession at a predetermined frame rate. Processing is not performed if no images are input.

[0030] The image input unit 12 performs image processing on images directly input from the endoscope 9 (for example, RAW images) for display on the display device 30. If an image processed by an image processing processor is input, no image processing is required.

[0031] <Step S20> Site Inference The site inference unit 13 is a first AI processing unit that uses a first model to infer the confidence level PN (N=1 to n) that the input endoscopic image is an image of one of n different sites. The value of the confidence level P is in the range of (0 to 1). (Confidence level P=0) means a probability of 0%, and (Confidence level P=1) means a probability of 100%.

[0032] Furthermore, the region inference unit 13 does not need to infer all images that are continuously input at a predetermined frame rate. For example, the region inference unit 13 processes only 20 fps (20 frames / second) image data from 120 fps (120 frames / second) image data.

[0033] The machine learning model in the AI ​​processing unit is a deep neural network (DNN) that performs deep learning with multiple hidden layers. The model can also be a convolutional neural network (CNN), R-CNN (Regions with CNN features) which utilizes CNN, or FCN (Fully Convolutional Networks), etc.

[0034] The first model is trained using images of n different parts of the upper digestive tract as training data. As shown in Table 1 below, in this embodiment, n = 15.

[0035]

[0036] Each of the 15 body parts corresponds to one of the m types of organs. As shown in Table 2 below, in this embodiment, m = 6.

[0037]

[0038] The number of types of body parts n is, for example, between 2 and 100. The number of types of organs m is (m < n), for example, between 2 and 20.

[0039] The site inference unit 13 estimates the probability (confidence level P1 to P15) that the endoscopic image represents one of 15 different sites. For example, as shown in the upper part of Figure 4, the site inference unit 13 outputs an inference result consisting of 15 confidence levels (P1 to P15), such as a confidence level P12 of 0.1 for site 12 (antrum) and a confidence level P14 of 0.6 for site 14 (duodenal bulb).

[0040] <Maximum confidence level PX for site X> Out of 15 types of sites (sites 1 to 15), site X (confidence level PX) is the site with the highest confidence level P.

[0041] <Step S30> Is the corresponding second model running? The CPU 11, for example, determines whether the second model Y corresponding to part X is running. For example, if part X is part 9 (mid-body), the corresponding organ is (organ 5: stomach). The second model Y corresponding to (organ 5: stomach) is second model 5. If the running second model is second model 5 (YES), the process proceeds to step S65. That is, the running second model 5 continues to be used.

[0042] <Step S35> (Count) C = 0 (excluding CX) The organ identification unit 15 identifies the observed organ corresponding to the observed site if the observed site X with the highest confidence level P is the same site consecutively for a number of times equal to or greater than the first determination value TC1. For this purpose, the organ identification unit 15 performs a counting process for the number of detections. Step S35 is the count initialization process.

[0043] The count numbers C1 to C15 indicate the number of detections corresponding to each of parts 1 to 15. The count numbers C1 to C15 stored in the memory 16 are initialized, for example, at startup.

[0044] When the second model corresponding to the observed organ is not in operation (S30: NO), the count numbers C1 to C15 set for each of parts 1 to 15 are initialized (C = 0). At this time, the count number CX corresponding to the part X with the maximum confidence level P is not initialized but retained.

[0045] <Step S40> CX = CX + 1 One is added to the count number CX of part X.

[0046] <Step S45> CX ≥ TC1 "TC1" indicates a first determination value for determining the number of times part X has been the same consecutive part. The first determination value TC1 is, for example, 1 or more and 10 or less. The first determination value TC1 may be the same value for all parts, or may be different for each part or for each corresponding organ. For example, the first determination value TC1 for the part of the stomach is set to be less than the first determination value TC1 for the esophagus or duodenum.

[0047] It is determined whether the count number CX of part X is equal to or greater than the first determination value TC1. If the count number CX is less than the first determination value TC1 (NO), the process proceeds to step S65. That is, the second model in operation is continuously used.

[0048] <Step S50> When the part X with the maximum specific confidence level P of part X is the same continuously for a first determination value TC1 or more, that is, when the count number CX is equal to or greater than the first determination value TC1 (S45: YES), part X is specified as the observed part during observation.

[0049] That is, the part specifying unit 14 specifies an observed part that is a part of the endoscopic image based on a predetermined condition using the confidence level P from among a plurality of parts.

[0050] <Step S55> Specific Organ The specific organ specifying unit 15 for the corresponding organ specifies, for example, an observed organ Y corresponding to the observed part X using a correspondence table (Table 2) stored in the memory 16.

[0051] <Step S60> Activation of the corresponding second model (ON) The ON / OFF control unit 29 of the lesion detection device 20 activates the CAD (second model Y) corresponding to the observed organ Y. The ON / OFF control unit 29 also stops (OFF) all CADs other than (second model Y).

[0052] <Step S65> Lesion detection processing The activated CAD of the lesion detection device 20 infers the lesion from the endoscopic image. For example, the activated gastric CAD 30 performs inference using the second model 5. The inference result is output as a confidence level PP (PP = 0 to 1) inferred by the CAD corresponding to the observed organ Y.

[0053] For example, the confidence level PP1 for normal tissue is 0.66, and the confidence level PP2 for diseased tissue is 0.34 (PP1 + PP2 = 1.0). The lesion detection device 20 may output either confidence level PP1 or confidence level PP2.

[0054] Furthermore, when processor 2 is started, i.e., at the start of the examination, there are no organs identified as being observed, and therefore no CAD is running. For this reason, lesion detection processing is not performed.

[0055] <Step S70> The display device 30 displays the observation site identified by the site identification unit 14, and / or the observation organ identified by the organ identification unit 15, as numerical values, illustrations, colors, etc. The display device 30 may also serve as a notification unit that notifies the observed organ.

[0056] Figures 3A and 3B show examples of the screen of the display device 30. The screen in Figure 3A displays the endoscopic image 30A, the status of the determination device 10 (AUTO mode) 30B, and the name of the detected observation area (antrum) as "Detected Area" 30C. Furthermore, the fact that the CAD currently in operation is (stomach) is graphically displayed using a figure 30D representing a stomach.

[0057] In the screen shown in Figure 3B, the "detection site" 30B is the duodenal bulb, but since the count CX is less than the first judgment value TC1, the CAD that is running is graphically displayed as (stomach).

[0058] Although not shown in the diagram, if the confidence level PP2 for identifying lesion tissue is above a predetermined threshold, the lesion candidate may be superimposed on the endoscopic image 35A, or the confidence level PP2 may be displayed numerically or in color. For example, lesion candidates with a confidence level PP2 of 0.7 or higher may be superimposed on the endoscopic image 35A in red. Conversely, if the confidence level PP2 for identifying lesion tissue is below a predetermined threshold, no display is made by the lesion detection device 20. Then, the process from step S10 is repeated.

[0059] As described above, in the determination device 10 of this embodiment, the part identification unit 14 identifies a part as an observation part when the part with the highest confidence level P is the same part for a continuous period of time equal to or greater than the first determination value TC1.

[0060] The operation method of the determination device 10 will be specifically explained using Figure 4. In this embodiment, the first determination value TC1 is set for each site. The first determination value TC1(N) is the determination value for site N. Specifically, the first determination value TC1(12) for the antrum (site 12) and the first determination value TC1(14) for the duodenal bulb (site 14) are set to "2".

[0061] <Image F> Assume that the duodenal CAD (second model 6) of the lesion detection device 20 is running. Also, the count C14 (not displayed) of the duodenal bulb (region 14) after processing of Image F is set to "1".

[0062] <Image F+1> The region X, which has the highest confidence level P, is the duodenal bulb (region 14) for the second time in a row. The duodenal CAD (second model 6) corresponding to the duodenal bulb (region 14) is already running (S30: YES). Therefore, the duodenal CAD (second model 6) of the lesion detection device 20, which is running, infers the lesion from the endoscopic image (S65).

[0063] <Image F+2> The region X, which has the highest confidence level P, is again the duodenal bulb (region 14). Therefore, the duodenal CAD (second model 6) of the active lesion detection device 20 infers the lesion from the endoscopic image (S65).

[0064] <Image F+3> The region X with the highest confidence level P is the antrum (region 12). Since the gastric CAD (second model 5) corresponding to the identified stomach (organ 5) is not activated (S30: NO), 1 is added to the count C12 of the antrum (region 12) (S40). Counts C other than count C12 are reset to zero.

[0065] Since the count C(12) is less than the first judgment value TC1(12) (S45: NO), the duodenal CAD (second model 6) of the active lesion detection device 20 infers the lesion from the endoscopic image (S65).

[0066] <Image F+4> The region X, which has the highest confidence level P, is the antrum (region 12) for the second time in a row. Since the gastric CAD (second model 5) corresponding to the identified organ, the stomach (organ 5), is not activated (S30), 1 is added to the count C (12) (S40), and the count C (12) becomes "2".

[0067] Since the count C(12) is greater than or equal to the first judgment value TC1(12) (S45: YES), the antrum (part 12) is identified as the observation site (S50). Then, the stomach (organ 5), which is the organ corresponding to the antrum (part 12), is identified as the observation organ (S55). Then, the gastric CAD (second model 5) corresponding to the stomach (organ 5) is activated (S65), and lesion detection processing is performed using the gastric CAD (second model 5) (S65).

[0068] <Image F+5> The region X with the highest confidence level P is the duodenal bulb (region 14). Since the duodenal CAD (second model 6) corresponding to the duodenal bulb (region 14) is not activated (S30: NO), 1 is added to the count C (14) (S40). Since the count C14 is less than the first judgment value TC1 (14) (S45: NO), the gastric CAD (second model 5) of the activated lesion detection device 20 infers the lesion in the endoscopic image (S65).

[0069] Furthermore, the "organ CAD" may include a "boundary CAD" that performs lesion detection processing on images containing multiple organs. For example, if the confidence level of site 4 (esophagus) and the confidence level of site 5 (esophagogastric junction) are both above a predetermined confidence level P, the boundary organ including the esophagus and stomach may be identified, and the corresponding boundary CAD may infer lesions in the endoscopic image.

[0070] As described above, the processor of this embodiment operates by using a first model, which has been trained on multiple images of multiple body parts as training data, to infer the confidence level of each endoscopic image being an image of one of the multiple body parts, and then, from among the multiple body parts, identifying the observation site that is the body part of the endoscopic image based on predetermined conditions using the confidence level, and identifying the observation organ corresponding to the observation site.

[0071] The processor program of this embodiment uses a first model, which has been trained using multiple images of multiple body parts as training data, to infer the confidence level of each endoscopic image being an image of one of the multiple body parts from a plurality of endoscopic images that are input sequentially. The program then causes the computer to perform the following processes: identify the observation site that is the body part of the endoscopic image from among the multiple body parts based on predetermined conditions using the confidence level, and identify the observation organ corresponding to the observation site.

[0072] The non-temporary storage medium, the storage device 32, may store only a part of the program. Furthermore, the program may be distributed or provided via a communication network. Users can execute all or part of the processing, and thus perform the processing of the processor described above, by installing the program from the storage medium to their computer, or by downloading the program via the communication network and installing it to their computer.

[0073] Furthermore, even if the observation site inferred by the site inference unit 13 is different, if the organ corresponding to the observation site is the same for several consecutive times, the organ identification unit 15 may identify the corresponding organ as the observed organ. In other words, after the organ identification unit 15 has provisionally determined the observed organ, if an organ different from the organ corresponding to the currently running CAD is provisionally observed for a certain number of consecutive times, it may finalize it as the observed organ.

[0074] In other words, the organ identification unit 15 provisionally determines the organ to be observed, and if a certain number of consecutive times the same organ is provisionally determined as the organ to be observed after a different organ from the immediately preceding endoscopic image has been provisionally determined, it makes a final decision to change the organ.

[0075] The organ identification unit 15 maintains the organ that was previously determined as the organ to be observed if, after an organ different from the immediately preceding endoscopic image has been provisionally determined as the organ to be observed, the same organ is not provisionally determined as the organ to be observed for a certain number of consecutive times.

[0076] <Modifications of the First Embodiment> The processor 2A-2E1 and determination device 10A-10E1 of the modified embodiments and embodiments described below are similar to the processor 2 and determination device 10 and have the same effects. For this reason, in the following, components with the same function as the processor 2 and determination device 10 are denoted by the same reference numerals, and are not shown or described. In the flowchart, the same processes are denoted by the same step numbers, and are not described.

[0077] <Modification 1 of the First Embodiment> In the processor 2A of this modification, the part identification unit 14 of the determination device 10A determines in step S31 whether the confidence level PX of the part X with the highest confidence level is equal to or greater than the second determination value TC2, as shown in the flowchart of Figure 5. The second determination value TC2 may be the same value for all parts, or it may be different for each part or organ.

[0078] If the confidence level PX is equal to or greater than the second determination value TC2 (S31: YES), processing corresponding to the count number CX of part X is performed in steps S35 to S70.

[0079] If the confidence level PX is less than the second judgment value TC2 (S31: NO), in step S65, lesion detection processing is performed using the second model that has already been activated.

[0080] The judgment device 10A of processor 2A is less likely to identify the wrong organ than the judgment device 10.

[0081] <Modification 2 of the First Embodiment> In the processor 2B of this modification, as shown in the flowchart of Figure 6, the part identification unit 14 of the determination device 10B determines in step S32 whether the confidence difference ΔP between the part X with the highest confidence P and the part X2 with the second highest confidence P is consecutively greater than or equal to the third determination value TC3.

[0082] The confidence difference ΔP represents the difference between the confidence level PX of site X, which has the highest confidence level P, and the confidence level PX2 of site X2, which has the second highest confidence level P. The confidence difference ΔP is determined by a third judgment value TC3. The third judgment value TC3 may be the same for all sites, or it may differ for each site or organ.

[0083] If the confidence difference ΔP is greater than or equal to the third judgment value TC3 (S32: YES), processing is performed in steps S35 to S70 according to the count CX of part X.

[0084] If the confidence difference ΔP is less than the third judgment value TC3 (S32: NO), in step S65, lesion detection processing is performed using the second model that has already been activated.

[0085] The judgment device 10B of processor 2B is less likely to identify the wrong organ than the judgment device 10.

[0086] The operation method of the part identification unit 14 may include steps S31 and S32. That is, in step S31 (the confidence level PX is equal to or greater than the second determination value TC2) and step S32 (the difference ΔP in confidence levels between the part X with the highest confidence level P and the part X2 with the second highest confidence level P is equal to or greater than the third determination value TC3), processing may be performed according to the count number CX of part X.

[0087] <Second Embodiment> In the processor 2C of this embodiment, the site inference unit 13 also infers a confidence level P3 that the endoscopic image is an inappropriate image. An inappropriate image is a so-called out-of-focus image, a blurred image, etc. For example, a third model that has been trained using inappropriate / appropriate images as training data outputs a confidence level P3, and if the confidence level P3 is greater than or equal to a predetermined value, the endoscopic image is inferred to be an inappropriate image.

[0088] Furthermore, the site inference unit 13 estimates a confidence level P4 for "undeterminable," meaning the endoscopic image is not one of the n types of sites. For example, a fourth model trained on undeterminable / determinable images as training data outputs a confidence level P4, and if the confidence level P4 is greater than or equal to a predetermined value, the endoscopic image is inferred to be an undeterminable image. The site inference unit 13 may also infer "undeterminable" if all of the confidence levels P for the n types of sites output by the first model are less than or equal to a predetermined value. The site inference unit 13 may be configured to infer (n+2) confidence levels for the n types of sites, inappropriate, and undeterminable states of the endoscopic image using a single model.

[0089] Figure 7 is a flowchart showing the operation method of processor 2C in this embodiment.

[0090] In step S20, if the endoscopic image is inferred to be inappropriate or undeterminable, the determination device 10A performs lesion detection processing in step S65 using the second model that has already been activated.

[0091] In the screen shown in Figure 8, the "detection site" is displayed as "undeterminable," and the second model (CAD) that is already running is graphically displayed as (stomach).

[0092] The determination device 10C of processor 2C can identify the observed organ more stably and efficiently than the determination device 10.

[0093] <Third Embodiment> In this embodiment, the processor 2D considers the confidence level PN of site N as the organ confidence level PY of organ Y corresponding to site N, and identifies organs that are tentatively determined to be observed (determined organs) if the first cumulative value SPY, which is the accumulation of confidence levels PY, is equal to or greater than the fourth determination value TC4. The operation method of the processor 2D will be explained below in accordance with the flowchart in Figure 9.

[0094] <Step S25> The organ identification unit 15 of the provisional organ identification determination device 10D identifies the organ Y corresponding to the site X with the highest confidence level P as the provisional observation organ.

[0095] <Step S26> Is the provisional organ Y different from the provisional organ in the previous frame? If the provisional organ Y is different from the provisional organ in the previous frame (S26: YES), in step S27, the cumulative confidence value of the provisional organ is reset (SPY = 0). If the provisional organ Y matches the provisional organ in the previous frame (S26: NO), in step S28, the confidence value of the provisional organ is accumulated (SPY = SPY + PY).

[0096] <Step S29> SPY ≥ TC4? If the cumulative confidence value SPY of the provisionally determined organ Y is greater than or equal to the fourth judgment value TC4 (S28: YES), the provisionally determined organ is officially determined as the organ to be observed, and lesion detection processing is performed using the model corresponding to the observed organ Y. If the cumulative confidence value of the provisionally determined organ Y is less than the judgment value (S28: NO), in step S65, lesion detection processing is performed using the second model that has already been activated. The fourth judgment value TC4 may be different for each organ.

[0097] The judgment device 10D of processor 2D is less likely to identify the wrong organ than the judgment device 10.

[0098] <Fourth Embodiment> In the processor 2E of this embodiment, the organ identification unit 15 of the determination device 10E considers the confidence levels PN of multiple sites N (N=1 to n) as the confidence level PM of the corresponding organ M (M=1 to m), and obtains a score SM corresponding to the confidence level PM. For example, the score SM stored in the memory 16 includes negative values, as shown in Table 3 below.

[0099]

[0100] The operation method of the processor 2E will be explained below according to the flowchart in Figure 10. <Step S31> Convert confidence level PN to score SM of corresponding organ The site identification unit 14 converts the confidence level PN (N=1 to n) of each site N to the score SM of the corresponding organ M, for example, using the correspondence table (Table 3) stored in memory 16. <Step S32> SSM = SSM + SM A second cumulative value SSM is calculated for each organ M. <Step S33> SSM ≥ TC6 If there are no organs whose second cumulative value SSM is equal to or greater than the sixth judgment value TS6 (NO), the second cumulative value SS is initialized in step S61 (SS=0) and then the processing from step S65 is carried out. If there is an organ whose second cumulative value SSM is equal to or greater than the sixth judgment value TS6 (YES), organ Y is identified as the organ under observation (step S55). <Step S59> If the second model Y corresponding to the identified organ Y is running (YES), the process from step S61 is performed. If the second model Y corresponding to the identified organ Y is not running (NO), the corresponding second model is started in step S60.

[0101] <Step S61> The cumulative value SS is corrected to a predetermined range. The second cumulative value SS is corrected if it exceeds a predetermined range, so that it falls within the predetermined range. That is, if the second cumulative value SS is less than a predetermined lower limit, it is corrected to the lower limit. If the second cumulative value SS is greater than a predetermined upper limit, it is corrected to the upper limit. For example, the lower limit of the second cumulative value SS is "-3", and the upper limit is "6".

[0102] The judgment device 10E of processor 2E is less likely to identify the wrong organ than the judgment device 10.

[0103] The range of values ​​described above is not limited to that range and can be increased or decreased as appropriate. Furthermore, the present invention is not limited to the embodiments described above, and various changes and modifications can be made as long as they do not alter the essence of the invention.

[0104] 1... Endoscope system 2, 2A-2E... Endoscopic image diagnostic support processor (processor) 7... Operation buttons 8... Camera unit 9... Endoscope 10, 10A-10E... Observed organ determination device (determination device) 11... CPU 12... Image input unit 13... Site inference unit 14... Site identification unit 15... Organ identification unit 16... Memory 20... Lesion detection device 21-26... Second model 29... ON / OFF control unit 30... Display device 31... Notification unit 32... Storage device

Claims

1. An endoscopic image diagnostic support processor comprising: a site inference unit that, in response to multiple endoscopic images input in succession, uses a first model trained on multiple images of multiple sites as training data to infer the confidence level of each endoscopic image being an image of one of the multiple sites; a site identification unit that identifies the observation site of the endoscopic image based on predetermined conditions using the confidence level; and an organ identification unit that identifies the observation organ corresponding to the observation site.

2. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit identifies the organ corresponding to the observation site as the observation organ when the site with the highest confidence level, or the organ corresponding to the site with the highest confidence level, is the same for a number of consecutive times equal to or greater than a first determination value.

3. The endoscope image diagnostic support processor according to claim 1, wherein the site identification unit identifies the site where the confidence level is the highest for a consecutive number of times equal to or greater than the first determination value as the observation site.

4. The endoscope image diagnostic support processor according to claim 1, wherein the site identification unit identifies as an observation site any site where the difference in confidence between the site with the highest confidence level and the site with the second highest confidence level is consecutively equal to or greater than a second determination value.

5. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit identifies the organ corresponding to the site with the highest confidence level as a provisionally determined organ, and identifies the provisionally determined organ as the observed organ if the first cumulative value obtained by accumulating the confidence levels of the sites with the highest confidence levels is equal to or greater than the third determination value.

6. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit considers the confidence level of the site corresponding to each of the multiple organs as the organ confidence level, obtains a score for each of the multiple organs corresponding to the respective organ confidence level, the score includes negative values, and identifies the site where the second cumulative value obtained by accumulating the scores is equal to or greater than the fourth determination value as the observed organ.

7. The endoscopic image diagnostic support processor according to claim 1, further comprising a notification unit that notifies the observed organ identified by the organ identification unit.

8. The endoscopic image diagnostic support processor according to claim 1, wherein the site inference unit infers a degree of confidence that the image is an inappropriate image and a degree of confidence that the image is an undeterminable image, and the site identification unit and the organ identification unit do not perform any processing when the site inference unit infers that the image is an inappropriate image or an undeterminable image.

9. The endoscopic image diagnostic support processor according to claim 1, further comprising a lesion detection device that infers lesions in an image using a plurality of second models that have been trained using a plurality of lesion images for a plurality of organs as training data, wherein the lesion detection device uses a second model from the plurality of second models that corresponds to the observed organ identified by the organ identification unit.

10. The endoscopic image diagnostic support processor according to claim 9, wherein the site inference unit infers the degree of confidence that the image is an inappropriate image and the degree of confidence that the image is an undeterminable image, and when the site inference unit infers that the image is an inappropriate image or an undeterminable image, the organ identification unit does not perform any processing, and the lesion detection device uses the second model corresponding to the observed organ identified immediately before by the organ identification unit.

11. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit makes a final decision to change the observed organ if, after an organ different from the organ in the immediately preceding endoscopic image has been provisionally determined as the observed organ, the same organ has been provisionally determined as the observed organ a number of times equal to or greater than a first determination value.

12. The endoscopic image diagnostic support processor according to claim 11, wherein, after an organ different from the organ in the immediately preceding endoscopic image has been provisionally determined as the observed organ, if the same organ is not provisionally determined as the observed organ for a number of consecutive times equal to or greater than the first determination value, the organ that was definitively determined immediately before is maintained as the observed organ.

13. The endoscopic image diagnostic support processor according to claim 12, further comprising a lesion detection device that infers lesions in an image using a plurality of second models that have been trained using a plurality of lesion images for a plurality of organs as training data, wherein the lesion detection device detects lesions corresponding to the observed organ identified by the organ identification unit.

14. The endoscopic image diagnostic support processor according to claim 12, further comprising: a notification unit that notifies the area identified by the area identification unit and the lesion detected by the lesion detection device.

15. The endoscopic image diagnostic support processor according to claim 1, wherein the organ identification unit maintains the previously determined organ if the site identification unit is unable to identify the observation site.

16. A method for operating an endoscopic image diagnostic support processor that, in response to multiple endoscopic images input in succession, uses a first model trained on multiple images of multiple body parts as training data to infer the confidence level of each endoscopic image being an image of one of the multiple body parts, identifies an observation site from among the multiple body parts that is the body part of the endoscopic image based on predetermined conditions using the confidence levels, and identifies the observation organ corresponding to the observation site.

17. A program for an endoscopic image diagnostic support processor that, in response to multiple endoscopic images input in succession, uses a first model trained on multiple images of multiple body parts as training data to infer the confidence level for each endoscopic image to be an image of one of the multiple body parts, identifies the observation site that is the body part of the endoscopic image from among the multiple body parts based on predetermined conditions using the confidence levels, and causes the computer to perform the process of identifying the observation organ corresponding to the observation site.

Images