Endoscopy assistance device, endoscopy assistance method, and recording medium

Abstract

Provided is an endoscopy assistance device, wherein object detection means detects an object included in an endoscopic image obtained by imaging the inside of the large intestine by means of an endoscopic camera. A position information acquisition means acquires position information indicating the position of the endoscopic camera in the large intestine. A display control means displays a schematic diagram having a shape imitating the intestinal tract of the large intestine and, when a predetermined instruction is given upon detection of the object, the display control means adds a mark to the position of the schematic diagram corresponding to the position of the endoscopic camera indicated by the position information. The endoscopy assistance device can be used to assist decision-making by a user conducting an endoscopy.

Description

Endoscope examination support device, endoscope examination support method, and recording medium

[0001] The present disclosure relates to a technology that can be used when presenting information for supporting an endoscope examination.

[0002] A technology for presenting information for supporting an endoscope examination is known.

[0003] Specifically, for example, Patent Document 1 discloses a perspective of determining whether it is an insertion step of inserting an endoscope to a folding point in a lumen or a removal step of removing the endoscope from the folding point in the lumen in an endoscope examination support device. Further, Patent Document 1 discloses a perspective of performing notification indicating that there is a lesion detected in the insertion step near the imaging unit of the endoscope in the removal step. Further, Patent Document 1 discloses a perspective of detecting a portion having a state different from a normal state on the appearance as a lesion by inputting an endoscope image into an identifier that has machine-learned the image features of the lesion.

[0004] International Publication No. WO2023 / 100310

[0005] According to the technology disclosed in Patent Document 1, in the insertion step, a judgment regarding the necessity of detailed observation for each lesion detected from the endoscope image is not made. Therefore, according to the technology disclosed in Patent Document 1, for example, there is a problem that a notification corresponding to a lesion that does not require detailed observation among each lesion detected in the insertion step may be made in the removal step.

[0006] One object of the present disclosure is to provide an endoscope examination support device capable of easily grasping where a portion determined to require detailed observation during an endoscope examination is.

[0007] In one aspect of this disclosure, the endoscopic examination support device includes object detection means for detecting objects included in endoscopic images obtained by imaging the inside of the large intestine with an endoscope camera, position information acquisition means for acquiring position information indicating the position of the endoscope camera inside the large intestine, and display control means for displaying a schematic diagram having a shape that mimics the intestinal tract of the large intestine, and when a predetermined instruction is given in conjunction with the detection of the object, adding a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the position information.

[0008] In another aspect of this disclosure, the endoscopic examination support method is a computer-operated endoscopic examination support method that detects an object included in an endoscopic image obtained by imaging the inside of the large intestine with an endoscopic camera, acquires positional information indicating the position of the endoscopic camera in the large intestine, displays a schematic diagram having a shape that mimics the intestinal tract of the large intestine, and adds a mark to the position of the schematic diagram corresponding to the position of the endoscopic camera indicated by the positional information when a predetermined instruction is given in conjunction with the detection of the object.

[0009] In yet another aspect of this disclosure, the recording medium records a program that causes a computer to perform a process of detecting an object contained in an endoscopic image obtained by imaging the inside of the large intestine with an endoscope camera, acquiring positional information indicating the position of the endoscope camera in the large intestine, displaying a schematic diagram having a shape that mimics the intestinal tract of the large intestine, and, when a predetermined instruction is given in conjunction with the detection of the object, adding a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the positional information.

[0010] According to this disclosure, it is possible to easily identify which areas require detailed observation during an endoscopic examination.

[0011] A diagram showing the general configuration of the endoscopic examination system relating to this disclosure. A block diagram showing an example of the hardware configuration of the endoscopic examination support device relating to this disclosure. A block diagram showing an example of the functional configuration of the endoscopic examination support device relating to this disclosure. A diagram showing an example of an endoscopic image. A diagram showing an example of a display image generated by the endoscopic examination support device relating to this disclosure. A diagram to explain a specific example of the processing related to the display of elements included in the sub-display area of ​​the display image. A diagram to explain a specific example of the processing related to the display of elements included in the sub-display area of ​​the display image. A diagram to explain a specific example of the processing related to the display of elements included in the sub-display area of ​​the display image. A flowchart showing an example of the processing performed in the endoscopic examination support device relating to this disclosure. A block diagram showing another example of the functional configuration of the endoscopic examination support device relating to this disclosure. A flowchart showing another example of the processing performed in the endoscopic examination support device relating to this disclosure.

[0012] Preferred embodiments of this disclosure will be described below with reference to the drawings.

[0013] <First Embodiment> [System Configuration] Figure 1 is a diagram showing the schematic configuration of an endoscopic examination system according to the present disclosure. As shown in Figure 1, the endoscopic examination system 100 comprises an endoscopic examination support device 1, a display device 2, and an endoscope scope 3 connected to the endoscopic examination support device 1.

[0014] The endoscopic examination support device 1 acquires video footage (hereinafter also referred to as endoscopic video) from the endoscope scope 3, which includes time-series images obtained by imaging the subject during the endoscopic examination, and displays the images on the display device 2 for confirmation by the user, such as a physician, performing the endoscopic examination. In addition, the endoscopic examination support device 1 acquires images of the inside of the large intestine obtained during the endoscopic examination from the endoscope scope 3 as endoscopic video.

[0015] In a colonoscopy, the user inserts the endoscope 3 from the anus to the deepest part near the appendix, and then performs a detailed observation while withdrawing the endoscope 3. In such a case, the user may not be able to identify the part of the colon that requires detailed observation during the period leading up to the insertion of the endoscope 3 into the deepest part of the colon when withdrawing the endoscope 3. Therefore, the endoscopy support device 1 detects objects included in the endoscopic image, acquires positional information indicating the position of the endoscope camera in the colon, displays a schematic diagram having a shape that mimics the intestinal tract of the colon on the display device 2, and when a predetermined instruction is given in conjunction with the detection of the object, a mark is added to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the positional information. As a result, according to this embodiment, it is possible to easily understand which part requires detailed observation during the endoscopy. Furthermore, the endoscopy support device 1 can be used to support the decision-making of the user performing the endoscopy.

[0016] The display device 2 includes, for example, a liquid crystal monitor. The display device 2 also displays display images, etc., output from the endoscopy support device 1. According to this disclosure, for example, processing of the endoscopy support device 1 may be performed in response to operations performed on a touch panel provided on the display device 2.

[0017] The endoscope scope 3 mainly comprises an operating unit 36 ​​for the user to input commands such as air insufflation, water insufflation, angle adjustment, and imaging instructions; a flexible shaft 37 that is inserted into the organ being examined in the patient; a tip 38 that incorporates an endoscope camera such as an image sensor; and a connection unit 39 for connecting to the endoscopy support device 1. The endoscope camera can obtain an endoscopic image by imaging the inside of the large intestine, which is a tubular organ. The endoscope camera can also obtain an endoscopic image by imaging the inside of the patient into which the shaft 37 of the endoscope scope 3 is inserted.

[0018] [Hardware Configuration] Figure 2 is a block diagram showing an example of the hardware configuration of an endoscopy support device according to the present disclosure. The endoscopy support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. Each of these elements is connected via a data bus 19.

[0019] The processor 11 executes predetermined processes by running programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer. The processor 11 also performs processes such as displaying a schematic diagram having a shape similar to the intestinal tract of the large intestine on the display device 2, and adding a mark to the position of the schematic diagram corresponding to the current position of the endoscope camera.

[0020] Memory 12 is composed of various volatile memories used as working memory, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memory that stores information necessary for processing the endoscopic examination support device 1. Memory 12 may also include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or it may include a storage medium such as a removable flash memory or disk medium. Memory 12 stores a program for the endoscopic examination support device 1 to execute each of the processes related to this disclosure. In addition, memory 12 temporarily stores a series of endoscopic images obtained by the endoscope scope 3 during an endoscopic examination, based on the control of the processor 11.

[0021] Interface 13 performs interface operations between the endoscopic examination support device 1 and external devices. For example, interface 13 supplies display images generated by processor 11 to display device 2. Interface 13 also supplies illumination light generated by light source unit 15 to endoscope scope 3. Interface 13 also supplies electrical signals indicating endoscopic images supplied from endoscope scope 3 to processor 11. Interface 13 also supplies endoscopic images extracted from endoscopic images to processor 11. Interface 13 may be a communication interface such as a network adapter for wired or wireless communication with external devices, or it may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc.

[0022] The input unit 14 generates an input signal corresponding to user operation. The input unit 14 includes at least one device, such as a button, touch panel, remote controller, foot switch, and voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope scope 3. The light source unit 15 may also incorporate a pump for supplying water or air to the endoscope scope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0023] DB17 stores endoscopic images and other data acquired from the patient's past endoscopic examinations. DB17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or it may include a storage medium such as a removable flash memory. Alternatively, instead of having DB17 within the endoscopic examination system 100, DB17 may be located on an external server, and relevant information may be obtained from the server via communication.

[0024] The endoscopic examination support device 1 may also be equipped with a sensor capable of measuring the rotation and translation of the endoscope camera, such as a magnetic sensor. Furthermore, the endoscopic examination support device 1 may have an endoscope insertion shape observation device that observes the shape of the shaft 37 inserted inside the patient, for example, based on the detection result of a magnetic field generated from a coil provided on the shaft 37 of the endoscope scope 3.

[0025] [Functional Configuration] Figure 3 is a block diagram showing an example of the functional configuration of an endoscopic examination support device according to the present disclosure. As shown in Figure 3, the endoscopic examination support device 1 includes an object detection unit 21, a position information acquisition unit 22, a trigger generation unit 23, an information storage unit 24, and a display control unit 25.

[0026] The object detection unit 21 functions as an object detection means. The object detection unit 21 also performs processing related to the detection of objects contained in the endoscopic image EG. For example, the object detection unit 21 performs threshold determination for each of the probabilities PP, KP, IP, and ZP described later, and obtains the detection result of objects contained in the endoscopic image EG based on the determination result of the threshold determination. The object detection unit 21 also generates object detection information BJ that shows the detection result of objects contained in the endoscopic image EG, and outputs the generated object detection information BJ to the display control unit 25. The object detection information BJ includes, for example, information indicating the type, position, and size of the object detected from the endoscopic image EG. The object detection unit 21 also includes a polyp detection unit 21A, a diverticulum detection unit 21B, an inflammation detection unit 21C, and a residue detection unit 21D.

[0027] The polyp detection unit 21A has an image recognition model PM that has been trained using images containing polyps. The polyp detection unit 21A also obtains the probability PP that a polyp is present as an object in the endoscopic image EG by inputting the endoscopic image EG into the image recognition model PM. In this disclosure, "image recognition model" may be replaced with "machine learning model".

[0028] The diverticulum detection unit 21B has an image recognition model KM that has been trained using images including diverticula. The diverticulum detection unit 21B also inputs an endoscopic image EG into the image recognition model KM to obtain the probability KP that a diverticulum is present as an object in the endoscopic image EG.

[0029] The inflammation detection unit 21C has an image recognition model IM that has been trained using images containing inflammation. The inflammation detection unit 21C also inputs an endoscopic image EG into the image recognition model IM to obtain the probability IP that inflammation is present as an object in the endoscopic image EG.

[0030] The residue detection unit 21D has an image recognition model ZM that has been trained using images containing residue. The residue detection unit 21D also inputs an endoscopic image EG into the image recognition model ZM to obtain the probability ZP that residue is present as an object in the endoscopic image EG.

[0031] According to the process described above, the object detection unit 21 can detect whether or not at least one of the four types of objects—polyps, diverticula, inflammation, and residues—is included in the endoscopic image EG. Furthermore, according to the process described above, the object detection unit 21 can detect objects included in the endoscopic image EG using a trained machine learning model.

[0032] Furthermore, according to this disclosure, the object detection unit 21 may, for example, use a model that integrates part or all of the image recognition models PM, KM, IM, and ZM to perform processing related to the detection of objects included in the endoscopic image EG.

[0033] The position information acquisition unit 22 functions as a means for acquiring position information. The position information acquisition unit 22 also has an identification model LM that has been learned using endoscopic images, etc. The identification model LM is learned to identify anatomical landmarks in the large intestine, such as the ileocecal valve and the splenic flexure. The position information acquisition unit 22 inputs the endoscopic image EG into the identification model LM and estimates the location in the large intestine where the endoscopic image EG was obtained based on the landmark identification result obtained, and acquires the estimated location as the current position of the tip 38 of the endoscope scope 3 in the large intestine. The position information acquisition unit 22 also outputs information indicating the current position of the tip 38 in the large intestine as position information LJ to the display control unit 25. In this disclosure, the current position of the tip 38 indicated by the position information LJ can be rephrased as the current position of the endoscope camera.

[0034] Furthermore, according to this disclosure, the position information acquisition unit 22 may acquire the current position of the tip 38 of the endoscope scope 3 in the large intestine based on, for example, the shape of the shaft 37 observed by the endoscope insertion shape observation device. Also according to this disclosure, the position information acquisition unit 22 may acquire the current position of the tip 38 of the endoscope scope 3 in the large intestine based on, for example, the landmark identification result obtained by processing using the identification model LM and the shape of the shaft 37 observed by the endoscope insertion shape observation device.

[0035] According to the process described above, the position information acquisition unit 22 can estimate the position within the large intestine where the endoscopic image EG was obtained based on the identification result of anatomical landmarks contained in the endoscopic image EG, and acquire the estimated position as the position of the endoscopic camera within the large intestine.

[0036] The trigger generation unit 23 determines whether the input signal NS generated by the input unit 14 or the operation unit 36 ​​is a signal indicating a predetermined instruction. If the trigger generation unit 23 determines that the input signal NS is a signal indicating a predetermined instruction, it generates a trigger signal TS and outputs the generated trigger signal TS to the display control unit 25.

[0037] The information storage unit 24 stores endoscopic examination information NJ for each patient undergoing endoscopic examination. The endoscopic examination information NJ includes, for example, patient identification information IJ and area of ​​interest information TJ related to a specific area of ​​interest within the patient's large intestine. Details of the information included in area of ​​interest information TJ will be explained later.

[0038] The display control unit 25 functions as a display control means. The display control unit 25 generates a display image HG including the endoscopic image EG obtained in the current endoscopic examination and outputs the display image HG to the display device 2. The display control unit 25 can also set the display mode of predetermined elements included in the display image HG in response to the input signal NS generated by the input unit 14 or the operation unit 36.

[0039] The display control unit 25 uses object detection information BJ obtained from the object detection unit 21 to display the position of the object indicated by the object detection information BJ within the endoscopic image EG. The display control unit 25 also uses position information LJ obtained from the position information acquisition unit 22 to display the current insertion state of the endoscope scope 3 in the large intestine. Furthermore, when a trigger signal TS is input from the trigger generation unit 23 while the endoscope scope 3 is inserted in the large intestine, the display control unit 25 displays a focus mark TM indicating the area of ​​interest in the large intestine. When the display control unit 25 displays a focus mark TM in the current endoscopic examination, it stores focus area information TJ, which includes information indicating the position corresponding to the focus mark TM in the large intestine, in the information storage unit 24. Furthermore, by referring to the focus area information TJ of the endoscopic examination information NJ read from the information storage unit 24, the display control unit 25 can display the focus mark TM displayed in the previous endoscopic examination together with the focus mark TM of the current endoscopic examination.

[0040] [Specific Examples] Next, specific examples relating to this disclosure will be described. Figure 4 is a diagram showing an example of an endoscopic image. In the following explanation, we will use as an example the case in which an endoscopic image EG1 as shown in Figure 4 is obtained as the shaft 37 of the endoscope scope 3 is inserted into the large intestine.

[0041] The object detection unit 21 inputs the endoscopic image EG1 to each of the polyp detection unit 21A, the diverticulum detection unit 21B, the inflammation detection unit 21C, and the residue detection unit 21D.

[0042] The polyp detection unit 21A obtains the probability PP1 that a polyp exists as an object included in the endoscopic image EG1 by inputting the endoscopic image EG1 to the image recognition model PM. Also, the diverticulum detection unit 21B obtains the probability KP1 that a diverticulum exists as an object included in the endoscopic image EG1 by inputting the endoscopic image EG1 to the image recognition model KM. Also, the inflammation detection unit 21C obtains the probability IP1 that an inflammation exists as an object included in the endoscopic image EG1 by inputting the endoscopic image EG1 to the image recognition model IM. Also, the residue detection unit 21D obtains the probability ZP1 that a residue exists as an object included in the endoscopic image EG1 by inputting the endoscopic image EG1 to the image recognition model ZM.

[0043] The object detection unit 21 determines whether each of the probabilities PP1, KP1, IP1, and ZP1 is greater than or equal to the threshold value TH1. Also, the object detection unit 21 specifies one probability XP1 that is greater than or equal to the threshold value TH1 and has the maximum value among the probabilities PP1, KP1, IP1, and ZP1, and obtains the object corresponding to the specified one probability XP1 as the detection result of the object BT included in the endoscopic image EG1.

[0044] According to the above-described processing, the object detection unit 21 can obtain a detection result that the object BT included in the endoscopic image EG1 is a polyp, for example, when XP1 = PP1. Also, according to the above-described processing, the object detection unit 21 can obtain a detection result that the object BT included in the endoscopic image EG1 is a diverticulum, for example, when XP1 = KP1. Also, according to the above-described processing, the object detection unit 21 can obtain a detection result that the object BT included in the endoscopic image EG1 is an inflammation, for example, when XP1 = IP1. Also, according to the above-described processing, the object detection unit 21 can obtain a detection result that the object BT included in the endoscopic image EG1 is a residue, for example, when XP1 = ZP1.

[0045] According to the above-described process, the object detection unit 21 can obtain, as the detection result of the object BT included in the endoscopic image EG1, one type of object having the highest probability of existence and whose existence probability is equal to or higher than the threshold value TH1 among the four types of objects: polyp, diverticulum, inflammation, and residue.

[0046] The object detection unit 21 generates object detection information BJ1 indicating the detection result of the object BT included in the endoscopic image EG1, and outputs the generated object detection information BJ1 to the display control unit 25.

[0047] The position information acquisition unit 22 estimates the position in the large intestine where the endoscopic image EG1 was obtained based on the identification result of the landmark obtained by inputting the endoscopic image EG1 into the identification model LM, and acquires the estimated position as the current position of the distal end portion 38 disposed in the large intestine. Further, the position information acquisition unit 22 outputs information indicating the current position of the distal end portion 38 disposed in the large intestine to the display control unit 25 as position information LJ1.

[0048] The trigger generation unit 23 determines whether or not the input signal NS1 generated by the input unit 14 or the operation unit 36 is a signal indicating a predetermined instruction.

[0049] Specifically, for example, the trigger generation unit 23 determines whether or not the input signal NS1 corresponds to a signal indicating either an instruction related to still image shooting or an instruction related to completion of visual confirmation. The instruction related to still image shooting is performed, for example, when recording an image including the object BT as the target site image BG. The instruction related to completion of visual confirmation is performed, for example, when the user visually confirms the actual state of the object BT or the like.

[0050] When the trigger generation unit 23 determines that the input signal NS1 is a signal indicating a predetermined instruction, the trigger generation unit 23 generates a trigger signal TS1 and outputs the generated trigger signal TS1 to the display control unit 25.

[0051] FIG. 5 is a diagram showing an example of a display image generated by the endoscopic examination support device according to the present disclosure. The display control unit 25 generates a display image HG1 having a main display area MA and a sub-display area SA as shown in FIG. 5.

[0052] The main display area MA includes, for example, an endoscopic image EG1 containing an object BT, and a detection frame DW.

[0053] The display control unit 25 uses the object detection information BJ1 obtained from the object detection unit 21 to add a detection frame DW around the object BT in the endoscopic image EG1. The detection frame DW is generated as a frame having a predetermined shape, such as a rectangle. In this specific example, the endoscopic images displayed in the main display area MA, regardless of whether or not they include object BT, can be collectively referred to as the main image MG.

[0054] The sub-display area SA includes, for example, a sub-image SG, a schematic diagram of the intestinal tract DZ, a schematic diagram of the insertion state SZ, and a focus mark TM.

[0055] Here, we will explain a specific example of the processing related to the display of elements included in the sub-display area SA of the displayed image HG1. Unless otherwise specified, the processing related to the following specific example will be explained as the processing when the insertion operation of inserting the endoscope scope 3 into the innermost part of the large intestine is being performed.

[0056] The display control unit 25, for example, displays a sub-image SG in the sub-display area SA and displays a schematic diagram of the intestinal tract DZ at a position along the outer edge of the sub-image SG, after the timing when an input signal NS1 indicating the start of an endoscopic examination is received.

[0057] The display control unit 25 can display images corresponding to the status of the endoscopic examination as sub-images SG.

[0058] The display control unit 25 can, for example, display the main image MG acquired at the time when object BT, indicated by object detection information BJ1, is detected during the endoscopic examination, as a sub-image SG.

[0059] The display control unit 25 can, for example, display a sub-image SG of a target area corresponding to the position of the closest target mark TM from the current position of the tip 38 indicated by the position information LJ1. Furthermore, when the display control unit 25 detects that an insertion operation of the endoscope scope 3 is being performed based on the time change in the position of the tip 38 indicated by the position information LJ1, it can display a sub-image SG of a target area corresponding to the position of the latest target mark TM that the tip 38 has passed. The display control unit 25 can also, for example, display a sub-image SG of a target area obtained in the current endoscopic examination. Additionally, the display control unit 25 can, for example, display a sub-image SG of a target area obtained in the previous endoscopic examination by referring to the target area information TJ of the endoscopic examination information NJ read from the information storage unit 24.

[0060] According to the process described above, the display control unit 25 can display as a sub-image SG the image corresponding to the position of the focus mark TM closest to the position of the endoscope camera indicated by the position information LJ1, from among the focus area image TG obtained when the focus mark TM was added in the previous endoscopy and the focus area image TG obtained when the focus mark TM was added in the current endoscopy.

[0061] Furthermore, if the display control unit 25 cannot acquire an image to be displayed as a sub-image SG, for example, it may display a predetermined image, such as a monochrome image, as the sub-image SG in the sub-display area SA.

[0062] Figure 6 is a diagram illustrating a specific example of the processing related to the display of elements included in the sub-display area of ​​the displayed image. The display control unit 25 can display a schematic diagram of the intestinal tract DZ having the shape shown in Figure 6 by, for example, adding a region having a band-shaped space that mimics the shape of the intestinal tract of the large intestine to a position along the outer edge of the sub-image SG. Note that in Figures 6 to 9, the specific display content of the sub-image SG is omitted for illustrative purposes.

[0063] According to the intestinal tract schematic diagram DZ, the opening leading from the lower central outer edge of sub-image SG into the interior of the band-shaped space represents the anus, which corresponds to the insertion point of the endoscope scope 3. Also, according to the intestinal tract schematic diagram DZ, the band-shaped space added to the outer edge from the lower central to the lower right of sub-image SG represents the intestinal tract from the rectum to the sigmoid colon. Furthermore, according to the intestinal tract schematic diagram DZ, the band-shaped space added to the outer edge from the lower right to the upper right of sub-image SG represents the descending colon. Furthermore, according to the intestinal tract schematic diagram DZ, for example, the band-shaped space added to the outer edge from the upper right to the upper left of sub-image SG represents the transverse colon. Furthermore, according to the intestinal tract schematic diagram DZ, for example, the band-shaped space added to the outer edge from the upper left to the lower left of sub-image SG represents the ascending colon. Furthermore, according to the intestinal tract schematic diagram DZ, the band-shaped space added to the outer edge near the lower left of sub-image SG represents the intestinal tract from the cecum to the appendix. Furthermore, according to the schematic diagram of the intestinal tract DZ, the part corresponding to the appendix is ​​the occluded or innermost part of the band-shaped space.

[0064] According to the processing described above, the display control unit 25 can display the endoscopic image EG1 and a sub-image SG that is different from the endoscopic image EG1 together, and can display a schematic diagram of the intestinal tract DZ at a position along the outer edge of the sub-image SG.

[0065] The display control unit 25 displays a schematic insertion state diagram SZ showing the current insertion state of the endoscope scope 3 in the large intestine, based on the current position of the tip portion 38 indicated by the position information LJ1 obtained after an input signal NS1 indicating the start of an endoscopic examination is received.

[0066] Figure 7 is a diagram illustrating a specific example of the processing related to the display of elements included in the sub-display area of ​​the displayed image. The display control unit 25 sets a predetermined position of the opening of the intestinal schematic diagram DZ as the starting point and sets the position of the intestinal schematic diagram DZ corresponding to the current position of the tip 38 indicated by the position information LJ1 as the ending point. The display control unit 25 then connects the starting point and the ending point set as described above with at least one line segment along the internal space of the intestinal schematic diagram DZ, thereby displaying an insertion state schematic diagram SZ having the shape shown in Figure 7. Furthermore, the display control unit 25 can change the length and / or number of line segments connecting the starting point and the ending point of the insertion state schematic diagram SZ according to the current position of the tip 38 indicated by the position information LJ1.

[0067] According to the schematic insertion diagram SZ in Figure 7, the user can determine that the current position of the tip 38 is in the upper part of the descending colon by checking the position of the tip of the line segment in the schematic insertion diagram SZ. The user can also determine that the endoscope scope 3 has been inserted to the upper part of the descending colon by checking the schematic insertion diagram SZ in Figure 7.

[0068] As shown in Figure 7, the attention marks TM1 and TM2 are added to the intestinal schematic diagram DZ at the positions where the tip 38 passed, as attention marks TM for this endoscopic examination. The display control unit 25 can add the attention marks TM1 and TM2 to the intestinal schematic diagram DZ by performing the same processing as the process for adding the attention mark TM3 described later.

[0069] As shown in Figure 7, attention marks TM4 and TM5 are added to the intestinal tract schematic diagram DZ at a position beyond the tip 38. The display control unit 25 can acquire information indicating the positions corresponding to attention marks TM4 and TM5 in the large intestine based on the attention area information TJ of the previous endoscopic examination contained in the endoscopic examination information NJ read from the information storage unit 24. The display control unit 25 can also display the attention marks TM4 and TM5 added to the intestinal tract schematic diagram DZ in the previous endoscopic examination together with the attention marks TM1 and TM2 added to the intestinal tract schematic diagram DZ in the current endoscopic examination. Furthermore, the display control unit 25 can set the display color of attention marks TM4 and TM5 to a predetermined color different from the display color of attention marks TM1 and TM2. In addition, the display control unit 25 can display the identification information IJ contained in the endoscopic examination information NJ read from the information storage unit 24 together with the attention marks TM4 and TM5.

[0070] When an object BT, indicated by object detection information BJ1, is detected, the display control unit 25 receives a trigger signal TS1 and, based on the current position of the tip portion 38 indicated by position information LJ1, adds a focus mark TM to the position in the intestinal tract schematic diagram DZ corresponding to the tip of the line segment in the insertion state schematic diagram SZ. The focus mark TM is added, for example, as a rectangular mark occupying a part of the band-shaped space in the intestinal tract schematic diagram DZ.

[0071] Figure 8 is a diagram illustrating a specific example of the processing related to the display of elements included in the sub-display area of ​​the displayed image. When an object BT indicated by object detection information BJ1 is detected and a trigger signal TS1 is input, the display control unit 25 can, for example, add a point of interest mark TM3 as shown in Figure 8 to the position of the intestinal tract schematic diagram DZ corresponding to the tip of the line segment of the insertion state schematic diagram SZ as shown in Figure 7.

[0072] According to the process described above, the display control unit 25 can add attention marks TM1, TM2, and TM3 to the intestinal schematic diagram DZ as attention marks TM for the current endoscopic examination. Furthermore, according to the process described above, when a predetermined instruction is given in conjunction with the detection of object BT, the display control unit 25 can add attention marks TM to the position in the intestinal schematic diagram DZ corresponding to the position of the endoscope camera indicated by the position information LJ1.

[0073] The display control unit 25 can add marks of different sizes as attention marks TM according to the detection interval of object BT indicated by the object detection information BJ1. The detection interval of object BT can be defined, for example, as the interval in which a single object included in the endoscopic image EG is continuously detected.

[0074] Here, the size of polyps and diverticula is considered to be smaller than the size of inflammation and residue. That is, the detection interval for polyps and diverticula can be considered to be narrower than the detection interval for inflammation and residue. Therefore, the display control unit 25 can, for example, add a relatively small size of attention mark TM to the intestinal schematic diagram DZ if the object BT indicated by the object detection information BJ1 is a polyp or a diverticulum. Also, the display control unit 25 can, for example, add a relatively large size of attention mark TM to the intestinal schematic diagram DZ if the object BT indicated by the object detection information BJ1 is inflammation or residue. In the example in Figure 8, the interval in which either a polyp or a diverticulum was detected in this endoscopic examination is indicated by attention marks TM1 and TM2. Also, in the example in Figure 8, the interval in which either inflammation or residue was detected in this endoscopic examination is indicated by attention mark TM3.

[0075] The display control unit 25 can add marks set to different display colors as attention marks TM according to the type of object BT indicated by the object detection information BJ1. Furthermore, the display control unit 25 can set the display color of the attention marks TM to a color desired by the user in response to the input signal NS1. In this specific example, for illustrative purposes, the display color of the attention marks is represented by hatching. As shown in the example in Figure 8, it is indicated that different objects BT were detected at the locations of attention marks TM1, TM2, and TM3, which are each marked with different hatching.

[0076] When the input signal NS1 corresponding to the trigger signal TS1 is an instruction related to still photography, the display control unit 25 can add a focus mark TM to the intestinal tract schematic diagram DZ and acquire the main image MG displayed in the main display area MA as the focus area image TG at the timing when the instruction is given.

[0077] When the display control unit 25 displays the attention mark TM3, it stores information indicating the location corresponding to the attention mark TM3 within the large intestine in the information storage unit 24 as the attention site information TJ for the current endoscopic examination. The location corresponding to the attention mark TM3 within the large intestine can be rephrased as, for example, the location of the attention mark TM3 added to the intestinal tract schematic diagram DZ.

[0078] According to this specific example, for instance, the image TG of the area of ​​interest acquired at the location where the attention mark TM3 is attached can be included in the area of ​​interest information TJ of the endoscopic examination. Furthermore, according to this specific example, for instance, information indicating the type of object BT corresponding to the attention mark TM3 can be included in the area of ​​interest information TJ of the endoscopic examination.

[0079] Note that the example in Figure 8 shows the situation after the attention marks TM1 and TM2 have been displayed. Therefore, if the situation shown in the example in Figure 8 occurs, the display control unit 25 stores information indicating the locations corresponding to the attention marks TM1, TM2, and TM3 in the large intestine in the information storage unit 24 as the attention site information TJ for this endoscopic examination.

[0080] Figure 9 is a diagram illustrating a specific example of the processing related to the display of elements included in the sub-display area of ​​the display image. After displaying the attention mark TM3, the display control unit 25 displays attention marks TM1 to TM5 and changes the shape of the line segments in the insertion state schematic diagram SZ according to the position information LJ1. Through this processing, the display control unit 25 can display each element having the display configuration shown in Figure 9 in the sub-display area SA of the display image HG1, for example, when the insertion operation of the endoscope scope 3 is completed. Furthermore, the display control unit 25 can detect that the insertion operation of the endoscope scope 3 is completed, for example, when it obtains information indicating that the current position of the tip 38 is in or near the appendix as the position information LJ1 of the current endoscopic examination.

[0081] By checking the sub-display area SA, which is displayed in a manner such as that shown in Figure 9, the user can recognize that the locations corresponding to the attention marks TM1 to TM3 in the large intestine are areas that the user determined required detailed observation during the current endoscopic examination. Furthermore, by checking the sub-display area SA, which is displayed in a manner such as that shown in Figure 9, the user can recognize that the locations corresponding to the attention marks TM4 and TM5 in the large intestine are areas that the user determined required detailed observation during the previous endoscopic examination. Additionally, by checking the sub-display area SA, which is displayed in a manner such as that shown in Figure 9, the user can perform the withdrawal operation to remove the endoscope scope 3, which has been inserted to the deepest part of the large intestine, and observe each of the locations corresponding to the attention marks TM1 to TM5 in the large intestine in detail.

[0082] [Processing Flow] Next, the processing flow performed in the endoscopic examination support device 1 will be explained. Figure 10 is a flowchart showing an example of processing performed in the endoscopic examination support device according to this disclosure.

[0083] First, the endoscopic examination support device 1 displays the endoscopic image EG obtained by the endoscope scope 3 as the main image MG in the main display area MA, and also displays images such as the image TG of the area of ​​interest obtained in the previous endoscopic examination as sub-images SG in the sub-display area SA (step S11).

[0084] Next, the endoscopy support device 1 displays a schematic diagram of the intestinal tract DZ, which has a band-shaped space that mimics the shape of the intestinal tract of the large intestine, at a position along the outer edge of the sub-image SG displayed in step S11 (step S12).

[0085] Next, the endoscopic examination support device 1 determines whether the information relating to the attention mark TM displayed during the previous endoscopic examination is included in the attention site information TJ read from the information storage unit 24 (step S13).

[0086] If the information relating to the attention mark™ displayed in the previous endoscopic examination is not included in the attention site information TJ (step S13: NO), the endoscopic examination support device 1 performs the process described in step S15 below. If the information relating to the attention mark™ displayed in the previous endoscopic examination is included in the attention site information TJ (step S13: YES), the endoscopic examination support device 1 adds the attention mark™ to the intestinal tract schematic diagram DZ (step S14), and then performs the process described in step S15.

[0087] The endoscopy support device 1 displays an insertion state schematic diagram SZ, which shows the current insertion state of the endoscope scope 3 in the large intestine using line segments, in the internal space of the intestinal tract schematic diagram DZ, according to the current position of the tip 38 indicated by the position information LJ obtained from the position information acquisition unit 22 (step S15).

[0088] Next, the endoscopic examination support device 1 determines whether or not it was able to detect object BT from the endoscopic image EG based on the object detection information BJ obtained from the object detection unit 21 (step S16).

[0089] If the endoscopic examination support device 1 cannot detect object BT from the endoscopic image EG (step S16: NO), it performs the process in step S19. If the endoscopic examination support device 1 can detect object BT from the endoscopic image EG (step S16: YES), it determines whether the input signal NS generated by the input unit 14 or the operation unit 36 ​​is a signal indicating a predetermined instruction (step S17).

[0090] If the input signal NS generated by the input unit 14 or the operation unit 36 ​​is not a signal indicating a predetermined instruction (step S17: NO), the endoscopy support device 1 performs the process described in step S19 below. If the input signal NS generated by the input unit 14 or the operation unit 36 ​​is a signal indicating a predetermined instruction (step S17: YES), the endoscopy support device 1 adds a point of attention mark TM to the position in the intestinal tract schematic diagram DZ corresponding to the tip of the line segment in the insertion state schematic diagram SZ (step S18), and then performs the process described in step S19.

[0091] The endoscopic examination support device 1 determines whether or not the insertion operation of the endoscope scope 3 has been completed based on the time change in the position of the tip portion 38 indicated by the position information LJ obtained from the position information acquisition unit 22 (step S19).

[0092] If the insertion of the endoscope scope 3 is not yet complete (step S19: NO), the endoscopy support device 1 returns to step S15. If the insertion of the endoscope scope 3 is complete (step S19: YES), the endoscopy support device 1 terminates the series of processes while maintaining the attention mark TM added to the intestinal tract schematic diagram DZ.

[0093] As described above, according to this embodiment, during the period when the endoscope is being inserted, a mark of interest can be added to indicate the location in the colon where at least one of the four types of objects—polyps, diverticula, inflammation, and residues—is present and which the user has determined requires more detailed observation. Furthermore, according to this embodiment, during the period when the endoscope is being removed, the location in the colon corresponding to the marked area of ​​interest can be observed in detail. Moreover, according to this embodiment, the area marked of interest in the current endoscopic examination and the area marked of interest in the previous endoscopic examination can be confirmed at the same time. Therefore, according to this embodiment, it is easy to understand which areas require more detailed observation during the endoscopic examination.

[0094] [Modifications] Modifications of this embodiment are described below. For simplicity, specific explanations of parts to which the previously described processes etc. can be applied will be omitted as appropriate.

[0095] (Modification 1) The display control unit 25 may, for example, when it detects that the endoscope scope 3 is being withdrawn based on the time change in the position of the tip portion 38 indicated by the position information LJ, display a sub-image SG of the area of ​​interest TG corresponding to the position of the point of interest mark TM that the tip portion 38 will next pass over.

[0096] (Modification 2) The display control unit 25 may, for example, display a sub-image SG of the area of ​​interest corresponding to the position of the mark of interest TM when an input signal NS indicating an instruction to select a mark of interest TM added to the schematic diagram of the intestinal tract DZ is input.

[0097] (Modification 3) The object detection unit 21 may include in the object detection information BJ one probability XP among probabilities PP, KP, IP, and ZP that is equal to or greater than the threshold TH1 and has the maximum value. Probability XP is acquired as a probability corresponding to the type of object BT included in the endoscopic image EG.

[0098] The display control unit 25 may, when the probability XP included in the object detection information BJ is greater than or equal to the threshold TH2 (>TH1), add a focus mark TM set to the display color of the object BT corresponding to the probability XP to the intestinal tract schematic diagram DZ, regardless of whether or not a trigger signal TS is input.

[0099] According to the process described in this modified example, the display control unit 25 can decide whether or not to add a focus mark TM to the intestinal tract schematic diagram DZ based on judgment criteria equivalent to those of the user.

[0100] (Modification 4) The display control unit 25 may, for example, set whether or not to display the attention marks TM added to the intestinal schematic diagram DZ during the current endoscopic examination in the next endoscopic examination, in response to the input signal NS generated by the input unit 14 or the operation unit 36. Alternatively, the display control unit 25 may, for example, set whether or not to display each of the multiple attention marks TM added to the intestinal schematic diagram DZ during the current endoscopic examination in the next endoscopic examination.

[0101] <Second Embodiment> Figure 11 is a block diagram showing another example of the functional configuration of the endoscopic examination support device according to the present disclosure.

[0102] The endoscopic examination support device 500 has the same hardware configuration as the endoscopic examination support device 1. The endoscopic examination support device 500 also includes an object detection means 511, a position information acquisition means 512, and a display control means 513.

[0103] The object detection means 511 can be implemented, for example, using the functions of the object detection unit 21. The location information acquisition means 512 can be implemented, for example, using the functions of the location information acquisition unit 22. The display control means 513 can be implemented, for example, using the functions of the display control unit 25.

[0104] Figure 12 is a flowchart showing another example of processing performed in the endoscopic examination support device according to this disclosure.

[0105] The object detection means 511 detects objects included in the endoscopic image obtained by imaging the inside of the large intestine with an endoscope camera (step S51).

[0106] The position information acquisition means 512 acquires position information indicating the position of the endoscope camera within the large intestine (step S52).

[0107] The display control means 513 displays a schematic diagram having a shape that mimics the intestinal tract of the large intestine (step S53), and when a predetermined instruction is given in conjunction with the detection of an object, it adds a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the position information (step S54).

[0108] According to this embodiment, it is possible to easily identify which areas require detailed observation during an endoscopic examination.

[0109] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0110] (Note 1) An endoscopic examination support device comprising: object detection means for detecting objects included in endoscopic images obtained by imaging the inside of the large intestine with an endoscope camera; position information acquisition means for acquiring position information indicating the position of the endoscope camera in the large intestine; and display control means for displaying a schematic diagram having a shape that mimics the intestinal tract of the large intestine, and when a predetermined instruction is given in conjunction with the detection of the object, adding a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the position information.

[0111] (Note 2) The endoscopic examination support device according to Note 1, wherein the display control means displays the endoscopic image and a sub-image different from the endoscopic image together, and displays the schematic diagram at a position along the outer edge of the sub-image.

[0112] (Note 3) The endoscopic examination support device according to Note 2, wherein the display control means displays a line segment indicating the insertion state of the endoscope in the large intestine based on the position of the endoscope camera indicated by the position information, and adds the mark to the position in the schematic diagram corresponding to the tip of the line segment.

[0113] (Note 4) The endoscopic examination support device described in Note 3, wherein the prescribed instruction is either an instruction related to still photography or an instruction related to the completion of visual confirmation.

[0114] (Note 5) The endoscopic examination support device according to Note 2, wherein the display control means adds marks having different sizes to the schematic diagram according to the detection interval of the object.

[0115] (Note 6) The endoscopic examination support device according to Note 2, wherein the object detection means obtains as the detection result of an object included in the endoscopic image one type of object from among a predetermined plurality of types of objects which has the highest probability of existence and whose probability of existence is equal to or greater than a first threshold, and the display control means adds the mark set to the display color corresponding to the one type of object to the schematic diagram when the probability of existence is equal to or greater than a second threshold which is greater than the first threshold, regardless of whether or not there is a predetermined instruction.

[0116] (Note 7) The endoscopic examination support device according to Note 6, wherein the specified multiple types of objects include polyps, diverticula, inflammation, and residues.

[0117] (Note 8) The endoscopic examination support device according to Note 2, wherein the display control means displays the marks added to the schematic diagram during the previous examination of the colon together with the marks added to the schematic diagram during the current examination of the colon.

[0118] (Note 9) The endoscopic examination support device according to Note 8, wherein the display control means displays as a sub-image the image corresponding to the position of the mark closest to the position of the endoscopic camera indicated by the position information, which is obtained when the mark was added during the previous examination of the colon and when the mark was added during the current examination of the colon.

[0119] (Note 10) The endoscopic examination support device according to Note 2, wherein the display control means sets whether or not to display the mark added to the schematic diagram during the current examination of the colon during the next examination of the colon.

[0120] (Note 11) The endoscopic examination support device according to Note 1, wherein the position information acquisition means estimates the position within the large intestine from which the endoscopic image was obtained based on the identification result of anatomical landmarks contained in the endoscopic image, and acquires the estimated position as the position of the endoscopic camera within the large intestine.

[0121] (Note 12) The endoscopic examination support device according to claim 1, wherein the object detection means detects the object using a trained machine learning model.

[0122] (Note 13) A computer-based method for assisting endoscopic examinations, comprising: detecting an object included in an endoscopic image obtained by imaging the inside of the large intestine with an endoscope camera; acquiring positional information indicating the position of the endoscope camera within the large intestine; displaying a schematic diagram having a shape that mimics the intestinal tract of the large intestine; and, when a predetermined instruction is given in conjunction with the detection of the object, adding a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the positional information.

[0123] (Note 14) A recording medium that records a program that causes a computer to perform the following process: to detect an object contained in an endoscopic image obtained by imaging the inside of the large intestine with an endoscope camera; to acquire positional information indicating the position of the endoscope camera inside the large intestine; to display a schematic diagram having a shape that mimics the intestinal tract of the large intestine; and, when a predetermined instruction is given in conjunction with the detection of the object, to add a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the positional information.

[0124] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0125] 1 Endoscopy support device 2 Display device 3 Endoscope scope 11 Processor 12 Memory 13 Interface 21 Object detection unit 22 Position information acquisition unit 23 Trigger generation unit 24 Information storage unit 25 Display control unit

Claims

1. An endoscopic examination support device comprising: object detection means for detecting objects included in endoscopic images obtained by imaging the inside of the large intestine with an endoscope camera; position information acquisition means for acquiring position information indicating the position of the endoscope camera within the large intestine; and display control means for displaying a schematic diagram having a shape that mimics the intestinal tract of the large intestine, and when a predetermined instruction is given in conjunction with the detection of the object, adding a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the position information.

2. The endoscopic examination support device according to claim 1, wherein the display control means displays the endoscopic image and a sub-image different from the endoscopic image together, and displays the schematic diagram at a position along the outer edge of the sub-image.

3. The endoscopic examination support device according to claim 2, wherein the display control means displays a line segment indicating the insertion state of the endoscope in the large intestine based on the position of the endoscope camera indicated by the position information, and adds the mark to the position in the schematic diagram corresponding to the tip of the line segment.

4. The endoscopic examination support device according to claim 3, wherein the predetermined instruction is either an instruction relating to still photography or an instruction relating to the completion of visual confirmation.

5. The endoscopic examination support device according to claim 2, wherein the display control means adds marks having different sizes to the schematic diagram according to the detection interval of the object.

6. The endoscopic examination support device according to claim 2, wherein the object detection means obtains as the detection result of an object included in the endoscopic image one type of object from among a predetermined plurality of types of objects which has the highest probability of existence and whose probability of existence is equal to or greater than a first threshold, and the display control means adds the mark set to the display color corresponding to the one type of object to the schematic diagram when the probability of existence is equal to or greater than a second threshold which is greater than the first threshold, regardless of whether or not there is a predetermined instruction.

7. The endoscopic examination support device according to claim 6, wherein the predetermined plurality of objects include polyps, diverticula, inflammation, and residues.

8. The endoscopic examination support device according to claim 2, wherein the display control means displays the marks added to the schematic diagram during the previous examination of the colon together with the marks added to the schematic diagram during the current examination of the colon.

9. The endoscopic examination support device according to claim 8, wherein the display control means displays as the sub-image the image corresponding to the position of the mark closest to the position of the endoscopic camera indicated by the position information, from among the endoscopic image obtained when the mark was added during the previous examination of the colon and the endoscopic image obtained when the mark was added during the current examination of the colon.

10. The endoscopic examination support device according to claim 2, wherein the display control means sets whether or not to display the mark added to the schematic diagram during the current examination of the colon during the next examination of the colon.

11. The endoscopic examination support device according to claim 1, wherein the position information acquisition means estimates the position within the large intestine from which the endoscopic image was obtained based on the identification result of anatomical landmarks contained in the endoscopic image, and acquires the estimated position as the position of the endoscopic camera within the large intestine.

12. The endoscopic examination support device according to claim 1, wherein the object detection means detects the object using a trained machine learning model.

13. A computer-operated method for assisting endoscopic examinations, comprising: detecting an object included in an endoscopic image obtained by imaging the inside of the large intestine with an endoscope camera; acquiring positional information indicating the position of the endoscope camera within the large intestine; displaying a schematic diagram having a shape that mimics the intestinal tract of the large intestine; and, when a predetermined instruction is given in conjunction with the detection of the object, adding a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the positional information.

14. A recording medium that records a program that causes a computer to perform the following processes: detecting an object contained in an endoscopic image obtained by imaging the inside of the large intestine with an endoscope camera; acquiring positional information indicating the position of the endoscope camera within the large intestine; displaying a schematic diagram having a shape that mimics the intestinal tract of the large intestine; and, when a predetermined instruction is given in conjunction with the detection of the object, adding a mark to the position on the schematic diagram corresponding to the position of the endoscope camera indicated by the positional information.

Images