Endoscopic examination support device, endoscopic examination support method, and program

Abstract

In this endoscopic examination assistance device, a three-dimensional model generation means generates a three-dimensional model of a luminal organ in which an endoscopic camera is disposed on the basis of an endoscopic image obtained by imaging the inside of the luminal organ with the endoscopic camera. An unobserved region detection means detects, as an unobserved region, a region estimated not to have been observed yet by the endoscope camera on the basis of the three-dimensional model. A display image generation means generates a display image including at least one among information indicating a position of the unobserved region in the endoscopic image, and information indicating a direction of the unobserved region outside the endoscopic image. The endoscopic examination assistance device can assist with the decision making of a user or the like.

Description

[Technical Field]

[0001] This disclosure relates to technologies that can be used when presenting information to support endoscopic examinations. [Background technology]

[0002] Technologies for providing information to support endoscopic examinations are already known.

[0003] Specifically, for example, Patent Document 1 discloses a method of displaying information indicating the analyzable and non-analyzable parts of the large intestine, based on an image of the inside of the large intestine, in relation to the structure of the large intestine. Also, for example, Patent Document 1 discloses a method of determining that parts located within the field of view of the image sensor, visible in the captured image, and with good imaging conditions are analyzable parts, and that the other parts are non-analyzable parts. Furthermore, for example, Patent Document 1 discloses a method of detecting parts of the aforementioned non-analyzable parts that are highly likely to be missed as missed parts. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] International Publication No. WO2021 / 171464 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, Patent Document 1 does not disclose any specific method for displaying information in real time indicating whether or not an area has actually been observed during an endoscopic examination.

[0006] Therefore, according to the technology disclosed in Patent Document 1, for example, an issue arises in that the number or frequency of operations performed to observe areas that have not been observed during an endoscopic examination may increase, potentially imposing an excessive burden on the operator performing the endoscopic examination.

[0007] One of the purposes of this disclosure is to provide an endoscopic examination support device that can reduce the burden placed on the operator performing the endoscopic examination. [Means for solving the problem]

[0008] From one perspective of this disclosure, endoscopic examination support devices, Based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscope camera, the endoscopy 3D model generation means that generates a 3D model of the tubular organ in which a mirror camera is positioned. and, Based on the aforementioned three-dimensional model, it is presumed that observation by the endoscopic camera was not performed. An unobserved region detection means that detects the region as an unobserved region, The position of the unobserved region within the endoscopic image Masks with a covering display configuration And, information indicating the direction of the unobserved area outside the endoscopic image, including A means for generating a display image, Equipped with 、 When the display image generation means detects an area within the endoscope image where the brightness is below a predetermined value as the unobserved area, it transitions the display of the mask from on to off when the brightness of that area exceeds the predetermined value and the area is continuously imaged for a predetermined time or longer.

[0009] In other respects of this disclosure, the computer-assisted endoscopic examination method performed is: Based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscope camera, a three-dimensional model of the tubular organ in which the endoscope camera is positioned is generated. Based on the aforementioned 3D model, areas that are presumed not to have been observed by the endoscope camera are detected as unobserved areas. The position of the unobserved region within the endoscopic image Masks with a covering display configurationand information indicating the direction of the unobserved region outside the endoscopic image including Generate a display image death, If an area within the endoscopic image whose brightness is below a predetermined value is detected as an unobserved area, and the brightness of that area exceeds the predetermined value and the area is continuously imaged for a predetermined time or longer, the display of the mask is switched from on to off.

[0010] In still another aspect of the present disclosure, the program Based on an endoscopic image obtained by imaging the inside of a luminal organ with an endoscopic camera, generate a three-dimensional model of the luminal organ in which the endoscopic camera is disposed, Based on the three-dimensional model, detect a region that is presumed not to be observed by the endoscopic camera as an unobserved region, The position of the unobserved region inside the endoscopic image Masks with a covering display configuration and information indicating the direction of the unobserved region outside the endoscopic image including Generate a display image death, If an area within the endoscopic image whose brightness is below a predetermined value is detected as an unobserved area, and the brightness of that area exceeds the predetermined value and that area is continuously imaged for a predetermined time or longer, the display of the mask is switched from on to off. Cause a computer to execute the processing

Advantages of the Invention

[0011] According to the present disclosure, the burden imposed on an operator performing an endoscopic examination can be reduced.

Brief Description of the Drawings

[0012] [Figure 1] A diagram showing a schematic configuration of an endoscopic examination system according to the first embodiment. [Figure 2] A block diagram showing a hardware configuration of an endoscopic examination support device according to the first embodiment. [Figure 3] A block diagram showing a functional configuration of an endoscopic examination support device according to the first embodiment. [Figure 4] A diagram for explaining a specific example of a display image. [Figure 5] A diagram for explaining another specific example of a display image. [Figure 6] A flowchart showing an example of processing performed in an endoscopic examination support device according to the first embodiment. [Figure 7] A block diagram showing the functional configuration of the endoscopic examination support device according to the second embodiment. [Figure 8] A flowchart showing an example of the processing performed in the endoscopic examination support device according to the second embodiment. [Modes for carrying out the invention]

[0013] Preferred embodiments of this disclosure will be described below with reference to the drawings.

[0014] <First Embodiment> [System Configuration] Figure 1 is a diagram showing the schematic configuration of an endoscopic examination system according to the first embodiment. As shown in Figure 1, the endoscopic examination system 100 comprises an endoscopic examination support device 1, a display device 2, and an endoscope scope 3 connected to the endoscopic examination support device 1.

[0015] The endoscopic examination support device 1 acquires video (hereinafter also referred to as "endoscopic video Ic"), which includes time-series images obtained by imaging the subject during the endoscopic examination, from the endoscope scope 3, and displays the image on the display device 2 for confirmation by the operator, such as a physician, performing the endoscopic examination. Specifically, the endoscopic examination support device 1 acquires video of the inside of the large intestine obtained during the endoscopic examination as endoscopic video Ic from the endoscope scope 3. Based on the image extracted from endoscopic video Ic (hereinafter also referred to as "endoscopic image"), the endoscopic examination support device 1 estimates the distance (hereinafter also referred to as "depth") between the surface of the large intestine, which is a tubular organ, and the endoscope camera provided at the tip 38 of the endoscope scope 3, as well as the change in the relative posture of the endoscope camera. Then, the endoscopic examination support device 1 generates a 3D model corresponding to the structure of the large intestine (intestinal tract) by performing 3D reconstruction based on the depth and the change in the relative posture of the endoscope camera. In addition, the endoscopic examination support device 1 detects areas that are estimated to be difficult to observe during the endoscopic examination, based on the endoscopic image. Furthermore, the endoscopic examination support device 1 detects candidate lesion regions, which are areas estimated to be potential lesions, based on the endoscopic image. The endoscopic examination support device 1 also detects areas where there are gaps in the 3D model due to the lack of or insufficient 3D reconstruction, as missing regions. The endoscopic examination support device 1 also detects at least one of the areas that are difficult to observe and the missing regions in the 3D model as unobserved regions. The endoscopic examination support device 1 generates a display image based on the endoscopic image corresponding to the current position of the endoscopic camera and the detection results of unobserved regions, and outputs the generated display image to the display device 2.

[0016] Furthermore, areas that are difficult to observe may include, for example, areas that are difficult to see due to insufficient brightness, areas that are difficult to see due to the degree of blurring, and areas where the condition of the mucosal surface cannot be seen due to the presence of residue. In addition, missing areas may include, for example, areas hidden by obstructions in the large intestine such as folds, and areas where imaging by the endoscopic camera has not been performed continuously for a predetermined time or longer. The aforementioned predetermined time may be set to, for example, 1 second.

[0017] Display device 2 has, for example, a liquid crystal monitor. Display device 2 also displays images output from endoscopic examination support device 1.

[0018] The endoscope scope 3 mainly consists of an operating unit 36 ​​for the operator to input information such as air insufflation, water insufflation, angle adjustment, and imaging instructions; a flexible shaft 37 that is inserted into the organ being examined by the patient; a tip section 38 that incorporates an endoscope camera such as a miniature image sensor; and a connection section 39 for connecting to the endoscopy support device 1.

[0019] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of an endoscopy support device according to the first embodiment. The endoscopy support device 1 mainly includes a processor 11, memory 12, interface 13, input unit 14, light source unit 15, sound output unit 16, and database (hereinafter referred to as "DB") 17. Each of these elements is connected via a data bus 19.

[0020] The processor 11 executes predetermined processes by running programs stored in memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit). Note that the processor 11 may be composed of multiple processors. The processor 11 is an example of a computer. The processor 11 also performs processing related to the generation of display images based on the endoscopic images contained in the endoscopic video Ic.

[0021] Memory 12 consists of various volatile memories used as working memory, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memory that stores information necessary for processing the endoscopy support device 1. Memory 12 may also include external storage devices such as hard disks connected to or built into the endoscopy support device 1, or it may include storage media such as removable flash memory or disk media. Memory 12 stores programs for the endoscopy support device 1 to execute each of the processes in this embodiment.

[0022] Furthermore, the memory 12 temporarily stores a series of endoscopic images Ic taken by the endoscope scope 3 during an endoscopic examination, based on the control of the processor 11.

[0023] Interface 13 performs interface operations between the endoscopic examination support device 1 and external devices. For example, interface 13 supplies the display image generated by processor 11 to display device 2. Interface 13 also supplies illumination light generated by light source unit 15 to endoscope scope 3. Interface 13 also supplies an electrical signal indicating the endoscopic image Ic supplied from endoscope scope 3 to processor 11. Interface 13 also supplies the endoscopic image extracted from the endoscopic image Ic to processor 11. Interface 13 may be a communication interface such as a network adapter for wired or wireless communication with external devices, or it may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc.

[0024] The input unit 14 generates input signals based on the operator's actions. The input unit 14 can be, for example, a button, touch panel, remote controller, or voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope scope 3. The light source unit 15 may also incorporate a pump for supplying water or air to the endoscope scope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0025] DB17 stores endoscopic images and other data acquired from the patient's past endoscopic examinations. DB17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or it may include a storage medium such as a removable flash memory. Alternatively, instead of having DB17 within the endoscopic examination system 100, DB17 may be located on an external server, and relevant information may be obtained from the server via communication.

[0026] The endoscopic examination support device 1 may also be equipped with a sensor capable of measuring the rotation and translation of the endoscope camera, such as a magnetic sensor.

[0027] [Functional Configuration] Figure 3 is a block diagram showing the functional configuration of the endoscopic examination support device according to the first embodiment. Functionally, the endoscopic examination support device 1 includes a depth estimation unit 21, a camera posture estimation unit 22, a 3D reconstruction unit 23, a difficult-to-observe area detection unit 24, an unobserved area detection unit 25, a lesion candidate detection unit 26, and a display image generation unit 27.

[0028] The depth estimation unit 21 performs a process to estimate depth from the endoscopic image using a trained image recognition model or the like. That is, the depth estimation unit 21 functions as a distance estimation means and can estimate the distance between the surface of the lumen organ and the endoscopic camera placed inside the lumen organ based on the endoscopic image obtained by imaging the inside of the lumen organ with the endoscopic camera. The depth estimation unit 21 also outputs the depth estimated by the above process to the 3D reconstruction unit 23.

[0029] The camera posture estimation unit 22 performs a process to estimate the rotation and translation of the endoscope camera (i.e., the relative posture change of the endoscope camera; hereinafter simply referred to as "camera posture change") from the point where the first endoscope image was taken to the point where the second endoscope image was taken, using, for example, two temporally consecutive endoscope images. The camera posture estimation unit 22 also performs a process to estimate the camera posture change using, for example, a trained image recognition model. In other words, the camera posture estimation unit 22 functions as a posture change estimation means and can estimate the relative posture change of the endoscope camera based on endoscope images obtained by imaging the inside of a tubular organ with the endoscope camera. The camera posture estimation unit 22 also outputs the camera posture change estimated by the above process to the 3D reconstruction unit 23. The camera posture estimation unit 22 may also use measurement data obtained from a magnetic sensor or the like to estimate the camera posture change.

[0030] Here, the image recognition models used in the depth estimation unit 21 and the camera pose estimation unit 22 are machine learning models that have been pre-trained to estimate depth and camera pose changes from endoscopic images. Hereafter, these will also be referred to as the "depth estimation model" and the "camera pose estimation model." The depth estimation model and the camera pose estimation model can be generated by so-called supervised learning.

[0031] For training a depth estimation model, training data such as endoscopic images with depth assigned as a ground truth label is used. The endoscopic images and depth data used for training are collected in advance from an endoscope camera and a ToF (Time of Flight) sensor installed on the endoscope. Specifically, pairs of RGB images obtained from the endoscope camera and depth data obtained from the ToF sensor are created as training data, and training is performed using this created training data.

[0032] In addition, for the training of the camera pose estimation model, for example, teacher data with the camera pose changes assigned as correct labels to endoscopic images is used. In this case, the camera pose changes can be obtained by using a sensor capable of detecting rotation and translation, such as a magnetic sensor or the like. That is, a pair of the RGB image obtained by the endoscopic camera and the camera pose changes obtained by the sensor is created as teacher data, and training is performed using the teacher data.

[0033] The teacher data used for the training of the depth estimation model and the camera pose estimation model may be created from the simulation video of the endoscope using CG (computer graphics). Thereby, a large amount of teacher data can be created at high speed. By the machine learning device learning the relationship between the endoscopic image, the depth, and the camera pose changes using the teacher data, the depth estimation model and the camera pose estimation model are generated.

[0034] In addition, the depth estimation model and the camera pose estimation model may be generated by self-supervised learning. For example, in self-supervised learning, teacher data is created using motion parallax. Specifically, in self-supervised learning, a pair of endoscopic images I i and endoscopic image I j , a Depth CNN (Convolutional Neural Network) for estimating the depth from the endoscopic image I i , and a Pose CNN for estimating the relative pose from the endoscopic image I i and the endoscopic image I j are prepared. Then, based on the depth and relative pose estimated by each, the endoscopic image I i is reconstructed from the endoscopic image I j (this is also referred to as "endoscopic image I i→j "). Then, the difference between the reconstructed endoscopic image I i→j and the actual endoscopic image I j is used as the loss to train the model.

[0035] The 3D reconstruction unit 23 generates a 3D model corresponding to the structure of the large intestine (intestinal tract) during endoscopic examination by performing 3D reconstruction processing based on the depth obtained from the depth estimation unit 21 and the relative posture change of the endoscope camera obtained from the camera posture estimation unit 22. The 3D reconstruction unit 23 then outputs the 3D model, the relative posture change of the endoscope camera, and the position of the endoscope camera to the unobserved area detection unit 25.

[0036] In other words, the 3D model generation means of this embodiment includes a depth estimation unit 21, a camera pose estimation unit 22, and a 3D reconstruction unit 23.

[0037] The difficult-to-observe area detection unit 24 detects as difficult-to-observe areas areas in an endoscopic image that fall under at least one of the following categories: areas where the brightness is below a predetermined value, areas where the amount of blur is above a predetermined value, and areas where residue is present. In other words, based on the endoscopic image, the difficult-to-observe area detection unit 24 detects as difficult-to-observe areas areas in the interior of tubular organs that are estimated to be difficult to observe with an endoscopic camera. The difficult-to-observe area detection unit 24 then outputs the detection result of the difficult-to-observe areas to the unobserved area detection unit 25.

[0038] The unobserved area detection unit 25 detects areas in the 3D model where there are gaps, based on the relative posture changes of the endoscope camera, the position of the endoscope camera, and the 3D model. Specifically, the unobserved area detection unit 25 detects areas as missing areas that correspond to at least one of the following in the 3D model: areas hidden by occluding objects such as folds, and areas where imaging by the endoscope camera has not been performed continuously for a predetermined time or longer. The observed area detection unit 26 also detects areas as missing areas in the 3D model acquired from the 3D reconstruction unit 23 in the most recent 5 seconds where 3D reconstruction has not been performed continuously for 1 second or more. Furthermore, the unobserved area detection unit 25 performs processing to identify areas in the 3D model generated by the 3D reconstruction unit 23 that correspond to the detection results of hard-to-observe areas obtained from the hard-to-observe area detection unit 24. The unobserved area detection unit 25 also detects hard-to-observe areas and missing areas in the 3D model as unobserved areas. In other words, the unobserved area detection unit 25 detects areas that are presumed not to have been observed by the endoscope camera, based on a three-dimensional model of the tubular organ where the endoscope camera is positioned, as unobserved areas. Furthermore, the unobserved area detection unit 25 can obtain the latest detection results corresponding to the observation history of the large intestine (intestinal tract) by the endoscope camera as the detection result of unobserved areas in the three-dimensional model. The unobserved area detection unit 25 then outputs the relative posture change of the endoscope camera, the position of the endoscope camera, the three-dimensional model, and the detection result of the unobserved area to the display image generation unit 27.

[0039] The lesion candidate detection unit 26 uses a trained image recognition model or the like to detect lesion candidate regions in the endoscopic image, which are areas that are estimated to be lesion candidates. Specifically, the lesion candidate detection unit 26 detects areas containing polyps, for example, as lesion candidate regions. In other words, the lesion candidate detection unit 26 detects lesion candidate regions, which are areas that are estimated to be lesion candidates, based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscopic camera. The lesion candidate detection unit 26 then outputs the detection results of the lesion candidate regions to the display image generation unit 27.

[0040] The display image generation unit 27 generates a display image during an endoscopic examination based on the endoscopic image, the relative posture change of the endoscopic camera, the position of the endoscopic camera, the detection result of the candidate lesion area, the 3D model, and the detection result of the unobserved area in the 3D model, and outputs the generated display image to the display device 2. The display image generation unit 27 also sets the display state (on / off, etc.) of each piece of information included in the display image.

[0041] The displayed image only needs to include at least one of the following: information indicating the location of an unobserved area within the endoscopic image, which corresponds to the field of view of the endoscope camera; and information indicating the direction of an unobserved area outside the endoscopic image, which corresponds to the field of view of the endoscope camera. Such information can be generated, for example, using the detection results of unobserved areas accumulated during the endoscopic examination. In other words, the display image generation unit 27 generates a display image that includes at least one of the following: information indicating the location of an unobserved area within the endoscopic image obtained by imaging the inside of a tubular organ with an endoscope camera; and information indicating the direction of an unobserved area outside the endoscopic image. Furthermore, when the display image generation unit 27 detects that an unobserved area within the endoscopic image has been continuously imaged for a predetermined period of time or longer, it switches the display of the information indicating the location of the unobserved area from on to off.

[0042] Furthermore, the displayed image may include at least one of the following: information indicating the location of a candidate lesion region within the endoscopic image; information indicating the direction of a candidate lesion region outside the endoscopic image; and information indicating the latest detection result of a candidate lesion region. Such information can be generated, for example, using the detection results of candidate lesion regions accumulated during the endoscopic examination.

[0043] According to this embodiment, for example, if multiple unobserved regions detected during an endoscopic examination are located outside the endoscopic image, the displayed image should include information indicating the direction of the unobserved region that is closest to the current position of the endoscopic camera, or the one with the largest area among the multiple unobserved regions.

[0044] Furthermore, according to this embodiment, for example, if multiple candidate lesion regions detected during an endoscopic examination are located outside the endoscopic image, the displayed image should include information indicating the direction of the candidate lesion region that is closest to the current position of the endoscopic camera among the multiple candidate lesion regions.

[0045] [Example Display] Next, we will explain specific examples of display images shown on the display device 2. Figure 4 is a diagram illustrating a specific example of a display image.

[0046] The display image DA in Figure 4 is the image displayed on the display device 2 during an endoscopic examination. The display image DA also includes an endoscopic image 41, a candidate lesion image 42, unobserved direction indicators 43A and 43B, and a lesion direction indicator 44.

[0047] Endoscopic image 41 is an image included in the endoscopic video Ic obtained during the endoscopic examination. Endoscopic image 41 also includes subjects within the field of view at the current position of the endoscope camera and is updated in accordance with the movement of the endoscope camera. Furthermore, endoscopic image 41 includes unobserved area masks 41A and 41B, which indicate the location of unobserved areas within the endoscope image 41.

[0048] The unobserved area mask 41A is displayed in a manner that covers areas within the endoscopic image 41 where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or longer. Furthermore, the unobserved area mask 41A is removed from the endoscopic image 41, for example, when imaging by the endoscopic camera is performed continuously for a predetermined period of time or longer.

[0049] The unobserved area mask 41B is displayed in a manner that covers areas within the endoscopic image 41 that are difficult to see due to insufficient brightness. Furthermore, the unobserved area mask 41B remains displayed even if imaging by the endoscopic camera continues for a predetermined time or longer, for example, if the brightness in the endoscopic image 41 is below a predetermined value. Also, the unobserved area mask 41B is removed from the endoscopic image 41 if, for example, the brightness in the endoscopic image 41 is detected to have exceeded a predetermined value, and imaging by the endoscopic camera continues for a predetermined time or longer.

[0050] The candidate lesion image 42 is smaller in size than the endoscopic image 41 and is positioned to the right of the endoscopic image 41. Furthermore, the candidate lesion image 42 is an image generated by superimposing lesion position information 42A onto another endoscopic image acquired before the time when the endoscopic image 41 was acquired.

[0051] The lesion location information 42A is displayed as information indicating the latest detection result of the candidate lesion area. Also, as shown in the display example in Figure 4, the lesion location information 42A is displayed as a circular marker surrounding the candidate lesion area.

[0052] The unobserved direction indicators 43A and 43B are displayed as information indicating the direction of the unobserved area outside the endoscopic image 41.

[0053] In the example shown in Figure 4, an unobserved direction indicator 43A, which has a mark indicating upward, is displayed adjacent to the upper edge of the endoscopic image 41, and an unobserved direction indicator 43B, which has a mark indicating left, is displayed adjacent to the left edge of the endoscopic image 41. In other words, the unobserved direction indicators 43A and 43B in Figure 4 can inform the operator that an unobserved area outside the endoscopic image 41 is located in the upper left direction relative to the current position of the endoscopic camera.

[0054] In this embodiment, either one of the unobserved direction indicators 43A and 43B may be displayed. Specifically, for example, if the unobserved direction indicator 43A is displayed while the unobserved direction indicator 43B is not displayed, the operator can be informed that an unobserved area outside the endoscopic image 41 is located upward relative to the current position of the endoscopic camera. Also, for example, if the unobserved direction indicator 43B is displayed while the unobserved direction indicator 43A is not displayed, the operator can be informed that an unobserved area outside the endoscopic image 41 is located to the left relative to the current position of the endoscopic camera.

[0055] Furthermore, in this embodiment, indicators similar to the unobserved direction indicators 43A and 43B may be displayed, for example, at a position adjacent to the lower end of the endoscopic image 41 and at a position adjacent to the right end of the endoscopic image 41. In such a case, the operator can be informed that an unobserved area outside the endoscopic image 41 exists in one of eight directions (up, upper right, right, lower right, down, lower left, left, and upper left) relative to the current position of the endoscope camera.

[0056] The lesion direction indicator 44 is displayed as information indicating the direction of a candidate lesion area outside the endoscopic image 41.

[0057] In this example, as shown in Figure 4, a lesion direction indicator 44 with a mark indicating a leftward direction is displayed adjacent to the left edge of the endoscopic image 41. That is, the lesion direction indicator 44 in Figure 4 can inform the operator that a candidate lesion area outside the endoscopic image 41 is located to the left of the current position of the endoscopic camera.

[0058] In this embodiment, for example, indicators similar to the lesion direction indicator 44 may be displayed at positions adjacent to the upper edge of the endoscopic image 41, adjacent to the lower edge of the endoscopic image 41, and adjacent to the right edge of the endoscopic image 41. In such cases, the operator can be informed that a candidate lesion area outside the endoscopic image 41 is located in one of eight directions (up, upper right, right, lower right, down, lower left, left, and upper left) relative to the current position of the endoscope camera.

[0059] According to the display image DA in Figure 4, during an endoscopic examination, the position of the unobserved area within the endoscopic image 41, the direction of the unobserved area outside the endoscopic image 41, and the direction of the candidate lesion area outside the endoscopic image 41 can be displayed simultaneously. Furthermore, according to the display image DA in Figure 4, the display of the unobserved area masks 41A and 41B transitions from on to off depending on the observation status during the endoscopic examination. In addition, according to the display image DA in Figure 4, the display of the indicator showing the direction of the unobserved area outside the endoscopic image 41 can be set to on or off depending on the position and / or orientation of the endoscopic camera during the endoscopic examination, and the display of the indicator showing the direction of the candidate lesion area outside the endoscopic image 41 can also be set to on or off.

[0060] On the other hand, in this embodiment, instead of the display image DA shown in Figure 4, a display image DB as shown in Figure 5 may be displayed on the display device 2. Figure 5 is a diagram illustrating another specific example of a display image. In the following, for simplicity, specific explanations of parts to which the above configuration can be applied will be omitted as appropriate.

[0061] The display image database in Figure 5 shows the images displayed on the display device 2 during an endoscopic examination. The display image database also includes endoscopic images 51, candidate lesion images 42, unobserved direction indicators 43A and 43B, lesion direction indicator 44, and unobserved area confirmation images 55.

[0062] Endoscopic image 51 corresponds to the image obtained by removing the unobserved area masks 41A and 41B from endoscopic image 41.

[0063] The unobserved area confirmation image 55 corresponds to an image obtained by reducing the size of the endoscopic image 51 and adding information indicating the location of the unobserved area within the endoscopic image 51. The unobserved area confirmation image 55 is located to the right of the endoscopic image 51 and below the lesion candidate image 42. The unobserved area confirmation image 55 is updated simultaneously with the update of the endoscopic image 51. The unobserved area confirmation image 55 also includes unobserved area masks 55A and 55B, which are information indicating the location of the unobserved area within the endoscopic image 51.

[0064] The unobserved area mask 55A is displayed in a manner that covers areas within the endoscopic image 51 where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or longer. Furthermore, the unobserved area mask 55A is removed from the unobserved area confirmation image 55, for example, when imaging by the endoscopic camera is performed continuously for a predetermined period of time or longer.

[0065] The unobserved area mask 55B is displayed in a manner that covers areas within the endoscopic image 51 that are difficult to see due to insufficient brightness. Furthermore, the unobserved area mask 55B remains displayed even if imaging by the endoscopic camera continues for a predetermined time or longer, for example, if the brightness in the endoscopic image 51 is below a predetermined value. Also, the unobserved area mask 55B is removed from the unobserved area confirmation image 55 if, for example, the brightness in the endoscopic image 51 is detected to have exceeded a predetermined value and imaging by the endoscopic camera continues for a predetermined time or longer.

[0066] According to the display image database in Figure 5, it is possible to display information similar to that contained in the display image DA in Figure 4 while maintaining a state where the entire area of ​​the endoscopic image obtained during the endoscopic examination can be viewed in detail. Furthermore, according to the display image database in Figure 5, the display state (on / off) of each piece of information can be changed in the same way as in the display image DA in Figure 4.

[0067] [Processing flow] Next, the processing flow performed in the endoscopic examination support device according to the first embodiment will be described. Figure 6 is a flowchart showing an example of the processing performed in the endoscopic examination support device according to the first embodiment.

[0068] First, the endoscopic examination support device 1 estimates the depth from the endoscopic images obtained during the endoscopic examination (step S11).

[0069] Next, the endoscopic examination support device 1 estimates the change in camera posture from two temporally consecutive endoscopic images obtained during the endoscopic examination (step S12).

[0070] Next, the endoscopic examination support device 1 generates a 3D model corresponding to the structure of the large intestine (intestinal tract) during the endoscopic examination by performing a 3D reconstruction process based on the depth obtained in step S11 and the camera posture change obtained in step S12 (step S13).

[0071] Next, the endoscopic examination support device 1 detects areas that are difficult to observe based on the endoscopic images obtained during the endoscopic examination (step S14).

[0072] Next, the endoscopic examination support device 1 detects missing regions in the 3D model generated in step S13 (step S15).

[0073] Next, the endoscopic examination support device 1 detects the region corresponding to the difficult-to-observe region detected in step S14 and the region corresponding to the missing region detected in step S15 in the 3D model generated in step S13 as unobserved regions (step S16).

[0074] Next, the endoscopic examination support device 1 detects candidate lesion areas based on the endoscopic images obtained during the endoscopic examination (step S17).

[0075] Next, the endoscopic examination support device 1 generates a display image (step S18) that includes at least one of the following: information indicating the location of the unobserved area within the endoscopic image obtained during the endoscopic examination, and information indicating the direction of the unobserved area outside the endoscopic image, as well as information indicating the direction of the candidate lesion area outside the endoscopic image, based on the detection result of the unobserved area obtained in step S16 and the detection result of the candidate lesion area obtained in step S17. The display image generated in step S18 is then displayed on the display device 2.

[0076] In this embodiment, the processing in step S12 may be performed before step S11, or the processing in step S11 may be performed simultaneously with the processing in step S12.

[0077] As described above, according to this embodiment, the display state of information indicating the location of an unobserved area within the endoscopic image and information indicating the direction of an unobserved area outside the endoscopic image can be changed according to the position and / or orientation of the endoscopic camera during endoscopic examination. Furthermore, as described above, according to this embodiment, for example, by continuously imaging an unobserved area within the endoscopic image for a predetermined period of time or longer, the display of information indicating the location of the unobserved area can be switched from on to off. Therefore, according to this embodiment, the burden on the operator performing the endoscopic examination can be reduced. This can also be used to support user decision-making.

[0078] <Second Embodiment> Figure 7 is a block diagram showing the functional configuration of the endoscopic examination support device according to the second embodiment.

[0079] The endoscopic examination support device 70 according to this embodiment has the same hardware configuration as the endoscopic examination support device 1. The endoscopic examination support device 70 also includes a 3D model generation means 71, an unobserved area detection means 72, and a display image generation means 73.

[0080] Figure 8 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the second embodiment.

[0081] The 3D model generation means 71 generates a 3D model of the tubular organ in which the endoscopic camera is located, based on endoscopic images obtained by imaging the inside of the tubular organ with the endoscopic camera (step S71).

[0082] The unobserved area detection means 72 detects areas that are presumed not to have been observed by the endoscope camera, based on a three-dimensional model, as unobserved areas (step S72).

[0083] The display image generation means 73 generates a display image that includes at least one of the following: information indicating the location of an unobserved area within the endoscopic image, and information indicating the direction of an unobserved area outside the endoscopic image (step S73).

[0084] According to this embodiment, the burden on the operator performing the endoscopic examination can be reduced.

[0085] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0086] (Note 1) A 3D model generation means generates a 3D model of the tubular organ in which the endoscopic camera is positioned, based on endoscopic images obtained by imaging the inside of the tubular organ with an endoscopic camera, An unobserved area detection means that detects areas that are presumed not to have been observed by the endoscope camera based on the three-dimensional model, A display image generation means that generates a display image including at least one of the following: information indicating the position of the unobserved region within the endoscope image and information indicating the direction of the unobserved region outside the endoscope image. An endoscopic examination support device equipped with the following features.

[0087] (Note 2) The endoscopic examination support device according to Appendix 1, wherein the unobserved area detection means detects at least one of the following as the unobserved area: an observation-difficult area, which is an area inside the tubular organ that is presumed to be difficult to observe with the endoscopic camera, and a missing area, which is an area in the three-dimensional model where a gap has occurred.

[0088] (Note 3) The aforementioned difficult-to-observe region is a region in the endoscopic image that corresponds to at least one of the following: a region where the brightness is below a predetermined value, a region where the amount of blur is above a predetermined value, and a region where residue is present. The endoscopic examination support device according to Appendix 2, wherein the unobserved area detection means detects the area in the three-dimensional model corresponding to the difficult-to-observe area as the unobserved area.

[0089] (Note 4) The endoscopic examination support device according to Appendix 2, wherein the missing region is a region in the three-dimensional model that is hidden by an obstruction within the tubular organ, and a region in which imaging by the endoscopic camera has not been performed continuously for a predetermined time or longer.

[0090] (Note 5) The endoscopy support device according to Appendix 1, wherein the display image generation means detects that the unobserved region located within the endoscope image has been continuously imaged for a predetermined period of time or longer, and then switches the display of information indicating the location of the unobserved region within the endoscope image from on to off.

[0091] (Note 6) The system further includes a lesion candidate detection means that detects lesion candidate regions, which are areas estimated to be lesion candidates based on the aforementioned endoscopic images, using a machine learning model. The endoscopic examination support device according to Appendix 1, wherein the display image generation means generates the display image which includes information indicating the direction of the candidate lesion region outside the endoscopic image.

[0092] (Note 7) The endoscopic examination support device according to Appendix 6, wherein the display image generation means generates the display image including information indicating the latest detection result of the candidate lesion region.

[0093] (Note 8) Based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscope camera, a three-dimensional model of the tubular organ in which the endoscope camera is positioned is generated. Based on the aforementioned 3D model, areas that are presumed not to have been observed by the endoscope camera are detected as unobserved areas. An endoscopy support method for generating a display image that includes at least one of the following: information indicating the location of the unobserved region within the endoscope image, and information indicating the direction of the unobserved region outside the endoscope image.

[0094] (Note 9) Based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscope camera, a three-dimensional model of the tubular organ in which the endoscope camera is positioned is generated. Based on the aforementioned 3D model, areas that are presumed not to have been observed by the endoscope camera are detected as unobserved areas. A recording medium that records a program causing a computer to perform a process of generating a display image that includes at least one of the following: information indicating the position of the unobserved region within the endoscopic image, and information indicating the direction of the unobserved region outside the endoscopic image.

[0095] This application claims priority based on the international application PCT / JP2022 / 029426 filed on 1 August 2022, and incorporates all of its disclosures herein.

[0096] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of symbols]

[0097] 1. Endoscopic examination support device 2 Display device 3 Endoscope 11 processors 12 memory 13 Interfaces 21 Depth estimation part 22 Camera posture estimation unit 23 3D Restoration Section 24. Detection unit for areas that are difficult to observe 25 Unobserved Area Detection Unit 26. Lesion candidate detection unit 27 Display Image Generation Unit 100 Endoscopy System

Claims

1. Based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscope camera, the endoscopy 3D model generation means for generating a 3D model of the tubular organ in which a mirror camera is positioned. and, Based on the aforementioned three-dimensional model, it is presumed that observation by the endoscopic camera was not performed. An unobserved region detection means that detects the region as an unobserved region, A display image generation means that generates a display image including a mask having a display mode that covers the location of the unobserved region within the endoscope image, and information indicating the direction of the unobserved region outside the endoscope image. Equipped with, The display image generation means is an endoscopy support device that, when an area within the endoscope image with a brightness below a predetermined value is detected as an unobserved area, transitions the display of the mask from on to off when the brightness exceeds the predetermined value and the area is continuously imaged for a predetermined time or longer.

2. The endoscopic examination support device according to claim 1, wherein the unobserved area detection means detects at least one of the following as the unobserved area: an observation-difficult area, which is an area inside the tubular organ that is presumed to be difficult to observe with the endoscopic camera, and a missing area, which is an area in the three-dimensional model where a gap has occurred.

3. The aforementioned difficult-to-observe region is a region in the endoscopic image that corresponds to at least one of the following: a region where the brightness is below a predetermined value, a region where the amount of blur is above a predetermined value, and a region where residue is present. The unobserved region detection means corresponds to the difficult-to-observe region in the three-dimensional model. The endoscopic examination support device according to claim 2, which detects the region as the unobserved region.

4. The endoscopic examination support device according to claim 2, wherein the missing region is a region in the three-dimensional model that is hidden by an obstruction within the tubular organ, and a region in which imaging by the endoscope camera has not been performed continuously for a predetermined time or longer.

5. The invention further comprises a lesion candidate detection means that detects a lesion candidate region, which is a region estimated to be a lesion candidate based on the endoscopic image, using a machine learning model, The endoscopic examination support device according to claim 1, wherein the display image generation means generates the display image which includes information indicating the direction of the candidate lesion region outside the endoscopic image.

6. The endoscopic examination support device according to claim 5, wherein the display image generation means generates the display image including information indicating the latest detection result of the candidate lesion region.

7. A computer-assisted method for endoscopy, Based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscope camera, a three-dimensional model of the tubular organ in which the endoscope camera is positioned is generated. Based on the aforementioned three-dimensional model, areas that are presumed not to have been observed by the endoscope camera are detected as unobserved areas. A display image is generated that includes a mask having a display mode that covers the location of the unobserved region within the endoscopic image, and information indicating the direction of the unobserved region outside the endoscopic image. An endoscopy support method in which, when an area within the endoscope image whose brightness is below a predetermined value is detected as an unobserved area, the display of the mask is switched from on to off when the brightness of the area exceeds the predetermined value and the area is continuously imaged for a predetermined time or longer.

8. Based on endoscopic images obtained by imaging the inside of a tubular organ with an endoscope camera, a three-dimensional model of the tubular organ in which the endoscope camera is located is generated. Based on the aforementioned three-dimensional model, areas that are presumed not to have been observed by the endoscope camera are detected as unobserved areas. A display image is generated that includes a mask having a display mode that covers the location of the unobserved region within the endoscopic image, and information indicating the direction of the unobserved region outside the endoscopic image. A program that, when an area within the endoscopic image whose brightness is below a predetermined value is detected as an unobserved area, causes the computer to execute a process to switch the display of the mask from on to off when the brightness of that area exceeds the predetermined value and the area is continuously imaged for a predetermined time or longer.

Images