Endoscopic examination support device, endoscopic examination support method, and program

Abstract

In this endoscopic examination assistance device, a three-dimensional model generation means generates a three-dimensional model of a luminal organ in which an endoscopic camera is disposed on the basis of an endoscopic image obtained by imaging the inside of the luminal organ with the endoscopic camera. An unobserved region detection means detects, as an unobserved region, a region estimated not to have been observed yet by the endoscope camera on the basis of the three-dimensional model. A display image generation means generates a display image including at least one among information indicating a position of the unobserved region in the endoscopic image, and information indicating a direction of the unobserved region outside the endoscopic image. The endoscopic examination assistance device can assist with the decision making of a user or the like.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

[0001] The present disclosure relates to techniques that can be used to present information to assist in endoscopic examinations.

[0002] 2. Description of the Related Art Conventionally, techniques for presenting information to assist endoscopic examinations have been known.

[0003] Specifically, for example, Patent Document 1 discloses a view of displaying information indicating analyzable and unanalyzable portions of the large intestine in association with the structure of the large intestine based on an image of the inside of the large intestine. Patent Document 1 also discloses a view of determining portions that are located within the field of view of an imaging element, visible in the captured image, and under good imaging conditions as analyzable portions, and determining other portions as unanalyzable portions. Patent Document 1 also discloses a view of detecting, among the unanalyzable portions, portions that are highly likely to be ultimately overlooked as missed portions.

[0004] International Publication No. WO2021 / 171464

[0005] However, Patent Document 1 does not disclose a specific method for updating and presenting information indicating whether or not an area has actually been observed during an endoscopic examination in real time.

[0006] Therefore, with the technology disclosed in Patent Document 1, for example, there is a problem that, during an endoscopic examination, the number or frequency of operations required to observe areas that are not being observed increases, which may impose an excessive burden on the surgeon performing the endoscopic examination.

[0007] One object of the present disclosure is to provide an endoscopic examination support device that can reduce the burden imposed on an operator performing an endoscopic examination.

[0008] In one aspect of the present disclosure, an endoscopic examination support device comprises: a three-dimensional model generation means for generating a three-dimensional model of a tubular organ in which an endoscopic camera is positioned, based on an endoscopic image obtained by capturing an image of the inside of the tubular organ with the endoscopic camera; an unobserved area detection means for detecting, based on the three-dimensional model, an area that is estimated not to have been observed by the endoscopic camera as an unobserved area; and a display image generation means for generating a display image including at least one of information indicating the position of the unobserved area inside the endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image.

[0009] In another aspect of the present disclosure, an endoscopic examination support method includes generating a three-dimensional model of a tubular organ in which an endoscopic camera is located based on an endoscopic image obtained by capturing an image of the inside of the tubular organ with the endoscopic camera; detecting, based on the three-dimensional model, an area that is estimated not to have been observed by the endoscopic camera as an unobserved area; and generating a display image that includes at least one of information indicating the position of the unobserved area inside the endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image.

[0010] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute the following processes: generate a three-dimensional model of a tubular organ in which an endoscopic camera is located based on an endoscopic image obtained by capturing an image of the inside of the tubular organ with the endoscopic camera; detect, based on the three-dimensional model, an area that is estimated not to have been observed by the endoscopic camera as an unobserved area; and generate a display image that includes at least one of information indicating the position of the unobserved area inside the endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image.

[0011] According to the present disclosure, the burden imposed on the surgeon performing the endoscopic examination can be reduced.

[0012] FIG. 1 is a diagram showing a schematic configuration of an endoscopic examination system according to a first embodiment. FIG. 2 is a block diagram showing the hardware configuration of an endoscopic examination support device according to the first embodiment. FIG. 3 is a block diagram showing the functional configuration of an endoscopic examination support device according to the first embodiment. FIG. 4 is a diagram for explaining a specific example of a display image. FIG. 5 is a diagram for explaining another specific example of a display image. A flowchart showing an example of processing performed in the endoscopic examination support device according to the first embodiment. A block diagram showing the functional configuration of an endoscopic examination support device according to a second embodiment. A flowchart showing an example of processing performed in the endoscopic examination support device according to the second embodiment.

[0013] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0014] <First embodiment> [System configuration] Fig. 1 is a diagram showing the schematic configuration of an endoscopic examination system according to Embodiment 1. As shown in Fig. 1, the endoscopic examination system 100 includes an endoscopic examination support device 1, a display device 2, and an endoscope 3 connected to the endoscopic examination support device 1.

[0015] The endoscopic examination support device 1 acquires from the endoscope 3 a video (hereinafter also referred to as the "endoscopic video Ic") including time-series images obtained by capturing images of a subject during an endoscopic examination, and displays a display image on the display device 2 for review by an operator, such as a doctor, performing the endoscopic examination. Specifically, the endoscopic examination support device 1 acquires from the endoscope 3 a video of the interior of the large intestine obtained during the endoscopic examination as the endoscopic video Ic. Based on an image extracted from the endoscopic video Ic (hereinafter also referred to as the "endoscopic image"), the endoscopic examination support device 1 estimates the distance (hereinafter also referred to as the "depth") between the surface of the large intestine, which is a hollow organ, and an endoscopic camera provided at the distal end 38 of the endoscope 3, as well as changes in the relative orientation of the endoscopic camera. The endoscopic examination support device 1 then generates a 3D model corresponding to the structure of the large intestine (intestinal tract) by performing 3D reconstruction based on the depth and the changes in the relative orientation of the endoscopic camera. Furthermore, the endoscopic examination support device 1 detects difficult-to-observe areas, which are areas estimated to be difficult to observe during endoscopic examination, based on the endoscopic image. The endoscopic examination support device 1 also detects lesion candidate regions, which are regions estimated to be lesion candidates, based on the endoscopic image. The endoscopic examination support device 1 also detects regions in the 3D model that are missing due to incomplete or insufficient 3D reconstruction as missing regions. The endoscopic examination support device 1 also detects at least one of a difficult-to-observe region and a missing region in the 3D model as an unobserved region. The endoscopic examination support device 1 also generates a display image based on the endoscopic image corresponding to the current position of the endoscopic camera and the detection results of the unobserved region, and outputs the generated display image to the display device 2.

[0016] The difficult-to-observe region may include, for example, a region that is difficult to see due to insufficient brightness, a region that is difficult to see due to the intensity of blur, and a region where the state of the mucosal surface cannot be seen due to the presence of residue. Furthermore, the missing region may include, for example, a region hidden by an obstruction in the large intestine, such as a fold, and a region where imaging by the endoscopic camera has not been performed for a predetermined period of time or more. The predetermined period of time may be set to, for example, one second.

[0017] The display device 2 has, for example, a liquid crystal monitor, etc. The display device 2 displays images, etc. output from the endoscopic examination support device 1.

[0018] The endoscope 3 mainly has an operation unit 36 ​​that allows the surgeon to input instructions such as air supply, water supply, angle adjustment, and photography instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that incorporates an endoscopic camera such as an ultra-small imaging element, and a connection unit 39 for connecting to the endoscopic examination support device 1.

[0019] 2 is a block diagram showing the hardware configuration of the endoscopic examination support device according to the first embodiment. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0020] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer. The processor 11 also performs processing related to the generation of a display image based on an endoscopic image included in the endoscopic video Ic.

[0021] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and nonvolatile memories that store information necessary for processing by the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs for the endoscopic examination support device 1 to execute each process in this embodiment.

[0022] Furthermore, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images Ic captured by the endoscope 3 during an endoscopic examination.

[0023] The interface 13 acts as an interface between the endoscopic examination assistance device 1 and an external device. For example, the interface 13 supplies a display image generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic video Ic supplied from the endoscope 3 to the processor 11. The interface 13 also supplies an endoscopic image extracted from the endoscopic video Ic to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0024] The input unit 14 generates an input signal based on an operation by the surgeon. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also include a built-in pump for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0025] The DB 17 stores endoscopic images and the like acquired during past endoscopic examinations of the subject. The DB 17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory. Instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be acquired from the server via communication.

[0026] The endoscopic examination support device 1 may also be provided with a sensor, such as a magnetic sensor, that can measure the rotation and translation of the endoscopic camera.

[0027] 3 is a block diagram showing the functional configuration of the endoscopic examination support device according to the first embodiment. The endoscopic examination support device 1 functionally includes a depth estimation unit 21, a camera posture estimation unit 22, a three-dimensional reconstruction unit 23, a difficult-to-observe region detection unit 24, an unobserved region detection unit 25, a lesion candidate detection unit 26, and a display image generation unit 27.

[0028] The depth estimation unit 21 performs a process of estimating depth from an endoscopic image using a trained image recognition model, etc. That is, the depth estimation unit 21 functions as a distance estimation means, and can estimate the distance between the surface of a hollow organ and an endoscopic camera placed inside the hollow organ based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera. The depth estimation unit 21 also outputs the depth estimated by the above-mentioned process to the three-dimensional reconstruction unit 23.

[0029] The camera posture estimation unit 22 performs a process of estimating the rotation and translation of the endoscopic camera from the shooting point of the first endoscopic image to the shooting point of the second endoscopic image (i.e., the change in the relative posture of the endoscopic camera; hereinafter, simply referred to as "camera posture change") using, for example, two temporally consecutive endoscopic images. The camera posture estimation unit 22 also performs a process of estimating the camera posture change using, for example, a trained image recognition model. That is, the camera posture estimation unit 22 functions as a posture change estimation unit and can estimate the change in the relative posture of the endoscopic camera based on endoscopic images obtained by capturing the inside of a tubular organ with the endoscopic camera. The camera posture estimation unit 22 also outputs the camera posture change estimated by the above-mentioned process to the three-dimensional reconstruction unit 23. The camera posture estimation unit 22 may also estimate the camera posture change using measurement data obtained from a magnetic sensor or the like.

[0030] Here, the image recognition models used in the depth estimation unit 21 and the camera posture estimation unit 22 are machine learning models that have been trained in advance to estimate depth and camera posture changes from endoscopic images. Hereinafter, these will also be referred to as the "depth estimation model" and the "camera posture estimation model." The depth estimation model and the camera posture estimation model can be generated by so-called supervised learning.

[0031] For example, training data in which depths are assigned as correct labels to endoscopic images is used to train the depth estimation model. The endoscopic images and depths used for training are collected in advance from an endoscopic camera and a ToF (Time of Flight) sensor provided on the endoscope. That is, pairs of RGB images obtained by the endoscopic camera and depths obtained by the ToF sensor are created as training data, and training is performed using the created training data.

[0032] Furthermore, training data in which changes in camera posture are assigned as correct labels to endoscopic images is used to train the camera posture estimation model. In this case, changes in camera posture can be acquired using a sensor capable of detecting rotation and translation, such as a magnetic sensor. That is, pairs of RGB images acquired by an endoscopic camera and changes in camera posture acquired by a sensor are created as training data, and training is performed using the training data.

[0033] The training data used to train the depth estimation model and the camera pose estimation model may be created from a simulated image of an endoscope using computer graphics (CG). This allows a large amount of training data to be created quickly. The machine learning device uses the training data to learn the relationship between the endoscopic image and changes in depth and camera pose, thereby generating the depth estimation model and the camera pose estimation model.

[0034] The depth estimation model and the camera pose estimation model may be generated by self-supervised learning. For example, in self-supervised learning, training data is generated using motion parallax. Specifically, in self-supervised learning, ｉ and endoscopic image I ｊ and the endoscopic image I ｉ A Depth CNN (Convolutional Neural Network) that estimates the depth from the endoscopic image I ｉ and the endoscopic image I ｊ Based on the depth and relative pose estimated by each of them, the endoscopic image I ｉ Endoscopic image I ｊ is reconstructed (this is called the "endoscopic image I"). ｉ→ｊ The reconstructed endoscopic image I ｉ→ｊ and actual endoscopic image I ｊ The model is trained using the difference between and as the loss.

[0035] The 3D reconstruction unit 23 generates a 3D model corresponding to the structure of the large intestine (intestinal tract) during endoscopic examination by performing 3D reconstruction processing based on the depth obtained from the depth estimation unit 21 and the change in relative posture of the endoscopic camera obtained from the camera posture estimation unit 22. Then, the 3D reconstruction unit 23 outputs the 3D model, the change in relative posture of the endoscopic camera, and the position of the endoscopic camera to the unobserved region detection unit 25.

[0036] That is, the three-dimensional model generating means of this embodiment includes a depth estimation unit 21, a camera attitude estimation unit 22, and a three-dimensional reconstruction unit 23.

[0037] The difficult-to-observe region detection unit 24 detects, for example, regions in the endoscopic image that correspond to at least one of a region where the brightness is below a predetermined value, a region where the amount of blur is above a predetermined value, and a region where residue exists as difficult-to-observe regions. That is, the difficult-to-observe region detection unit 24 detects, based on the endoscopic image, regions inside the tubular organ that are estimated to be difficult to observe with the endoscopic camera as difficult-to-observe regions. The difficult-to-observe region detection unit 24 then outputs the detection results of the difficult-to-observe regions to the unobserved region detection unit 25.

[0038] The unobserved region detection unit 25 detects missing regions in the three-dimensional model based on the relative posture change of the endoscopic camera, the position of the endoscopic camera, and the three-dimensional model. Specifically, the unobserved region detection unit 25 detects, as missing regions, at least one of a region in the three-dimensional model that is hidden by an obstruction such as a fold and a region where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or more. The observed region detection unit 26 detects, as missing regions, a region in the three-dimensional model acquired from the three-dimensional reconstruction unit 23 within the last five seconds that has not been three-dimensionally reconstructed continuously for one second or more. The unobserved region detection unit 25 also performs processing to identify regions in the three-dimensional model generated by the three-dimensional reconstruction unit 23 that correspond to the detection results of the difficult-to-observe regions obtained from the difficult-to-observe region detection unit 24. The unobserved region detection unit 25 also detects difficult-to-observe regions and missing regions in the three-dimensional model as unobserved regions. That is, the unobserved region detection unit 25 detects, based on a three-dimensional model of the hollow organ where the endoscopic camera is located, regions that are estimated not to have been observed by the endoscopic camera as unobserved regions. Furthermore, the unobserved region detection unit 25 can obtain the most recent detection results based on the observation history of the large intestine (intestinal tract) by the endoscopic camera as the detection results of the unobserved regions in the three-dimensional model. The unobserved region detection unit 25 then outputs the relative posture change of the endoscopic camera, the position of the endoscopic camera, the three-dimensional model, and the detection results of the unobserved regions to the display image generation unit 27.

[0039] Lesion candidate detection unit 26 uses a trained image recognition model or the like to detect lesion candidate regions in endoscopic images that are estimated to be lesion candidates. Specifically, lesion candidate detection unit 26 detects, for example, regions including polyps as lesion candidate regions. That is, lesion candidate detection unit 26 detects lesion candidate regions that are estimated to be lesion candidates based on endoscopic images obtained by capturing the interior of a hollow organ with an endoscopic camera. Then, lesion candidate detection unit 26 outputs the detection results of the lesion candidate regions to display image generation unit 27.

[0040] During an endoscopic examination, the display image generation unit 27 generates a display image based on the endoscopic image, the relative change in the posture of the endoscopic camera, the position of the endoscopic camera, the detection result of a lesion candidate region, the three-dimensional model, and the detection result of an unobserved region in the three-dimensional model, and outputs the generated display image to the display device 2. In addition, the display image generation unit 27 sets the display state (display on / off, etc.) of each piece of information included in the display image.

[0041] The display image may include at least one of information indicating the position of an unobserved region within the endoscopic image, which corresponds to the field of view of the endoscopic camera, and information indicating the direction of the unobserved region outside the endoscopic image, which corresponds to the field of view of the endoscopic camera. Such information can be generated, for example, using detection results of unobserved regions accumulated during an endoscopic examination. That is, the display image generator 27 generates a display image including at least one of information indicating the position of an unobserved region within an endoscopic image obtained by capturing an image of the interior of a tubular organ with the endoscopic camera, and information indicating the direction of the unobserved region outside the endoscopic image. Furthermore, when the display image generator 27 detects that an unobserved region within the endoscopic image has been continuously captured for a predetermined period of time or more, it transitions the display of the information indicating the position of the unobserved region from on to off.

[0042] The displayed image may also include at least one of information indicating the position of the lesion candidate region within the endoscopic image, information indicating the direction of the lesion candidate region outside the endoscopic image, and information indicating the latest detection result of the lesion candidate region. Such information can be generated, for example, using the detection results of the lesion candidate region accumulated during the endoscopic examination.

[0043] According to this embodiment, for example, if multiple unobserved areas detected during an endoscopic examination are located outside the endoscopic image, the displayed image may include information indicating the direction of the unobserved area that is closest to the current position of the endoscopic camera among the multiple unobserved areas, or the unobserved area that has the largest area among the multiple unobserved areas.

[0044] Furthermore, according to this embodiment, for example, if multiple lesion candidate areas detected during an endoscopic examination are located outside the endoscopic image, the displayed image may include information indicating the direction of one of the multiple lesion candidate areas that is closest to the current position of the endoscopic camera.

[0045] [Display Example] Next, a description will be given of a specific example of a display image displayed on the display device 2. Fig. 4 is a diagram for explaining a specific example of a display image.

[0046] 4 is an image displayed on the display device 2 during an endoscopic examination. The display image DA includes an endoscopic image 41, a lesion candidate image 42, unobserved direction indicators 43A and 43B, and a lesion direction indicator 44.

[0047] The endoscopic image 41 is an image included in the endoscopic video Ic obtained during the endoscopic examination. The endoscopic image 41 includes an object within the field of view of the endoscopic camera at its current position and is updated as the endoscopic camera moves. The endoscopic image 41 also includes unobserved area masks 41A and 41B, which are information indicating the positions of unobserved areas within the endoscopic image 41.

[0048] The unobserved region mask 41A is displayed in such a manner as to cover regions within the endoscopic image 41 that have not been imaged by the endoscopic camera for a predetermined period of time or more. The unobserved region mask 41A is also erased from the endoscopic image 41, for example, when image capture by the endoscopic camera has been performed for a predetermined period of time or more.

[0049] The unobserved region mask 41B is displayed in a manner that covers regions within the endoscopic image 41 that are difficult to see due to insufficient brightness. Furthermore, the unobserved region mask 41B continues to be displayed, for example, if the brightness in the endoscopic image 41 is equal to or less than a predetermined value, even if imaging by the endoscopic camera is continued for a predetermined period of time or more. Furthermore, the unobserved region mask 41B is erased from the endoscopic image 41, for example, if imaging by the endoscopic camera is continued for a predetermined period of time or more after it is detected that the brightness in the endoscopic image 41 has exceeded the predetermined value.

[0050] Lesion candidate image 42 is smaller in size than endoscopic image 41 and is positioned to the right of endoscopic image 41. Lesion candidate image 42 is an image generated by superimposing lesion position information 42A on another endoscopic image that was acquired before endoscopic image 41 was acquired.

[0051] Lesion location information 42A is displayed as information indicating the latest detection result of the lesion candidate region. In the display example of Fig. 4, lesion location information 42A is displayed as a circular marker surrounding the periphery of the lesion candidate region.

[0052] The unobserved direction indicators 43A and 43B are displayed as information indicating the direction of an unobserved area outside the endoscopic image 41.

[0053] 4, an unobserved direction indicator 43A having a mark indicating the upward direction is displayed in a position adjacent to the top end of the endoscopic image 41, and an unobserved direction indicator 43B having a mark indicating the leftward direction is displayed in a position adjacent to the left end of the endoscopic image 41. In other words, the unobserved direction indicators 43A and 43B in Fig. 4 can inform the surgeon that an unobserved area outside the endoscopic image 41 exists in the upper left direction relative to the current position of the endoscopic camera.

[0054] In this embodiment, either one of the unobserved direction indicators 43A and 43B may be displayed. Specifically, for example, when the unobserved direction indicator 43A is displayed but the unobserved direction indicator 43B is not displayed, the surgeon can be informed that an unobserved area outside the endoscopic image 41 is located above the current position of the endoscopic camera. Also, for example, when the unobserved direction indicator 43B is displayed but the unobserved direction indicator 43A is not displayed, the surgeon can be informed that an unobserved area outside the endoscopic image 41 is located to the left of the current position of the endoscopic camera.

[0055] Furthermore, in this embodiment, indicators similar to the unobserved direction indicators 43A and 43B may be further displayed, for example, at positions adjacent to the bottom edge of the endoscopic image 41 and at positions adjacent to the right edge of the endoscopic image 41. In such a case, the surgeon can be informed that an unobserved area outside the endoscopic image 41 exists in one of eight directions (upper, upper right, right, lower right, lower, lower left, left, and upper left) relative to the current position of the endoscopic camera.

[0056] The lesion direction indicator 44 is displayed as information indicating the direction of a lesion candidate region outside the endoscopic image 41 .

[0057] 4, a lesion direction indicator 44 having a mark indicating the left direction is displayed adjacent to the left edge of the endoscopic image 41. That is, the lesion direction indicator 44 in Fig. 4 can inform the surgeon that a lesion candidate region outside the endoscopic image 41 exists to the left of the current position of the endoscopic camera.

[0058] In this embodiment, indicators similar to the lesion direction indicator 44 may be further displayed, for example, at positions adjacent to the top end of the endoscopic image 41, the bottom end of the endoscopic image 41, and the right end of the endoscopic image 41. In such cases, the surgeon can be informed that a lesion candidate region outside the endoscopic image 41 exists in one of eight directions (upper, upper right, right, lower right, lower, lower left, left, and upper left) relative to the current position of the endoscopic camera.

[0059] According to the display image DA of Fig. 4 , during an endoscopic examination, it is possible to simultaneously display the position of an unobserved region within the endoscopic image 41, the direction of an unobserved region outside the endoscopic image 41, and the direction of a suspected lesion region outside the endoscopic image 41. Furthermore, according to the display image DA of Fig. 4 , the display of the unobserved region masks 41A and 41B transitions from on to off depending on the observation status during the endoscopic examination. Furthermore, according to the display image DA of Fig. 4 , it is possible to set the display of an indicator indicating the direction of an unobserved region outside the endoscopic image 41 to on or off, and it is also possible to set the display of an indicator indicating the direction of a suspected lesion region outside the endoscopic image 41 to on or off depending on the position and / or orientation of the endoscopic camera during the endoscopic examination.

[0060] Meanwhile, in this embodiment, instead of the display image DA shown in Fig. 4, for example, a display image DB as shown in Fig. 5 may be displayed on the display device 2. Fig. 5 is a diagram for explaining another specific example of a display image. Note that, for the sake of simplicity, hereinafter, specific descriptions of parts to which the above-described configurations can be applied will be omitted as appropriate.

[0061] 5 is an image displayed on the display device 2 during an endoscopic examination. The display image DB also includes an endoscopic image 51, a lesion candidate image 42, unobserved direction indicators 43A and 43B, a lesion direction indicator 44, and an unobserved region confirmation image 55.

[0062] The endoscopic image 51 corresponds to an image obtained by removing the unobserved region masks 41A and 41B from the endoscopic image 41.

[0063] The unobserved region check image 55 corresponds to an image obtained by reducing the endoscopic image 51 and adding information indicating the position of the unobserved region within the endoscopic image 51. The unobserved region check image 55 is located to the right of the endoscopic image 51 and below the lesion candidate image 42. The unobserved region check image 55 is updated simultaneously with the update of the endoscopic image 51. The unobserved region check image 55 also includes unobserved region masks 55A and 55B, which are information indicating the position of the unobserved region within the endoscopic image 51.

[0064] The unobserved region mask 55A is displayed in such a manner as to cover regions within the endoscopic image 51 that have not been imaged by the endoscopic camera for a predetermined period of time or more. The unobserved region mask 55A is also erased from the unobserved region confirmation image 55, for example, when image capture by the endoscopic camera has been performed for a predetermined period of time or more.

[0065] The unobserved region mask 55B is displayed in a manner that covers regions within the endoscopic image 51 that are difficult to see due to insufficient brightness. Furthermore, the unobserved region mask 55B continues to be displayed, for example, if the brightness in the endoscopic image 51 is equal to or less than a predetermined value, even if imaging by the endoscopic camera is continued for a predetermined period of time or more. Furthermore, the unobserved region mask 55B is erased from the unobserved region check image 55, for example, if imaging by the endoscopic camera is continued for a predetermined period of time or more after it is detected that the brightness in the endoscopic image 51 has exceeded the predetermined value.

[0066] According to the display image DB of Fig. 5, it is possible to display information similar to the information included in the display image DA of Fig. 4 while maintaining a state in which the entire endoscopic image obtained during the endoscopic examination can be confirmed in detail. Furthermore, according to the display image DB of Fig. 5, it is possible to change the display state (on / off of the display) of each piece of information in the same manner as the display image DA of Fig. 4.

[0067] [Processing Flow] Next, a description will be given of the flow of processing performed in the endoscopic examination support device according to the first embodiment. Fig. 6 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the first embodiment.

[0068] First, the endoscopic examination support device 1 estimates the depth from an endoscopic image obtained during the endoscopic examination (step S11).

[0069] Next, the endoscopic examination support device 1 estimates the change in camera posture from two temporally consecutive endoscopic images obtained during the endoscopic examination (step S12).

[0070] Next, the endoscopic examination support device 1 performs a three-dimensional reconstruction process based on the depth obtained in step S11 and the camera posture change obtained in step S12, thereby generating a three-dimensional model corresponding to the structure of the large intestine (intestinal tract) during the endoscopic examination (step S13).

[0071] Next, the endoscopic examination support device 1 detects a difficult-to-observe region based on the endoscopic image obtained during the endoscopic examination (step S14).

[0072] Next, the endoscopic examination support device 1 detects missing areas in the three-dimensional model generated in step S13 (step S15).

[0073] Next, the endoscopic examination support device 1 detects, as unobserved areas, areas in the three-dimensional model generated in step S13 that correspond to the difficult-to-observe areas detected in step S14 and areas that correspond to the missing areas detected in step S15 (step S16).

[0074] Next, the endoscopic examination support device 1 detects a lesion candidate region based on the endoscopic image obtained during the endoscopic examination (step S17).

[0075] Next, based on the detection result of the unobserved region obtained in step S16 and the detection result of the suspected lesion region obtained in step S17, the endoscopic examination support device 1 generates a display image including at least one of information indicating the position of the unobserved region within the endoscopic image obtained during the endoscopic examination and information indicating the direction of the unobserved region outside the endoscopic image, and information indicating the direction of the suspected lesion region outside the endoscopic image (step S18).The display image generated in step S18 is then displayed on the display device 2.

[0076] In this embodiment, the process of step S12 may be executed before step S11, or the process of step S11 may be executed simultaneously with the process of step S12.

[0077] As described above, according to this embodiment, the display state of information indicating the position of an unobserved area within an endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image can be changed depending on the position and / or orientation of the endoscopic camera during the endoscopic examination. Furthermore, as described above, according to this embodiment, for example, by continuously capturing an unobserved area within an endoscopic image for a predetermined period of time or more, the display of the information indicating the position of the unobserved area can be transitioned from on to off. Therefore, according to this embodiment, the burden imposed on the surgeon performing the endoscopic examination can be reduced. This can also be used to support the user's decision-making, etc.

[0078] Second Embodiment FIG. 7 is a block diagram showing the functional configuration of an endoscopic examination support device according to a second embodiment.

[0079] The endoscopic examination support device 70 according to this embodiment has the same hardware configuration as the endoscopic examination support device 1. The endoscopic examination support device 70 also includes a three-dimensional model generation unit 71, an unobserved region detection unit 72, and a display image generation unit 73.

[0080] FIG. 8 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the second embodiment.

[0081] The three-dimensional model generating means 71 generates a three-dimensional model of the hollow organ in which the endoscope camera is placed, based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscope camera (step S71).

[0082] The unobserved region detecting means 72 detects, based on the three-dimensional model, a region that is estimated not to have been observed by the endoscopic camera as an unobserved region (step S72).

[0083] The display image generating means 73 generates a display image including at least one of information indicating the position of an unobserved area inside the endoscopic image and information indicating the direction of an unobserved area outside the endoscopic image (step S73).

[0084] According to this embodiment, the burden imposed on the surgeon performing the endoscopic examination can be reduced.

[0085] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0086] (Supplementary Note 1) An endoscopic examination support device comprising: a three-dimensional model generation means for generating a three-dimensional model of a hollow organ in which an endoscopic camera is placed, based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera; an unobserved region detection means for detecting, based on the three-dimensional model, a region that is estimated not to have been observed by the endoscopic camera as an unobserved region; and a display image generation means for generating a display image including at least one of information indicating the position of the unobserved region inside the endoscopic image and information indicating the direction of the unobserved region outside the endoscopic image.

[0087] (Appendix 2) The endoscopic examination support device described in Appendix 1, wherein the unobserved area detection means detects at least one of a difficult-to-observe area, which is an area inside the tubular organ that is estimated to be difficult to observe using the endoscopic camera, and a missing area, which is an area where a gap has occurred in the three-dimensional model, as the unobserved area.

[0088] (Supplementary Note 3) The difficult-to-observe area is an area in the endoscopic image that corresponds to at least one of an area where brightness is below a predetermined value, an area where the amount of blur is above a predetermined value, and an area where residue exists, and the unobserved area detection means detects an area in the three-dimensional model that corresponds to the difficult-to-observe area as the unobserved area.

[0089] (Appendix 4) The endoscopic examination support device described in Appendix 2, wherein the missing area corresponds to at least one of an area in the three-dimensional model that is hidden by an obstruction within the tubular organ and an area where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or more.

[0090] (Appendix 5) The endoscopic examination support device described in Appendix 1, wherein the display image generating means transitions the display of information indicating the position of the unobserved area within the endoscopic image from on to off when it detects that the unobserved area within the endoscopic image has been continuously imaged for a predetermined period of time or more.

[0091] (Appendix 6) The endoscopic examination support device according to Appendix 1 further comprises a lesion candidate detection means for detecting a lesion candidate region, which is a region estimated to be a lesion candidate, based on the endoscopic image using a machine-learned model, and the display image generation means generates the display image including information indicating the direction of the lesion candidate region outside the endoscopic image.

[0092] (Supplementary Note 7) The endoscopic examination support device according to Supplementary Note 6, wherein the display image generating means generates the display image including information indicating the latest detection result of the lesion candidate region.

[0093] (Appendix 8) An endoscopic examination support method that generates a three-dimensional model of a tubular organ in which an endoscopic camera is located based on an endoscopic image obtained by capturing an image of the inside of the tubular organ with the endoscopic camera, detects an area that is estimated not to have been observed by the endoscopic camera as an unobserved area based on the three-dimensional model, and generates a display image that includes at least one of information indicating the position of the unobserved area inside the endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image.

[0094] (Appendix 9) A recording medium having recorded thereon a program that causes a computer to execute the following processes: based on an endoscopic image obtained by capturing an image of the inside of a hollow organ with an endoscopic camera, generate a three-dimensional model of the hollow organ in which the endoscopic camera is located; based on the three-dimensional model, detect an area that is estimated not to have been observed by the endoscopic camera as an unobserved area; and generate a display image that includes at least one of information indicating the position of the unobserved area inside the endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image.

[0095] This application claims priority based on International Application PCT / JP2022 / 029426, filed August 1, 2022, the disclosure of which is incorporated herein in its entirety.

[0096] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0097] REFERENCE SIGNS LIST 1 Endoscopy support device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 21 Depth estimation unit 22 Camera attitude estimation unit 23 Three-dimensional reconstruction unit 24 Difficult-to-observe region detection unit 25 Unobserved region detection unit 26 Lesion candidate detection unit 27 Display image generation unit 100 Endoscopy system

Claims

1. a three-dimensional model generating means for generating a three-dimensional model of the hollow organ in which the endoscope camera is disposed, based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscope camera; an unobserved region detection means for detecting a region that is estimated not to have been observed by the endoscope camera as an unobserved region based on the three-dimensional model; a display image generating means for generating a display image including at least one of information indicating a position of the unobserved region inside the endoscopic image and information indicating a direction of the unobserved region outside the endoscopic image; An endoscopic examination support device comprising:

2. The endoscopic examination support device according to claim 1, wherein the unobserved area detection means detects as the unobserved area at least one of a difficult-to-observe area, which is an area inside the tubular organ that is estimated to be difficult to observe using the endoscopic camera, and a missing area, which is an area where a gap has occurred in the three-dimensional model.

3. the difficult-to-observe region corresponds to at least one of a region in the endoscopic image whose brightness is equal to or lower than a predetermined value, a region whose amount of blur is equal to or higher than a predetermined value, and a region in which residue is present, 3. The endoscopic examination support device according to claim 2, wherein the unobserved region detection means detects, as the unobserved region, a region in the three-dimensional model that corresponds to the difficult-to-observe region.

4. The endoscopic examination support device according to claim 2, wherein the missing area corresponds to at least one of an area in the three-dimensional model that is hidden by an obstruction within the tubular organ and an area where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or more.

5. The endoscopic examination support device according to claim 1, wherein the display image generating means transitions a display of information indicating the position of the unobserved area within the endoscopic image from on to off when it detects that the unobserved area within the endoscopic image has been continuously imaged for a predetermined period of time or more.

6. The method further includes a lesion candidate detection means for detecting a lesion candidate region, which is a region estimated to be a lesion candidate, based on the endoscopic image by a machine learning model; The endoscopic examination support device according to claim 1 , wherein the display image generating means generates the display image including information indicating a direction of the lesion candidate region outside the endoscopic image.

7. The endoscopic examination support device according to claim 6 , wherein the display image generating means generates the display image including information indicating a latest detection result of the lesion candidate region.

8. A computer-implemented method for assisting endoscopic examination, comprising: generating a three-dimensional model of the hollow organ in which the endoscopic camera is disposed based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera; Based on the three-dimensional model, a region that is estimated not to have been observed by the endoscope camera is detected as an unobserved region; An endoscopic examination support method for generating a display image including at least one of information indicating the position of the unobserved area within the endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image.

9. generating a three-dimensional model of the hollow organ in which the endoscopic camera is disposed based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera; Based on the three-dimensional model, a region that is estimated not to have been observed by the endoscope camera is detected as an unobserved region; A program that causes a computer to execute a process of generating a display image that includes at least one of information indicating the position of the unobserved area within the endoscopic image and information indicating the direction of the unobserved area outside the endoscopic image.