Endoscopic examination assistance device, endoscopic examination assistance method, and recording medium

Abstract

In this endoscopic examination assistance device, a three-dimensional model generation means generates a three-dimensional point cloud and a three-dimensional model of a hollow organ in which an endoscope camera is placed, on the basis of an endoscopic image obtained by capturing the inside of the hollow organ using the endoscope camera. An unobserved region detection means detects a region estimated not to have been captured using the endoscope camera as an unobserved region, on the basis of the three-dimensional point cloud and the three-dimensional model. A drawing means displays a diagram showing the positional relationship between the hollow organ, the unobserved region inside the hollow organ, and the endoscope camera. The endoscopic examination assistance device is capable of assisting decision making by users in the medical field.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

[0001] The present disclosure relates to processing endoscopy images.

[0002] In an endoscopic examination, an endoscope is inserted into a hollow organ such as the stomach or large intestine to observe the interior, but during this examination, there may be an unobserved region that is not observed by the examiner. From this perspective, for example, Patent Document 1 proposes an image processing device that determines an unobserved region.

[0003] JP 2024-008815 A

[0004] However, the technique of Patent Document 1 does not indicate the positional relationship between the unobserved region and the endoscopic camera, so the examiner may not be able to appropriately correct the position or direction of the endoscopic camera.

[0005] One object of the present disclosure is to provide an endoscopic examination support device that makes it possible to easily grasp the positional relationship between an unobserved area and an endoscopic camera during endoscopic examination.

[0006] In one aspect of the present disclosure, an endoscopic examination support device comprises: a three-dimensional model generation means for generating a three-dimensional point cloud and a three-dimensional model of a hollow organ in which an endoscopic camera is positioned, based on endoscopic images obtained by capturing images of the inside of the hollow organ with the endoscopic camera; an unobserved area detection means for detecting, based on the three-dimensional point cloud and the three-dimensional model, an area that is estimated not to have been photographed by the endoscopic camera, as an unobserved area; and a drawing means for displaying a diagram showing the positional relationship between the hollow organ, the unobserved area inside the hollow organ, and the endoscopic camera.

[0007] In another aspect of the present disclosure, a method for supporting endoscopic examination includes generating a three-dimensional point cloud and a three-dimensional model of a tubular organ where an endoscopic camera is positioned based on endoscopic images obtained by capturing an image of the inside of the tubular organ with the endoscopic camera; detecting, based on the three-dimensional point cloud and the three-dimensional model, an area that is estimated not to have been photographed by the endoscopic camera as an unobserved area; and displaying a diagram showing the positional relationship between the tubular organ, the unobserved area inside the tubular organ, and the endoscopic camera.

[0008] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute the following processes: generate a three-dimensional point cloud and a three-dimensional model of the tubular organ in which the endoscopic camera is located based on endoscopic images obtained by capturing images of the inside of the tubular organ with the endoscopic camera; detect, based on the three-dimensional point cloud and the three-dimensional model, areas that are estimated not to have been captured by the endoscopic camera as unobserved areas; and display a diagram showing the positional relationship between the tubular organ, the unobserved areas inside the tubular organ, and the endoscopic camera.

[0009] According to the present disclosure, it is possible to easily grasp the positional relationship between an unobserved area and an endoscopic camera.

[0010] 1 is a block diagram showing a schematic configuration of an endoscopic examination system; FIG. 2 is a block diagram showing a hardware configuration of an endoscopic examination support device; FIG. 3 is a block diagram showing a functional configuration of an endoscopic examination support device; FIG. 4 shows an example of display data creation by a plan view drawing unit; FIG. 5 shows another example of display data creation by a plan view drawing unit; FIG. 6 shows an example of display data creation by a 3D model drawing unit; FIG. 7 shows another example of display data creation by a 3D model drawing unit; FIG. 8 shows an example of display by a display device; FIG. 9 shows another example of display by a display device; FIG. 10 is a flowchart of processing by an endoscopic examination support device; FIG. 11 is a block diagram showing a functional configuration of another endoscopic examination support device according to the present disclosure; FIG. 12 is a flowchart of processing by another endoscopic examination support device according to the present disclosure.

[0011] Preferred embodiments of the present disclosure will now be described with reference to the drawings. First Embodiment System Configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. The endoscopic examination system 100 detects an observation area and an unobserved area during an examination (including treatment) using an endoscope. The endoscopic examination system 100 then displays the positional relationship between the unobserved area and the endoscopic camera, as well as the size and shape of the unobserved area, in a manner that makes it easy to understand. Note that the observation area in this embodiment refers to an area that has been photographed by the endoscopic camera. Furthermore, the unobserved area in this embodiment refers to an area that the endoscopic camera can photograph but has not yet photographed.

[0012] As shown in FIG. 1 , the endoscopic examination system 100 mainly includes an endoscopic examination support device 1, a display device 2, and an endoscope 3 connected to the endoscopic examination support device 1.

[0013] The endoscopic examination support device 1 acquires images (hereinafter also referred to as "endoscopic images Ic") captured by the endoscope 3 during endoscopic examination from the endoscope 3 and displays display data on the display device 2 for the examiner (physician) performing the endoscopic examination to review. Specifically, the endoscopic examination support device 1 acquires, as endoscopic images Ic, video images of the inside of an organ captured by the endoscope 3 during endoscopic examination. The endoscopic examination support device 1 sequentially extracts frame images from the video endoscopic images Ic and estimates the distance between the surface of the colon and the endoscopic camera (hereinafter also referred to as "depth") and the change in the relative attitude of the endoscopic camera (hereinafter also referred to as "camera attitude"). The endoscopic examination support device 1 then creates a three-dimensional point cloud and a three-dimensional model of the organ from the depth map and the camera attitude, and detects observed and unobserved areas. The endoscopic examination support device 1 creates display data indicating the positional relationship between the unobserved area and the endoscopic camera, and displays the data on the display device 2.

[0014] The display device 2 is a display or the like that displays a predetermined image based on a display signal supplied from the endoscopic examination support device 1 .

[0015] The endoscope 3 mainly has an operation unit 36 ​​that allows the examiner to input instructions such as air supply, water supply, angle adjustment, and photography instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that has an endoscopic camera such as a miniature imaging element built in, and a connection unit 39 for connecting to the endoscopic examination support device 1.

[0016] The following explanation will be based mainly on the processing involved in an endoscopic examination of the large intestine, but the subject of examination is not limited to the large intestine, and may be any part of the digestive tract (digestive organs) such as the stomach, esophagus, small intestine, or duodenum.

[0017] 2 shows the hardware configuration of the endoscopic examination support device 1. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0018] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0019] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and nonvolatile memories that store information necessary for processing by the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs for the endoscopic examination support device 1 to execute each process in this embodiment.

[0020] Furthermore, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images Ic captured by the endoscope 3 during an endoscopic examination.

[0021] The interface 13 performs interface operations between the endoscopic examination assistance device 1 and an external device. For example, the interface 13 supplies the display data Id generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating an endoscopic image Ic supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0022] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0023] The DB 17 stores endoscopic images and the like acquired in past endoscopic examinations of the subject. The DB 17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory. Note that instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be acquired from the server via communication.

[0024] The endoscopic examination support device 1 may also be provided with a sensor, such as a magnetic sensor, that can measure the rotation and translation of the endoscopic camera.

[0025] 3 is a block diagram showing the functional configuration of the endoscopic examination support device 1. Functionally, the endoscopic examination support device 1 includes an interface 13, a depth estimation unit 101, a camera posture estimation unit 102, a three-dimensional reconstruction unit 103, an unobserved region detection unit 104, a plan view drawing unit 105, a 3D model drawing unit 106, and a display device 2.

[0026] The depth estimation unit 101, the camera posture estimation unit 102, the three-dimensional reconstruction unit 103, the unobserved area detection unit 104, the plan view drawing unit 105, and the 3D model drawing unit 106 are configured by the processor 11 shown in FIG.

[0027] The interface 13 sequentially inputs the endoscopic images Ic from the endoscope 3. The interface 13 extracts frame images from the endoscopic images Ic and outputs them to the depth estimation unit 101 and the camera posture estimation unit 102.

[0028] The depth estimation unit 101 sequentially estimates depth from input frame images and outputs a depth map to the three-dimensional reconstruction unit 103. The depth estimation unit 101 estimates depth from the frame images using, for example, a depth estimation model described below.

[0029] The camera posture estimation unit 102 sequentially estimates the camera posture from the input frame images and outputs the estimated posture to the three-dimensional reconstruction unit 103. For example, the camera posture estimation unit 102 uses two temporally consecutive frame images to estimate the rotation and translation of the endoscopic camera from the shooting point of the first frame image to the shooting point of the second frame image (i.e., the change in the relative posture of the endoscopic camera; hereinafter, simply referred to as the "camera posture"). Note that the camera posture estimation unit 102 may estimate the camera posture from the frame images using a camera posture estimation model described below, or may estimate the camera posture using measurement data from a magnetic sensor or the like.

[0030] The depth estimation unit 101 and the camera pose estimation unit 102 may estimate the camera pose and depth from multiple frame images while successively improving them by using a method such as DROID-SLAM. DROID-SLAM is a type of Visual SLAM that can estimate the camera pose and depth of each image using multiple frame images as input.

[0031] The three-dimensional reconstruction unit 103 performs a reconstruction process of a three-dimensional point cloud based on the depth map and the camera posture. The three-dimensional reconstruction unit 103 also generates a three-dimensional model of the large intestine based on the depth map and the camera posture or based on the reconstructed three-dimensional point cloud. The three-dimensional model of the large intestine is a three-dimensional model that approximates a cylindrical shape or a shape made up of multiple overlapping cylindrical shapes, and is meshed. The three-dimensional reconstruction unit 103 outputs the three-dimensional point cloud and the three-dimensional model to the unobserved region detection unit 104.

[0032] The unobserved region detection unit 104 detects observed regions and unobserved regions of the large intestine and outputs the results to the plan view drawing unit 105 and the 3D model drawing unit 106. For example, the unobserved region detection unit 104 overlays a three-dimensional point cloud on a three-dimensional model and determines which region of the three-dimensional model the three-dimensional point cloud corresponds to. The unobserved region detection unit 104 then estimates a region that includes the three-dimensional point cloud as an observed region, and estimates a region that does not include the three-dimensional point cloud as an unobserved region. The unobserved region detection unit 104 may output the detection results of the observed region and the unobserved region to both the plan view drawing unit 105 and the 3D model drawing unit 106, or to either one of them.

[0033] The plan view drawing unit 105 creates display data and outputs it to the display device 2. The plan view drawing unit 105 creates the display data by cutting the three-dimensional model at a predetermined position and drawing the position of the endoscopic camera on the cross-sectional view. The plan view drawing unit 105 can set the cutting position as shown in Figures 4 and 5 as the predetermined position.

[0034] 4A and 4B show an example of display data generated by the plan view drawing unit 105. Fig. 4A is a diagram for explaining a cutting position of a three-dimensional model. Fig. 4A includes a three-dimensional model 41, an unobserved region 42, an endoscopic camera 43, and a cutting surface 44.

[0035] The three-dimensional model 41 is a three-dimensional model of the large intestine, and approximates the large intestine with multiple cylinders. The unobserved region 42 is an unobserved region of the large intestine, and is indicated by diagonal lines. The endoscopic camera 43 indicates the current endoscopic camera. For convenience of illustration, in FIG. 4(A), the endoscopic camera 43 is shown outside the three-dimensional model. The cutting plane 44 indicates a cutting plane of the three-dimensional model. The cutting plane 44 is set so as to satisfy all of the following three conditions (conditions 1 to 3): (condition 1) It is a plane parallel to the height direction (Z-axis) of the cylinder. (condition 2) It is a plane passing through the center of gravity of the cylinder. (condition 3) It is a plane with the greatest distance between the center of gravity of the unobserved region 42 and the endoscopic camera.

[0036] 4(B) is a diagram showing display data. The display data in FIG. 4(B) includes a cross-sectional view 45, an unobserved region 46, an endoscopic camera 47, and an arrow 48. The cross-sectional view 45 shows a cross-sectional view when the three-dimensional model 41 is cut along a cutting plane 44. The endoscopic camera 47 indicates the current position of the endoscopic camera. The endoscopic camera 47 is represented by a cone, with the bottom of the cone indicating the lens side of the endoscopic camera. The arrow 48 indicates the direction of the endoscopic camera. In this way, the plan view drawing unit 105 can create display data that allows the positional relationship between the endoscopic camera and the unobserved region to be grasped.

[0037] FIG. 5 shows another example of display data created by the plan view drawing unit 105. This example shows a case where the three-dimensional model is cut at a position different from that shown in FIG. 4. FIG. 5(A) is a diagram for explaining the cutting position of the three-dimensional model. FIG. 5(A) includes a three-dimensional model 41a, an unobserved area 42a, an endoscopic camera 43a, and a cutting surface 44a. In FIG. 5(A), the cutting surface 44a is set to satisfy all of the following three conditions (conditions 4 to 6): (condition 4) It is a plane parallel to the height direction (Z-axis) of the cylinder. (condition 5) It is a plane passing through the center of gravity of the cylinder. (condition 6) It is a plane parallel to the Z-axis direction in the camera coordinate system. In other words, it is a plane on a plane where the endoscopic camera is always facing directly to the side.

[0038] 5(B) is a diagram showing display data. The display data in FIG. 5(B) includes a cross-sectional view 45a, an unobserved region 46a, an endoscopic camera 47a, and an arrow 48a. The cross-sectional view 45a shows a cross section of the three-dimensional model 41a cut along the cutting plane 44a. In this way, the plan view drawing unit 105 can create display data that allows the positional relationship between the endoscopic camera and the unobserved region to be grasped.

[0039] The plan view drawing unit 105 creates either or both of the display data shown in Fig. 4(B) and the display data shown in Fig. 5(B). Note that if there is an unobserved region around the entire circumference of the large intestine, the distance between the center of gravity of the unobserved region 42 and the endoscopic camera cannot be calculated, so the plan view drawing unit 105 creates the display data shown in Fig. 5(B).

[0040] The 3D model rendering unit 106 creates display data and outputs it to the display device 2. The 3D model rendering unit 106 creates the display data by rendering a three-dimensional model of the large intestine as viewed from a predetermined position and the position of the endoscopic camera. As shown in Fig. 6, the 3D model rendering unit 106 sets the camera position in the 3D viewer as a predetermined position.

[0041] 6A and 6B show an example of display data created by the 3D model rendering unit 106. Fig. 6A is a diagram for explaining the camera position in the 3D viewer. Fig. 6A includes a three-dimensional model 61, an unobserved area 62, and a camera position 63.

[0042] The three-dimensional model 61 is a three-dimensional model of the large intestine, approximating the large intestine with multiple cylinders. The unobserved region 62 is an unobserved region of the large intestine, and is indicated by diagonal lines. The camera position 63 is the camera position in the 3D viewer. In FIG. 6A , the camera of the 3D viewer is positioned so as to satisfy all of the following three conditions (conditions 7 to 9). (Condition 7) The unobserved region is positioned in a direction perpendicular to the plane in which it appears largest. Specifically, the 3D model rendering unit 106 performs principal component analysis on the unobserved region, and positions the camera in a direction perpendicular to the plane formed by the two axes of the first and second principal components. (Condition 8) The camera is positioned outside the three-dimensional model so as to view the unobserved region from the front. (Condition 9) The camera is positioned at a distance that allows the entire three-dimensional model to be seen.

[0043] Fig. 6(B) is a diagram showing display data. The display data in Fig. 6(B) includes a three-dimensional model 64, an unobserved area 65, and an endoscopic camera 66. The three-dimensional model 64 shows the three-dimensional model 61 as seen from a camera position 63. The unobserved area 65 is shown by filling in the mesh surfaces of the three-dimensional model 64. The endoscopic camera 66 shows the current position of the endoscopic camera. The endoscopic camera 66 is represented by a cone, with the bottom of the cone indicating the lens side of the endoscopic camera.

[0044] The 3D model rendering unit 106 may also set the predetermined position to a direction perpendicular to the cross-section shown in FIGS. 4 and 5 and at a distance at which the entire three-dimensional model can be seen. FIG. 7 shows another example of display data created by the 3D model rendering unit 106. FIG. 7(A) is a diagram illustrating the camera position in the 3D viewer. FIG. 7(A) includes a three-dimensional model 71, an unobserved region 72, a cross-section 73, and a camera position 74. The cross-section 73 is similar to the cross-section 44 shown in FIG. 4(A) and is set to satisfy all of the above-described conditions 1 to 3. The camera position 74 is the camera position in the 3D viewer, and is positioned perpendicular to the cross-section 73 and at a distance at which the entire three-dimensional model can be seen. FIG. 7(B) is a diagram illustrating display data. The display data in FIG. 7(B) includes a three-dimensional model 75, an unobserved region 76, and an endoscopic camera 77. The three-dimensional model 75 shows the three-dimensional model 71 as seen from the camera position 74.

[0045] In FIG. 7(A), the 3D model rendering unit 106 positions the camera of the 3D viewer based on the cross section shown in FIG. 4(A), but instead, the camera of the 3D viewer may be positioned based on the cross section shown in FIG. 5(A).

[0046] 6(B) and 7(B), the 3D model rendering unit 106 represents unobserved regions by filling in predetermined regions of the three-dimensional model, but instead, the unobserved regions may be represented semi-transparently. For example, if there are unobserved regions around the entire circumference of the large intestine, the 3D model rendering unit 106 represents the unobserved regions semi-transparently. This makes it possible to grasp the position of the endoscopic camera within the three-dimensional model.

[0047] In the above configuration, the interface 13, the depth estimation unit 101, the camera posture estimation unit 102, and the three-dimensional restoration unit 103 are examples of a three-dimensional model generation means, the unobserved area detection unit 104 is an example of an unobserved area detection means, and the floor plan drawing unit 105 and the 3D model drawing unit 106 are examples of a drawing means.

[0048] [Display Example] Next, a display example on the display device 2 will be described.

[0049] FIG. 8 shows an example of a display on the display device 2. In this example, an endoscopic image 91, a lesion history 92, and an unobserved region display area 93 are displayed on the display device 2. The endoscopic image 91 is an endoscopic image Ic during an examination and is updated as the endoscopic camera moves. The lesion history 92 indicates the most recent lesion detected during an endoscopic examination. The unobserved region display area 93 is an area that indicates an unobserved region. In FIG. 8, display data created by the plan view drawing unit 105 is displayed in the unobserved region display area 93. The display in the unobserved region display area 93 is updated at regular intervals.

[0050] FIG. 9 shows another example of display by the display device 2. This example shows an unobserved region represented by a three-dimensional model. In FIG. 9, an endoscopic image 91a, a lesion history 92a, and an unobserved region display area 93a are displayed on the display device 2. The unobserved region display area 93a is an area showing an unobserved region. In FIG. 9, display data created by the 3D model rendering unit 106 is displayed in the unobserved region display area 93a. The display of the unobserved region display area 93a is updated at regular intervals.

[0051] The displays shown in FIGS. 8 and 9 allow the doctor to understand the positional relationship between the unobserved region and the current position of the endoscopic camera.

[0052] 8 and 9, one piece of display data is displayed in the unobserved area display area, but two pieces of display data may be displayed in the unobserved area display area. For example, the display data created by the floor plan drawing unit 105 and the display data created by the 3D model drawing unit 106 may be displayed side by side in the unobserved area display area. Furthermore, the display data shown in FIGS. 4(B) and 5(B) may be displayed side by side in the unobserved area display area, or the display data shown in FIGS. 6(B) and 7(B) may be displayed side by side in the unobserved area display area.

[0053] [Machine Learning Model] Next, a "depth estimation model" and a "camera pose estimation model" will be described. The depth estimation model is a machine learning model that has been trained in advance to estimate depth from an endoscopic image. The camera pose estimation model is a machine learning model that has been trained in advance to estimate camera pose from an endoscopic image. The depth estimation model and the camera pose estimation model can be generated by so-called supervised learning.

[0054] For example, training data in which depths are assigned as correct labels to endoscopic images is used to train the depth estimation model. The endoscopic images and depths used for training are collected in advance from an endoscopic camera and a ToF (Time of Flight) sensor attached to the endoscope.

[0055] For example, training data in which camera posture changes are assigned as correct labels to endoscopic images is used to train the camera posture estimation model. In this case, the camera posture changes can be acquired using a sensor that can detect rotation and translation, such as a magnetic sensor.

[0056] The training data used to train the depth estimation model and the camera pose estimation model may be created from a simulated image of an endoscope using computer graphics (CG). This allows a large amount of training data to be generated quickly. The machine learning device uses the training data to learn the relationship between the endoscopic image and changes in depth and camera pose, thereby generating the depth estimation model and the camera pose estimation model.

[0057] The depth estimation model and the camera pose estimation model may be generated by self-supervised learning. For example, in self-supervised learning, training data is generated using motion parallax. Specifically, in self-supervised learning, ｉ and endoscopic image I ｊ Paired images of and endoscopic image I ｉ Depth CNN (Convolutional Neural Network) that estimates depth from endoscopic image I ｉ and endoscopic image I ｊThen, based on the depth and relative pose estimated by each of them, the endoscopic image I ｉ Endoscopic image I ｊ (This is called the "endoscopic image I" ｉ→ｊ The reconstructed endoscopic image I ｉ→ｊ and actual endoscopic image I ｊ The model is trained using the difference as the loss.

[0058] [Processing Flow] Next, a description will be given of the display processing for performing the above-described display. Fig. 9 is a flowchart of the processing performed by the endoscopic examination support device 1. This processing is realized by the processor 11 shown in Fig. 2 executing a program prepared in advance and operating as each element shown in Fig. 3.

[0059] First, the interface 13 acquires the endoscopic image Ic from the endoscope 3 (step S11). The interface 13 extracts frame images from the endoscopic image Ic and outputs them to the depth estimation unit 101 and the camera posture estimation unit 102.

[0060] Next, the depth estimation unit 101 estimates depth from the input frame images (step S12). The depth estimation unit 101 outputs a depth map to the three-dimensional reconstruction unit 103. Next, the camera posture estimation unit 102 estimates the camera posture from the input frame images (step S13). The camera posture estimation unit 102 outputs the camera posture to the three-dimensional reconstruction unit 103.

[0061] Next, the 3D reconstruction unit 103 performs a 3D point cloud reconstruction process based on the depth map and the camera posture (step S14). The 3D reconstruction unit 103 also generates a 3D model based on the depth map and the camera posture or based on the reconstructed 3D point cloud (step S15). The 3D reconstruction unit 103 outputs the 3D point cloud and the 3D model to the unobserved area detection unit 104.

[0062] Next, the unobserved region detection unit 104 detects observed regions and unobserved regions of the large intestine based on the 3D point cloud and the 3D model (step S16). The unobserved region detection unit 104 outputs the detection result to the plan view drawing unit 105 or the 3D model drawing unit 106.

[0063] Next, the plan view drawing unit 105 creates a plan view of the large intestine that depicts the positional relationship between the unobserved region and the endoscopic camera (step S17).The plan view drawing unit 105 then outputs the created plan view to the display device 2.

[0064] Next, the 3D model rendering unit 106 creates a three-dimensional model of the large intestine that depicts the positional relationship between the unobserved region and the endoscopic camera (step S18). The 3D model rendering unit 106 then outputs the created three-dimensional model to the display device 2. The display device 2 displays the input plan view and three-dimensional model (step S19). In this way, the display shown in Figures 8 and 9 is produced. Note that step S13 may be performed before step S12 or simultaneously with step S12.

[0065] 11 is a block diagram showing the functional configuration of an endoscopic examination support device according to Embodiment 2. The endoscopic examination support device 200 includes a three-dimensional model generation unit 201, an unobserved region detection unit 202, and a drawing unit 203.

[0066] 12 is a flowchart of processing by the endoscopic examination support device of the second embodiment. The three-dimensional model generation means 201 generates a three-dimensional point cloud and a three-dimensional model of the hollow organ where the endoscopic camera is located based on endoscopic images obtained by capturing images of the interior of the hollow organ with the endoscopic camera (step S201). The unobserved region detection means 202 detects, based on the three-dimensional point cloud and the three-dimensional model, regions that are estimated not to have been photographed by the endoscopic camera as unobserved regions (step S202). The drawing means 203 displays a diagram showing the positional relationship between the hollow organ, the unobserved region within the hollow organ, and the endoscopic camera (step S203).

[0067] The endoscopic examination support device 200 of the second embodiment makes it possible to easily grasp the positional relationship between an unobserved region and an endoscope during an endoscopic examination. Furthermore, the endoscopic examination support device 200 can support the decision-making of users (doctors) in the medical field.

[0068] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0069] (Supplementary Note 1) An endoscopic examination support device comprising: a three-dimensional model generation means for generating a three-dimensional point cloud and a three-dimensional model of a hollow organ in which an endoscopic camera is placed, based on endoscopic images obtained by capturing images of the inside of the hollow organ with the endoscopic camera; an unobserved area detection means for detecting, based on the three-dimensional point cloud and the three-dimensional model, an area that is estimated not to have been photographed by the endoscopic camera, as an unobserved area; and a drawing means for displaying a diagram showing the positional relationship between the hollow organ, the unobserved area inside the hollow organ, and the endoscopic camera.

[0070] (Supplementary Note 2) The endoscopic examination support device according to Supplementary Note 1, wherein the drawing means draws the figure as a plan view or a three-dimensional figure, and displays at least one of the plan view and the three-dimensional figure.

[0071] (Appendix 3) The endoscopic examination support device described in Appendix 1, wherein the three-dimensional model generation means generates a three-dimensional model of the tubular organ that approximates a cylindrical structure, and the drawing means cuts the three-dimensional model of the tubular organ along a predetermined cutting plane and draws the unobserved area and the endoscopic camera on a plan view showing the cross-sectional view of the cut plane.

[0072] (Appendix 4) The endoscopic examination support device according to Appendix 3, wherein the predetermined cross-section is parallel to the height direction of the cylinder, passes through the center of gravity of the cylinder, and is the plane where the distance between the center of gravity of the unobserved area and the endoscopic camera is the greatest.

[0073] (Supplementary Note 5) The endoscopic examination support device according to Supplementary Note 3, wherein the predetermined cross section is a plane that is parallel to the height direction of the cylinder, passes through the center of gravity of the cylinder, and is parallel to the z-axis, which is the depth direction in a camera coordinate system.

[0074] (Supplementary Note 6) The endoscopic examination support device according to Supplementary Note 5, wherein the drawing means cuts the three-dimensional model of the tubular organ along the predetermined cutting plane when the unobserved region exists around the entire circumference of the cylinder.

[0075] (Appendix 7) The endoscopic examination support device described in Appendix 1, wherein the three-dimensional model generation means generates a three-dimensional model of the tubular organ that approximates a cylindrical structure, and the drawing means draws the unobserved area and the endoscopic camera on the three-dimensional model of the tubular organ viewed from a predetermined position.

[0076] (Appendix 8) The endoscopic examination support device according to Appendix 7, wherein the predetermined position is a direction perpendicular to a plane in which the unobserved region appears largest, a direction in which the front of the unobserved region is visible, and a distance at which the entire three-dimensional model is visible.

[0077] (Supplementary Note 9) The endoscopic examination support device according to Supplementary Note 8, wherein the drawing means performs principal component analysis on the unobserved region and determines a plane having axes of a first principal component and a second principal component as a plane in which the unobserved region appears largest.

[0078] (Supplementary Note 10) The endoscopic examination support device according to Supplementary Note 7, wherein the predetermined position is in a direction perpendicular to a predetermined cross section of the three-dimensional model and at a distance at which the entire three-dimensional model can be seen.

[0079] (Appendix 11) The endoscopic examination support device according to Appendix 10, wherein the predetermined cross section is parallel to the height direction of the cylinder, passes through the center of gravity of the cylinder, and is a plane on which the distance between the center of gravity of the unobserved area and the endoscopic camera is the greatest.

[0080] (Supplementary Note 12) The endoscopic examination support device according to Supplementary Note 10, wherein the predetermined cross section is a plane that is parallel to a height direction of the cylinder, passes through the center of gravity of the cylinder, and is parallel to a z-axis that is a depth direction in a camera coordinate system.

[0081] (Supplementary Note 13) The endoscopic examination support device according to Supplementary Note 12, wherein the drawing means uses the predetermined cross section when the unobserved region exists around the entire circumference of the cylinder.

[0082] (Supplementary Note 14) The endoscopic examination support device according to Supplementary Note 1, wherein the three-dimensional model generation means estimates depth from the endoscopic image using a first machine learning model that estimates depth from an image, estimates the camera posture of the endoscopic camera from the endoscopic image using a second machine learning model that estimates camera posture from an image, and generates a three-dimensional model of the tubular organ based on the depth and the camera posture.

[0083] (Supplementary Note 15) The endoscopic examination support device according to Supplementary Note 2, wherein the drawing means displays the plan view and the three-dimensional figure side by side.

[0084] (Supplementary Note 16) The endoscopic examination support device according to Supplementary Note 2, wherein the drawing means displays the unobserved region by filling it with a predetermined color.

[0085] (Supplementary Note 17) The endoscopic examination support device according to Supplementary Note 2, wherein the drawing means displays the unobserved region in the three-dimensional figure by making it semitransparent.

[0086] (Supplementary Note 18) A method for supporting endoscopic examination, which generates a three-dimensional point cloud and a three-dimensional model of the hollow organ where the endoscopic camera is located based on endoscopic images obtained by capturing images of the inside of the hollow organ with an endoscopic camera, detects areas that are estimated not to have been photographed by the endoscopic camera as unobserved areas based on the three-dimensional point cloud and the three-dimensional model, and displays a diagram showing the positional relationship between the hollow organ, the unobserved areas inside the hollow organ, and the endoscopic camera.

[0087] (Supplementary Note 19) A recording medium having recorded thereon a program that causes a computer to execute the following processes: based on endoscopic images obtained by capturing images of the inside of a hollow organ with an endoscopic camera, generate a three-dimensional point cloud and a three-dimensional model of the hollow organ where the endoscopic camera is located; based on the three-dimensional point cloud and the three-dimensional model, detect areas that are estimated not to have been photographed by the endoscopic camera as unobserved areas; and display a diagram showing the positional relationship between the hollow organ, the unobserved areas inside the hollow organ, and the endoscopic camera.

[0088] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0089] REFERENCE SIGNS LIST 1 Endoscopy support device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 100 Endoscopy system 101 Depth estimation unit 102 Camera attitude estimation unit 103 Three-dimensional reconstruction unit 104 Unobserved region detection unit 105 Plan view drawing unit 106 3D model drawing unit

Claims

1. An endoscopic examination support device comprising: a three-dimensional model generation means for generating a three-dimensional point cloud and a three-dimensional model of a hollow organ in which an endoscopic camera is placed, based on endoscopic images obtained by capturing images of the inside of the hollow organ with the endoscopic camera; an unobserved area detection means for detecting, based on the three-dimensional point cloud and the three-dimensional model, areas that are estimated not to have been photographed by the endoscopic camera, as unobserved areas; and a drawing means for displaying a diagram showing the positional relationship between the hollow organ, the unobserved areas inside the hollow organ, and the endoscopic camera.

2. An endoscopic examination support device according to claim 1, wherein said drawing means draws said figure as a plan view or a three-dimensional figure, and displays at least one of said plan view and said three-dimensional figure.

3. The endoscopic examination support device according to claim 1, wherein the three-dimensional model generation means generates a three-dimensional model of the tubular organ that approximates a cylindrical structure, and the drawing means cuts the three-dimensional model of the tubular organ along a predetermined cutting plane and draws the unobserved area and the endoscopic camera on a plan view showing the cross-sectional view of the cut section.

4. An endoscopic examination support device as described in claim 3, wherein the specified cross section is parallel to the height direction of the cylinder, passes through the center of gravity of the cylinder, and is the plane where the distance between the center of gravity of the unobserved area and the endoscopic camera is the greatest.

5. An endoscopic examination support device as described in claim 3, wherein the predetermined cross section is a plane that is parallel to the height direction of the cylinder, passes through the center of gravity of the cylinder, and is parallel to the z-axis, which is the depth direction in the camera coordinate system.

6. An endoscopic examination support device according to claim 5, wherein said drawing means cuts the three-dimensional model of the tubular organ along said predetermined cutting plane when the unobserved region exists around the entire circumference of said cylinder.

7. An endoscopic examination support device as described in claim 1, wherein the three-dimensional model generation means generates a three-dimensional model of the tubular organ that approximates a cylindrical structure, and the drawing means draws the unobserved area and the endoscopic camera on the three-dimensional model of the tubular organ viewed from a predetermined position.

8. An endoscopic examination support device as described in claim 7, wherein the specified position is a direction perpendicular to a plane in which the unobserved area appears largest, a direction in which the front of the unobserved area is visible, and a distance at which the entire three-dimensional model is visible.

9. An endoscopic examination support device as described in claim 8, wherein the drawing means performs principal component analysis on the unobserved region and determines a plane having axes of the first and second principal components as the plane in which the unobserved region appears largest.

10. An endoscopic examination support device according to claim 7, wherein the predetermined position is in a direction perpendicular to a predetermined cross section of the three-dimensional model and at a distance at which the entire three-dimensional model can be seen.

11. An endoscopic examination support device as described in claim 10, wherein the specified cross section is a plane that is parallel to the height direction of the cylinder, passes through the center of gravity of the cylinder, and has the greatest distance between the center of gravity of the unobserved area and the endoscopic camera.

12. An endoscopic examination support device as described in claim 10, wherein the specified cross section is a plane that is parallel to the height direction of the cylinder, passes through the center of gravity of the cylinder, and is parallel to the z-axis, which is the depth direction in a camera coordinate system.

13. An endoscopic examination support device according to claim 12, wherein said drawing means uses said predetermined cross section when said unobserved region exists around the entire circumference of said cylinder.

14. The endoscopic examination support device of claim 1, wherein the three-dimensional model generation means estimates depth from the endoscopic image using a first machine learning model that estimates depth from an image, estimates the camera posture of the endoscopic camera from the endoscopic image using a second machine learning model that estimates camera posture from an image, and generates a three-dimensional model of the tubular organ based on the depth and the camera posture.

15. An endoscopic examination support device according to claim 2, wherein said drawing means displays said plan view and said three-dimensional figure side by side.

16. An endoscopic examination support device according to claim 2, wherein said drawing means displays said unobserved region by filling it with a predetermined color.

17. An endoscopic examination support device according to claim 2, wherein said drawing means displays said unobserved region in said three-dimensional figure in a semi-transparent manner.

18. A method for supporting endoscopic examinations, which generates a three-dimensional point cloud and a three-dimensional model of the hollow organ where the endoscopic camera is located based on endoscopic images obtained by capturing images of the inside of the hollow organ with an endoscopic camera, detects areas that are estimated not to have been photographed by the endoscopic camera as unobserved areas based on the three-dimensional point cloud and the three-dimensional model, and displays a diagram showing the positional relationship between the hollow organ, the unobserved areas inside the hollow organ, and the endoscopic camera.

19. A recording medium having recorded thereon a program that causes a computer to execute the following process: based on endoscopic images obtained by capturing images of the inside of a hollow organ using an endoscopic camera, generate a three-dimensional point cloud and a three-dimensional model of the hollow organ where the endoscopic camera is located; based on the three-dimensional point cloud and the three-dimensional model, detect areas that are estimated not to have been photographed by the endoscopic camera as unobserved areas; and display a diagram showing the positional relationship between the hollow organ, the unobserved areas inside the hollow organ, and the endoscopic camera.

Images