Endoscopic examination support device, endoscopic examination support method, and program

Abstract

In this endoscopic examination assistance device, an image acquisition means acquires a captured image upon the removal of an endoscope. A posture change estimation means estimates the change in a relative posture of an endoscope camera from the captured image. A distance estimation means estimates the distance between the surface of the large intestine and the endoscope camera, from the captured image. An intestinal tract direction estimation means estimates the intestinal tract direction of the large intestine, on the basis of the posture change and the distance. A calculation means calculates the direction in which the endoscope camera is to be directed, on the basis of the intestinal tract direction and the relative posture of the endoscope camera. An output means outputs a display screen image including the direction in which the endoscope camera is to be directed into a display device. The invention can be used for the assistance of a user in decision making or the like.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

[0001] The present disclosure relates to processing images related to endoscopy.

[0002] Because endoscopes are elastic and the large intestine itself is soft and has a complex shape, when a doctor operates the endoscope, the endoscopic camera may move to an unexpected position and come close to the wall of the large intestine. This may result in areas of the large intestine surface that cannot be observed, which may lead to overlooking a lesion. Patent Document 1 proposes providing an insertion system that presents recommended insertion methods when inserting a medical endoscope or the like into an insertion target.

[0003] International Publication No. WO2018 / 069992

[0004] However, Patent Document 1 only presents a method for inserting an endoscope, and does not present a direction of the endoscopic camera that allows appropriate observation of organs when the endoscope is removed.

[0005] One object of the present disclosure is to present an endoscopic camera direction suitable for observation during endoscopic examination.

[0006] In one aspect of the present disclosure, an endoscopic examination support device comprises: an image acquisition means for acquiring a captured image when an endoscope is removed; an attitude change estimation means for estimating a change in the relative attitude of the endoscopic camera from the captured image; a distance estimation means for estimating a distance between the surface of the large intestine and the endoscopic camera from the captured image; an intestinal direction estimation means for estimating the intestinal direction of the large intestine based on the attitude change and the distance; a calculation means for calculating the direction in which the endoscopic camera should be pointed based on the intestinal direction and the relative attitude of the endoscopic camera; and an output means for outputting a display image including the direction in which the endoscopic camera should be pointed to a display device.

[0007] In another aspect of the present disclosure, a method for supporting endoscopic examination includes acquiring a captured image when an endoscope is removed, estimating a change in the relative posture of an endoscopic camera from the captured image, estimating a distance between a surface of the large intestine and the endoscopic camera from the captured image, estimating an intestinal direction of the large intestine based on the change in posture and the distance, calculating a direction in which the endoscopic camera should be pointed based on the intestinal direction and the relative posture of the endoscopic camera, and outputting a display image including the direction in which the endoscopic camera should be pointed to a display device.

[0008] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute the following processes: acquire a captured image when an endoscope is removed; estimate a change in the relative posture of the endoscopic camera from the captured image; estimate a distance between the surface of the large intestine and the endoscopic camera from the captured image; estimate an intestinal direction of the large intestine based on the change in posture and the distance; calculate a direction in which the endoscopic camera should be pointed based on the intestinal direction and the relative posture of the endoscopic camera; and output a display image including the direction in which the endoscopic camera should be pointed to a display device.

[0009] According to the present disclosure, it is possible to present an endoscopic camera direction suitable for observation during endoscopic examination.

[0010] 1 is a block diagram showing a schematic configuration of an endoscopic examination system. FIG. 2 is a block diagram showing the hardware configuration of an endoscopic examination support device. FIG. 3 is a block diagram showing the functional configuration of the endoscopic examination support device. FIG. 4 shows an example of the direction in which an endoscopic camera should be pointed. FIG. 5 is a diagram showing an example of the display of calculation results. FIG. 6 is a diagram showing another example of the display of calculation results. FIG. 7 is a diagram showing another example of the display of calculation results. FIG. 8 is a flowchart of the direction calculation process of an endoscopic camera by the endoscopic examination support device. FIG. 9 is a block diagram showing the functional configuration of an endoscopic examination support device of a second embodiment. FIG. 10 is a flowchart of the processing by the endoscopic examination support device of the second embodiment.

[0011] Preferred embodiments of the present disclosure will be described below with reference to the drawings. <First Embodiment> [System Configuration] Fig. 1 shows a schematic configuration of an endoscopic examination system 100. The endoscopic examination system 100 estimates the direction of the intestinal tract and the direction of the endoscopic camera during an examination (including treatment) using an endoscope. If the direction of the endoscopic camera is not facing the direction of the intestinal tract, the endoscopic examination system 100 provides direction guidance to orient the endoscopic camera toward the intestinal tract. By following the guidance from the endoscopic examination system 100 and orienting the endoscopic camera toward the intestinal tract, the doctor can observe the entire intestinal tract. This reduces the areas that cannot be observed.

[0012] As shown in FIG. 1 , the endoscopic examination system 100 mainly includes an endoscopic examination support device 1 , a display device 2 , and an endoscope 3 connected to the endoscopic examination support device 1 .

[0013] The endoscopic examination support device 1 acquires from the endoscope 3 an image (i.e., a moving image; hereinafter, also referred to as "endoscopic image Ic") captured by the endoscope 3 during an endoscopic examination, and displays display data on the display device 2 for the examiner performing the endoscopic examination to confirm. Specifically, the endoscopic examination support device 1 acquires a moving image of the large intestine captured by the endoscope 3 during an endoscopic examination as the endoscopic image Ic. The endoscopic examination support device 1 extracts frame images from the endoscopic image Ic and, based on the frame images, estimates the distance between the surface of the large intestine and the endoscopic camera (hereinafter, also referred to as "depth") and the change in the relative orientation of the endoscopic camera. Then, based on the depth and the change in the relative orientation of the endoscopic camera, the endoscopic examination support device 1 performs three-dimensional reconstruction of the intestinal tract of the large intestine and estimates the direction of the intestinal tract. The endoscopic examination support device 1 estimates the direction in which the endoscopic camera should be pointed based on the intestinal direction and the relative orientation of the endoscopic camera.

[0014] The display device 2 is a display or the like that displays a predetermined image based on a display signal supplied from the endoscopic examination support device 1 .

[0015] The endoscope 3 mainly has an operation unit 36 ​​that allows the examiner to input instructions such as air supply, water supply, angle adjustment, and photography instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that has an endoscopic camera such as a miniature imaging element built in, and a connection unit 39 for connecting to the endoscopic examination support device 1.

[0016] 2 shows the hardware configuration of the endoscopic examination support device 1. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0017] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0018] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and nonvolatile memories that store information necessary for processing by the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs for the endoscopic examination support device 1 to execute each process in this embodiment.

[0019] Furthermore, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images Ic captured by the endoscope 3 during an endoscopic examination.

[0020] The interface 13 performs interface operations between the endoscopic examination assistance device 1 and an external device. For example, the interface 13 supplies the display data Id generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic video Ic supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0021] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0022] The DB 17 stores endoscopic images and the like acquired during past endoscopic examinations of the subject. The DB 17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory. Instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be acquired from the server via communication.

[0023] The endoscopic examination support device 1 may also be provided with a sensor, such as a magnetic sensor, that can measure the rotation and translation of the endoscopic camera.

[0024] 3 is a block diagram showing the functional configuration of the endoscopic examination support device 1. Functionally, the endoscopic examination support device 1 includes an interface 13, a depth estimation unit 21, a camera posture estimation unit 22, a three-dimensional reconstruction unit 23, an operation direction estimation unit 24, a lesion detection unit 25, and a display image generation unit 26.

[0025] An endoscopic video Ic is input to the endoscopic examination support device 1 from the endoscope 3. The endoscopic video Ic is input to the interface 13. The interface 13 extracts frame images (hereinafter also referred to as "endoscopic images") from the input endoscopic video Ic, and outputs them to the depth estimation unit 21, the camera posture estimation unit 22, and the lesion detection unit 25. The interface 13 also outputs the input endoscopic video Ic to the display image generation unit 26.

[0026] An endoscopic image is input to the depth estimation unit 21 from the interface 13. The depth estimation unit 21 estimates the depth from the input endoscopic image using a pre-prepared image recognition model or the like. The depth estimation unit 21 then outputs the estimated depth to the three-dimensional reconstruction unit 23.

[0027] The camera posture estimation unit 22 receives an endoscopic image from the interface 13. For example, the camera posture estimation unit 22 uses two temporally consecutive endoscopic images to estimate the rotation and translation of the endoscopic camera from the capture point of the first endoscopic image to the capture point of the second endoscopic image (i.e., the change in the relative posture of the endoscopic camera; hereinafter, simply referred to as "camera posture change"). The camera posture estimation unit 22 then outputs the estimated camera posture change of the endoscopic camera to the three-dimensional reconstruction unit 23. For example, the camera posture estimation unit 22 estimates the camera posture change from the input endoscopic image using a pre-prepared image recognition model or the like. The camera posture estimation unit 22 may also estimate the change in the relative posture of the endoscopic camera using measurement data from a magnetic sensor or the like.

[0028] Here, the image recognition models used by the depth estimation unit 21 and the camera posture estimation unit 22 are machine learning models that have been trained in advance to estimate depth and camera posture changes from endoscopic images. These are also referred to as the "depth estimation model" and the "camera posture estimation model." The depth estimation model and the camera posture estimation model can be generated by so-called supervised learning.

[0029] For example, training data in which depths are assigned as correct labels to endoscopic images is used to train the depth estimation model. The endoscopic images and depths used for training are collected in advance from an endoscopic camera and a ToF (Time of Flight) sensor attached to the endoscope. That is, pairs of RGB images captured by the endoscopic camera and depths are created as training data, and training is performed using the training data.

[0030] Furthermore, training data in which changes in camera posture are assigned as correct labels to endoscopic images is used to train the camera posture estimation model. In this case, the changes in camera posture can be acquired using a sensor that can detect rotation and translation, such as a magnetic sensor. That is, pairs of RGB images captured by an endoscopic camera and changes in camera posture are created as training data, and training is performed using the training data.

[0031] The training data used to train the depth estimation model and the camera pose estimation model may be created from a simulated image of an endoscope using computer graphics (CG). This allows a large amount of training data to be created quickly. The machine learning device uses the training data to learn the relationship between the endoscopic image and changes in depth and camera pose, thereby generating the depth estimation model and the camera pose estimation model.

[0032] The depth estimation model and the camera pose estimation model may be generated by self-supervised learning. For example, in self-supervised learning, training data is generated using motion parallax. Specifically, in self-supervised learning, ｉ and endoscopic image I ｊ Paired images of and endoscopic image I ｉDepth CNN (Convolutional Neural Network) that estimates depth from endoscopic image I ｉ and endoscopic image I ｊ Then, based on the depth and relative pose estimated by each of them, the endoscopic image I ｉ Endoscopic image I ｊ (This is called the "endoscopic image I" ｉ→ｊ The reconstructed endoscopic image I ｉ→ｊ and actual endoscopic image I ｊ The model is trained using the difference as the loss.

[0033] The three-dimensional reconstruction unit 23 performs three-dimensional reconstruction processing of the intestinal tract and estimates the intestinal direction based on the depth input from the depth estimation unit 21 and the change in the relative orientation of the endoscopic camera input from the camera orientation estimation unit 22. Then, the three-dimensional reconstruction unit 23 outputs the three-dimensional model, the intestinal direction, the change in the relative orientation of the endoscopic camera, and the position of the endoscopic camera to the operation direction estimation unit 24.

[0034] The operation direction estimation unit 24 receives the three-dimensional model, the intestinal tract direction, and the relative attitude change of the endoscopic camera from the three-dimensional reconstruction unit 23. Then, the operation direction estimation unit 24 calculates the direction in which the endoscopic camera should be pointed based on the intestinal tract direction and the relative attitude change of the endoscopic camera. Then, the operation direction estimation unit 24 outputs the three-dimensional model, the relative attitude change of the endoscopic camera, and the direction in which the endoscopic camera should be pointed to the display image generation unit 26.

[0035] FIG. 4 shows an example of the direction in which the endoscopic camera should be pointed. In FIG. 4, a three-dimensional model 31 of the intestinal tract, an intestinal tract direction 32, and an endoscopic camera direction 33 are shown on an XYZ coordinate system. The three-dimensional model 31 is a model of the intestinal tract three-dimensionally reconstructed by the three-dimensional reconstruction unit 23, and includes a detailed three-dimensional structure of the intestinal tract. However, for ease of explanation, FIG. 4 shows the three-dimensional model 31 as an approximation of a cylinder. The intestinal tract direction 32 is the length direction or axial direction of the intestinal tract, and is estimated based on the three-dimensional model 31 of the intestinal tract. The endoscopic camera direction 33 is the direction of the lens of the endoscopic camera, i.e., the imaging direction.

[0036] 4 , the operation direction estimation unit 24 calculates the angle between the intestinal tract direction 32 and the endoscopic camera direction 33, i.e., the deviation angle θ of the endoscopic camera direction 33 from the intestinal tract direction 32. If the deviation angle θ is equal to or greater than a predetermined threshold, the operation direction estimation unit 24 determines that the endoscopic camera is facing the intestinal wall. If the operation direction estimation unit 24 determines that the endoscopic camera is facing the intestinal wall, it calculates the direction in which the endoscopic camera should be pointed so that the direction of the endoscopic camera coincides with the direction of the intestinal tract (so that the deviation angle θ becomes zero), and outputs the result to the display image generation unit 26.

[0037] An endoscopic image is input to lesion detection unit 25 from interface 13. Lesion detection unit 25 then detects lesion candidates from the endoscopic image using a pre-prepared image recognition model or the like, and generates a lesion candidate image including the detected lesion candidates. Lesion detection unit 25 surrounds the lesion candidates on the lesion candidate image with an ellipse or the like, and outputs the image to display image generation unit 26.

[0038] The display image generation unit 26 generates display data using the three-dimensional model, the relative posture change of the endoscopic camera, the direction in which the endoscopic camera should be pointed, and the lesion candidate image input from the operation direction estimation unit 24 and the lesion detection unit 25, and outputs the display data to the display device 2.

[0039] In the above configuration, the interface 13 is an example of an image acquisition means, the depth estimation unit 21 is an example of a distance estimation means, the camera posture estimation unit 22 is an example of a posture change estimation means, the three-dimensional restoration unit 23 is an example of an intestinal direction estimation means, the operation direction estimation unit 24 is an example of a calculation means, and the display image generation unit 26 is an example of an output means.

[0040] [Display Example] Next, a display example on the display device 2 will be described.

[0041] 5 is an example of a display on the display device 2. In this example, the display device 2 displays an endoscopic image 41, a lesion history 42, a camera trajectory 43, a camera mark 44, an intestinal tract direction indicator 45, and a lesion direction indicator 46.

[0042] Endoscopic video 41 is an endoscopic video Ic during the examination and is updated as the endoscopic camera moves. Lesion history 42 is an image showing lesion candidates detected during the endoscopic examination, and uses lesion candidate images input from lesion detection unit 25. Lesion candidate sites detected by lesion detection unit 25 are indicated by ellipses 42a. Note that if lesion candidates are detected at multiple locations, lesion history 42 displays the image of the most recent lesion candidate.

[0043] The camera trajectory 43 indicates the trajectory of the endoscopic camera within a predetermined time. In FIG. 5 , a three-dimensional intestinal model 43a is represented as a cylinder, and a camera mark 44 indicating the orientation and position of the endoscopic camera at a predetermined time is superimposed on the intestinal model 43a to indicate the camera trajectory. The camera mark 44 schematically indicates the orientation and position of the endoscopic camera at different times. In FIG. 5 , the camera mark 44 is represented as a cone, with the base of the cone indicating the lens side of the endoscopic camera. The camera mark 44 is also color-coded according to time series, with darker colors indicating the most recent orientation and position of the endoscopic camera. Note that in FIG. 5 , the arrow indicates that the camera direction of the endoscopic camera is changing from the direction of the intestine to the direction of the intestinal wall.

[0044] The intestinal tract direction indicator 45 indicates the direction in which the endoscopic camera should be pointed so that it faces the intestinal tract. The intestinal tract direction indicator 45 is displayed when the endoscopic camera is facing the intestinal wall, specifically when the aforementioned deviation angle θ is equal to or greater than a predetermined threshold. In FIG. 5 , the intestinal tract direction indicator 45 is displayed at the left and top ends of the endoscopic image 41. This allows the physician to know that pointing the endoscopic camera toward the upper left will point the endoscopic camera toward the intestinal tract. Note that when the direction in which the endoscopic camera should be pointed is to the right, the intestinal tract direction indicator 45 is displayed at the right end of the endoscopic image 41, and when the direction in which the endoscopic camera should be pointed is downward, the intestinal tract direction indicator 45 is displayed at the bottom end of the endoscopic image 41. In this way, when the endoscopic camera is facing the intestinal wall, the intestinal tract direction indicator 45 is displayed in at least one of the top, bottom, left, and right ends of the endoscopic image 41, depending on the direction in which the endoscopic camera should be pointed.

[0045] Meanwhile, the lesion direction indicator 46 indicates the direction in which the endoscopic camera should be pointed so that it faces the lesion. The lesion direction indicator 46 is displayed when a lesion candidate is detected. In FIG. 5 , the lesion direction indicator 46 is displayed at the left edge of the endoscopic video 41. This allows the physician to understand that pointing the endoscopic camera to the left will point the endoscopic camera toward the lesion candidate. In this way, when a lesion candidate is detected, the lesion direction indicator 46 is displayed in at least one of the top, bottom, left, and right edges of the endoscopic video 41, depending on the position of the lesion candidate.

[0046] The display image generation unit 26 may generate display data for the camera trajectory 43 so as to display the intestinal tract model 43a viewed from a direction with as little overlap as possible between the multiple camera marks 44. For example, in the example of FIG. 6 , the camera trajectory 43 is displayed with the intestinal tract model 43a viewed from the intestinal tract direction. This causes the camera marks 44 to overlap, making it difficult for the doctor to properly grasp the trajectory of the endoscopic camera. Therefore, the display image generation unit 26 determines a direction that maximizes the variance of the camera directions indicated by the multiple camera marks 44 using principal component analysis or the like, and generates display data that displays the camera trajectory 43 with the intestinal tract model 43a viewed from that direction. This allows the display device 2 to properly display the trajectory of the endoscopic camera using the intestinal tract model viewed from a direction with as little overlap as possible between the camera marks 44, as shown in FIG. 5 .

[0047] Fig. 7 shows another example of display by the display device 2. In this example, the intestinal tract direction indicator and the lesion direction indicator are displayed as arrows. Specifically, in Fig. 7, an intestinal tract direction indicator 45a and a lesion direction indicator 46a are displayed on the endoscopic image 41. In Fig. 7, it is possible to show more detailed directions than when indicators are placed at the top, bottom, left, or right edges of the screen as in Fig. 5.

[0048] FIG. 8 shows another example of display by the display device 2. In the example of FIG. 6, the camera trajectory is displayed on an intestinal tract model 43a. In contrast, FIG. 8 shows an example in which the camera trajectory is displayed on an endoscopic image. Specifically, in the camera trajectory 43x of FIG. 8, a camera mark 44 indicating the orientation and position of the endoscopic camera at a given time is superimposed on the endoscopic image 43b. Here, the endoscopic image 43b is an endoscopic image taken in a past ideal imaging direction, for example, an endoscopic image taken with the endoscopic camera facing the intestinal tract. Furthermore, in the camera trajectory 43x of FIG. 8, in addition to the camera trajectory, the ideal camera position is indicated by a camera mark 44a represented by a black cone. In this case, the endoscopic image 43b shown in FIG. 8 can be an endoscopic image taken in the state indicated by the camera mark 44a. In the example of FIG. 8, the endoscopic camera trajectory is displayed on the actual endoscopic image, making it easier for the physician to intuitively grasp the ideal position of the endoscopic camera.

[0049] [Determination Process] Next, the display process for producing the above-mentioned display will be described. Fig. 9 is a flowchart of the process performed by the endoscopic examination support device 1. This process is realized by the processor 11 shown in Fig. 2 executing a program prepared in advance and operating as each element shown in Fig. 3. This process is also executed during an examination using an endoscope, i.e., when the endoscope 3 is removed.

[0050] First, the endoscopic video Ic is input from the endoscope 3 to the interface 13. The interface 13 acquires an endoscopic image from the input endoscopic video Ic (step S11). Next, the depth estimation unit 21 estimates the distance between the surface of the large intestine and the endoscopic camera from the endoscopic image using a pre-prepared image recognition model or the like (step S12). The camera posture estimation unit 22 estimates the relative posture change of the endoscopic camera from two temporally consecutive endoscopic images (step S13). Next, the 3D reconstruction unit 23 performs 3D reconstruction of the intestinal tract based on the distance between the surface of the large intestine and the endoscopic camera and the relative posture change of the endoscopic camera, thereby estimating the intestinal direction (step S14). The operation direction estimation unit 24 then calculates the direction in which the endoscopic camera should be pointed based on the relative posture change of the endoscopic camera and the intestinal direction (step S15).

[0051] The display image generator 26 generates display data using the three-dimensional model, the relative change in the posture of the endoscopic camera, and the direction in which the endoscopic camera should be pointed, and outputs the generated display data to the display device 2 (step S16). In this way, a display such as that shown in Fig. 5 is produced. Note that step S13 may be executed before step S12 or simultaneously with step S12.

[0052] This can be used to support the user in decision making, etc.

[0053] 10 is a block diagram showing the functional configuration of an endoscopic examination support device according to Embodiment 2. The endoscopic examination support device 70 includes an image acquisition unit 71, a posture change estimation unit 72, a distance estimation unit 73, an intestinal tract direction estimation unit 74, a calculation unit 75, and an output unit 76.

[0054] 11 is a flowchart of the processing performed by the endoscopic examination support device of the second embodiment. The image acquisition means 71 acquires a captured image when the endoscope is removed (step S71). The posture change estimation means 72 estimates a change in the relative posture of the endoscopic camera from the captured image (step S72). The distance estimation means 73 estimates the distance between the surface of the large intestine and the endoscopic camera from the captured image (step S73). The intestinal tract direction estimation means 74 estimates the intestinal tract direction of the large intestine based on the change in posture of the endoscopic camera and the distance between the surface of the large intestine and the endoscopic camera (step S74). The calculation means 75 calculates the direction in which the endoscopic camera should be pointed based on the intestinal tract direction and the relative posture of the endoscopic camera (step S75). The output means 76 outputs a display image including the direction in which the endoscopic camera should be pointed to a display device (step S76).

[0055] According to the endoscopic examination support device 70 of the second embodiment, it is possible to present the direction of the endoscopic camera that is suitable for observation during endoscopic examination.

[0056] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0057] (Supplementary Note 1) An endoscopic examination support device comprising: an image acquisition means for acquiring a captured image when an endoscope is removed; an attitude change estimation means for estimating a change in the relative attitude of an endoscopic camera from the captured image; a distance estimation means for estimating a distance between the surface of the large intestine and the endoscopic camera from the captured image; an intestinal tract direction estimation means for estimating the intestinal tract direction of the large intestine based on the attitude change and the distance; a calculation means for calculating the direction in which the endoscopic camera should be pointed based on the intestinal tract direction and the relative attitude of the endoscopic camera; and an output means for outputting a display image including the direction in which the endoscopic camera should be pointed to a display device.

[0058] (Supplementary Note 2) The endoscopic examination support device according to Supplementary Note 1, wherein the direction in which the endoscopic camera should be pointed is the direction of the intestinal tract.

[0059] (Supplementary Note 3) The endoscopic examination support device according to Supplementary Note 1, wherein the posture change estimation means estimates the posture change using a machine learning model that has been trained in advance to estimate depth and camera posture changes from endoscopic images.

[0060] (Appendix 4) The endoscopic examination support device according to Appendix 1, wherein the directions in which the endoscopic camera should be pointed are the intestinal tract direction and the lesion direction, and the output means outputs a display image that displays the intestinal tract direction and the lesion direction in a manner that allows them to be distinguished.

[0061] (Supplementary Note 5) The endoscopic examination support device according to Supplementary Note 1, wherein the intestinal tract direction estimation means creates a model of the large intestine based on the change in posture and the distance, and estimates the intestinal tract direction based on the model of the intestine.

[0062] (Supplementary Note 6) The endoscopic examination support device according to Supplementary Note 5, wherein the output means outputs a display image in which the trajectory of the change in posture is superimposed on the model of the intestinal tract.

[0063] (Supplementary Note 7) The endoscopic examination support device according to Supplementary Note 6, wherein the output means outputs a display image of the intestinal tract model viewed from a direction where there is little overlap of the trajectories of the posture changes.

[0064] (Supplementary Note 8) The endoscopic examination support device according to Supplementary Note 1, wherein the output means outputs a display image in which the trajectory of the change in posture and the direction in which the endoscopic camera should be pointed are superimposed on the captured image.

[0065] (Supplementary Note 9) A method for supporting endoscopic examination, comprising: acquiring a captured image when an endoscope is removed; estimating a change in the relative posture of an endoscopic camera from the captured image; estimating a distance between the surface of the large intestine and the endoscopic camera from the captured image; estimating the intestinal direction of the large intestine based on the change in posture and the distance; calculating a direction in which the endoscopic camera should be pointed based on the intestinal direction and the relative posture of the endoscopic camera; and outputting a display image including the direction in which the endoscopic camera should be pointed to a display device.

[0066] (Supplementary Note 10) A recording medium having recorded thereon a program that causes a computer to execute the following processes: acquire a captured image when an endoscope is removed; estimate a change in the relative posture of the endoscopic camera from the captured image; estimate a distance between the surface of the large intestine and the endoscopic camera from the captured image; estimate the intestinal direction of the large intestine based on the change in posture and the distance; calculate a direction in which the endoscopic camera should be pointed based on the intestinal direction and the relative posture of the endoscopic camera; and output a display image including the direction in which the endoscopic camera should be pointed to a display device.

[0067] This application claims priority based on International Application PCT / JP2022 / 029450, filed August 1, 2022, the disclosure of which is incorporated herein in its entirety.

[0068] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0069] REFERENCE SIGNS LIST 1 Endoscopy support device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 21 Depth estimation unit 22 Camera attitude estimation unit 23 Three-dimensional reconstruction unit 24 Operation direction estimation unit 25 Lesion detection unit 26 Display image generation unit 100 Endoscopy system

Claims

1. an image acquisition means for acquiring a photographed image when the endoscope is removed; a posture change estimation means for estimating a change in a relative posture of the endoscope camera from the captured image; a distance estimation means for estimating a distance between a surface of the large intestine and the endoscope camera from the captured image; an intestinal tract direction estimation means for estimating an intestinal tract direction of the large intestine based on the change in posture and the distance; a calculation means for calculating a direction in which the endoscope camera should be pointed based on the intestinal direction and a relative attitude of the endoscope camera; an output means for outputting a display image including a direction in which the endoscope camera should be pointed to a display device; An endoscopic examination support device comprising:

2. The endoscopic examination support device according to claim 1 , wherein the direction in which the endoscopic camera is to be pointed is toward the intestinal tract.

3. The endoscopic examination support device according to claim 1 , wherein the posture change estimation means estimates the posture change using a machine learning model that has been trained in advance to estimate depth and camera posture changes from an endoscopic image.

4. The direction in which the endoscope camera should be pointed is the intestinal direction and the lesion direction, The endoscopic examination support device according to claim 1 , wherein the output means outputs a display image that displays the intestinal tract direction and the lesion direction in a manner that allows the intestinal tract direction and the lesion direction to be distinguished from each other.

5. The endoscopic examination support device according to claim 1 , wherein the intestinal tract direction estimation means creates a model of the large intestine based on the change in posture and the distance, and estimates the intestinal tract direction based on the model of the intestine.

6. The endoscopic examination support device according to claim 5 , wherein the output means outputs a display image in which the trajectory of the change in posture is superimposed on the model of the intestinal tract.

7. The endoscopic examination support device according to claim 6 , wherein the output means outputs a display image of the intestinal tract model viewed from a direction in which there is little overlap of the trajectories of the posture changes.

8. 2. The endoscopic examination support device according to claim 1, wherein the output means outputs a display image in which the trajectory of the change in posture and the direction in which the endoscopic camera should be pointed are superimposed on the captured image.

9. A method for supporting endoscopic examination executed by a computer, comprising: Images were taken when the endoscope was removed, Estimating a change in a relative attitude of the endoscope camera from the captured image; estimating a distance between a surface of the large intestine and the endoscope camera from the captured image; estimating an intestinal direction of the large intestine based on the change in posture and the distance; calculating a direction in which the endoscope camera should be pointed based on the intestinal direction and the relative attitude of the endoscope camera; An endoscopic examination support method, comprising: outputting, to a display device, a display image including a direction in which the endoscopic camera should be pointed.

10. Images were taken when the endoscope was removed, Estimating a change in a relative attitude of the endoscope camera from the captured image; estimating a distance between a surface of the large intestine and the endoscope camera from the captured image; estimating an intestinal direction of the large intestine based on the change in posture and the distance; calculating a direction in which the endoscope camera should be pointed based on the intestinal direction and the relative attitude of the endoscope camera; A program that causes a computer to execute a process of outputting a display image including a direction in which the endoscope camera should be pointed to a display device.