Endoscopic examination support device, endoscopic examination support method, and program

Abstract

In this endoscopy assistance device, a three-dimensional model generation means generates, on the basis of an endoscopic image obtained by imaging the interior of a luminal organ using an endoscope camera, a three-dimensional model of the luminal organ in which the endoscope camera is disposed. An unobserved region detection means detects, as an unobserved region, a region inferred as not having been observed by means of the endoscope camera, on the basis of the three-dimensional model. A display image generation means generates, on the basis of the result of detecting the unobserved region, a display image including information that indicates the observation achievement degree for each of a plurality of portions of the luminal organ. The endoscopy assistance device can be used to assist, inter alia, decision-making by a user.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

[0001] The present disclosure relates to techniques that can be used to present information to assist in endoscopic examinations.

[0002] 2. Description of the Related Art Conventionally, techniques for presenting information to assist endoscopic examinations have been known.

[0003] Specifically, for example, Patent Document 1 discloses a method of displaying information indicating analyzable and unanalyzable parts of the large intestine in association with the structure of the large intestine, based on an image obtained by imaging the inside of the large intestine.

[0004] International Publication No. WO2021 / 171464

[0005] However, Patent Document 1 does not disclose a specific method for presenting information that enables confirmation of the observation status of multiple regions of the large intestine after the endoscopic examination is completed.

[0006] Therefore, with the technology disclosed in Patent Document 1, for example, when an endoscopic examination is completed and work must be performed to identify which parts of the large intestine have been observed, there is a risk that an excessive burden may be placed on the worker creating the report related to the endoscopic examination.

[0007] One object of the present disclosure is to provide an endoscopic examination support device that can reduce the burden imposed on an operator who creates a report related to an endoscopic examination.

[0008] In one aspect of the present disclosure, an endoscopic examination support device comprises: a three-dimensional model generation means for generating a three-dimensional model of a tubular organ in which an endoscopic camera is positioned, based on an endoscopic image obtained by capturing an image of the inside of the tubular organ with the endoscopic camera; an unobserved area detection means for detecting, based on the three-dimensional model, an area that is estimated not to have been observed by the endoscopic camera, as an unobserved area; and a display image generation means for generating, based on the detection result of the unobserved area, a display image including information indicating the degree of observation achievement for each of multiple areas of the tubular organ.

[0009] In another aspect of the present disclosure, a method for supporting endoscopic examination includes generating a three-dimensional model of a tubular organ in which an endoscopic camera is positioned based on an endoscopic image obtained by capturing an image of the interior of the tubular organ with the endoscopic camera; detecting, based on the three-dimensional model, areas that are estimated not to have been observed by the endoscopic camera as unobserved areas; and generating a display image including information indicating the degree of observation achievement for each of multiple parts of the tubular organ based on the detection results of the unobserved areas.

[0010] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute the following processes: generate a three-dimensional model of a tubular organ in which an endoscopic camera is located based on endoscopic images obtained by capturing images of the interior of the tubular organ with the endoscopic camera; detect, based on the three-dimensional model, areas that are estimated not to have been observed by the endoscopic camera as unobserved areas; and generate a display image that includes information indicating the degree of observation achievement for each of multiple parts of the tubular organ based on the detection results of the unobserved areas.

[0011] According to the present disclosure, it is possible to reduce the burden imposed on an operator who prepares a report related to an endoscopic examination.

[0012] FIG. 1 is a diagram showing a schematic configuration of an endoscopic examination system according to a first embodiment. FIG. 2 is a block diagram showing the hardware configuration of an endoscopic examination support device according to the first embodiment. FIG. 3 is a block diagram showing the functional configuration of an endoscopic examination support device according to the first embodiment. FIG. 4 is a diagram for explaining a specific example of an observation image. FIG. 5 is a diagram for explaining a specific example of an examination result image. A flowchart showing an example of processing performed in the endoscopic examination support device according to the first embodiment. A block diagram showing the functional configuration of an endoscopic examination support device according to a second embodiment. A flowchart showing an example of processing performed in the endoscopic examination support device according to the second embodiment.

[0013] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0014] <First embodiment> [System configuration] Fig. 1 is a diagram showing the schematic configuration of an endoscopic examination system according to Embodiment 1. As shown in Fig. 1, the endoscopic examination system 100 includes an endoscopic examination support device 1, a display device 2, and an endoscope 3 connected to the endoscopic examination support device 1.

[0015] The endoscopic examination support device 1 acquires from the endoscope 3 a video (hereinafter also referred to as the "endoscopic video Ic") including time-series images obtained by capturing images of a subject during an endoscopic examination, and displays a display image on the display device 2 for review by an operator, such as a doctor, performing the endoscopic examination. Specifically, the endoscopic examination support device 1 acquires from the endoscope 3 a video of the interior of the large intestine obtained during the endoscopic examination as the endoscopic video Ic. Based on an image extracted from the endoscopic video Ic (hereinafter also referred to as the "endoscopic image"), the endoscopic examination support device 1 estimates the distance (hereinafter also referred to as the "depth") between the surface of the large intestine, which is a hollow organ, and an endoscopic camera provided at the distal end 38 of the endoscope 3, as well as changes in the relative orientation of the endoscopic camera. The endoscopic examination support device 1 then generates a 3D model corresponding to the structure of the large intestine (intestinal tract) by performing 3D reconstruction based on the depth and the changes in the relative orientation of the endoscopic camera. Furthermore, the endoscopic examination support device 1 detects difficult-to-observe areas, which are areas estimated to be difficult to observe during endoscopic examination, based on the endoscopic image. The endoscopic examination support device 1 also detects lesion candidate regions, which are regions estimated to be lesion candidates, based on the endoscopic image. The endoscopic examination support device 1 also detects missing regions in the 3D model due to incomplete or insufficient 3D reconstruction. The endoscopic examination support device 1 also detects at least one of difficult-to-observe regions and missing regions in the 3D model as unobserved regions. The endoscopic examination support device 1 also acquires subject information based on the endoscopic image, indicating which of multiple regions of the large intestine a subject captured by an endoscopic camera in the large intestine corresponds to. Based on the unobserved region detection results and subject information, the endoscopic examination support device 1 also performs a mapping process to associate unobserved regions in the current endoscopic colon examination with a 3D model of the entire large intestine (hereinafter also referred to as a "large intestine model") that has been created in advance based on the general structure of the large intestine (intestinal tract). In addition, the endoscopic examination support device 1 uses a colon model or the like that has been subjected to the above-mentioned mapping processing to generate a display image including an overview of the examination results in the endoscopic examination, and outputs the generated display image to the display device 2.

[0016] The difficult-to-observe regions may include, for example, regions that are difficult to see due to insufficient brightness, regions that are difficult to see due to the intensity of blur, and regions where the state of the mucosal surface cannot be seen due to the presence of residue. The missing regions may include, for example, regions hidden by obstructions in the large intestine, such as folds, and regions where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or more. The predetermined period of time may be set to, for example, one second. The multiple regions of the large intestine may include, for example, the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, and cecum.

[0017] According to this embodiment, it is not necessary to perform processing related to the detection of difficult-to-observe regions. In such cases, it is sufficient to detect missing regions in the three-dimensional model as unobserved regions.

[0018] The display device 2 has, for example, a liquid crystal monitor, etc. The display device 2 displays images, etc. output from the endoscopic examination support device 1.

[0019] The endoscope 3 mainly has an operation unit 36 ​​that allows the surgeon to input instructions such as air supply, water supply, angle adjustment, and photography instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that has an endoscopic camera such as an ultra-small imaging element built in, and a connection unit 39 for connecting to the endoscopic examination support device 1.

[0020] 2 is a block diagram showing the hardware configuration of the endoscopic examination support device according to the first embodiment. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0021] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer. The processor 11 also performs processing related to generating a display image including an outline of the examination results of the endoscopic examination based on an endoscopic image included in the endoscopic video Ic.

[0022] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and nonvolatile memories that store information necessary for processing by the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs for the endoscopic examination support device 1 to execute each process in this embodiment.

[0023] Furthermore, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images Ic captured by the endoscope 3 during an endoscopic examination.

[0024] The interface 13 acts as an interface between the endoscopic examination assistance device 1 and an external device. For example, the interface 13 supplies a display image generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic video Ic supplied from the endoscope 3 to the processor 11. The interface 13 also supplies an endoscopic image extracted from the endoscopic video Ic to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0025] The input unit 14 generates an input signal based on an operation by the surgeon. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also include a built-in pump for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0026] The DB 17 stores endoscopic images and the like acquired during past endoscopic examinations of the subject. The DB 17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory. Instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be acquired from the server via communication.

[0027] The endoscopic examination support device 1 may also be provided with a sensor, such as a magnetic sensor, that can measure the rotation and translation of the endoscopic camera.

[0028] 3 is a block diagram showing the functional configuration of the endoscopic examination support device according to the first embodiment. The endoscopic examination support device 1 functionally includes a depth estimation unit 21, a camera posture estimation unit 22, a three-dimensional reconstruction unit 23, a difficult-to-observe region detection unit 24, an unobserved region detection unit 25, a subject information acquisition unit 26, a lesion candidate detection unit 27, a mapping processing unit 28, and a display image generation unit 29.

[0029] The depth estimation unit 21 performs a process of estimating depth from an endoscopic image using a trained image recognition model, etc. That is, the depth estimation unit 21 functions as a distance estimation means, and can estimate the distance between the surface of a hollow organ and an endoscopic camera placed inside the hollow organ based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera. The depth estimation unit 21 also outputs the depth estimated by the above-mentioned process to the three-dimensional reconstruction unit 23.

[0030] The camera posture estimation unit 22 performs a process of estimating the rotation and translation of the endoscopic camera from the capture point of the first endoscopic image to the capture point of the second endoscopic image (i.e., the change in the relative posture of the endoscopic camera; hereinafter, simply referred to as "camera posture change") using, for example, two temporally consecutive endoscopic images. The camera posture estimation unit 22 also performs a process of estimating the camera posture change using, for example, a trained image recognition model. That is, the camera posture estimation unit 22 functions as a posture change estimation unit and can estimate the change in the relative posture of the endoscopic camera based on endoscopic images obtained by capturing the inside of a tubular organ with the endoscopic camera. The camera posture estimation unit 22 also outputs the camera posture change estimated by the above-mentioned process to the three-dimensional reconstruction unit 23. The camera posture estimation unit 22 may also estimate the camera posture change using measurement data obtained from a magnetic sensor or the like.

[0031] Here, the image recognition models used in the depth estimation unit 21 and the camera posture estimation unit 22 are machine learning models that have been trained in advance to estimate depth and camera posture changes from endoscopic images. Hereinafter, these will also be referred to as the "depth estimation model" and the "camera posture estimation model." The depth estimation model and the camera posture estimation model can be generated by so-called supervised learning.

[0032] For example, training data in which depths are assigned as correct labels to endoscopic images is used to train the depth estimation model. The endoscopic images and depths used for training are collected in advance from an endoscopic camera and a ToF (Time of Flight) sensor provided on the endoscope. That is, pairs of RGB images obtained by the endoscopic camera and depths obtained by the ToF sensor are created as training data, and training is performed using the created training data.

[0033] Furthermore, training data in which changes in camera posture are assigned as correct labels to endoscopic images is used to train the camera posture estimation model. In this case, changes in camera posture can be acquired using a sensor capable of detecting rotation and translation, such as a magnetic sensor. That is, pairs of RGB images acquired by an endoscopic camera and changes in camera posture acquired by a sensor are created as training data, and training is performed using the training data.

[0034] The training data used to train the depth estimation model and the camera pose estimation model may be created from a simulated image of an endoscope using computer graphics (CG). This allows a large amount of training data to be created quickly. The machine learning device uses the training data to learn the relationship between the endoscopic image and changes in depth and camera pose, thereby generating the depth estimation model and the camera pose estimation model.

[0035] The depth estimation model and the camera pose estimation model may be generated by self-supervised learning. For example, in self-supervised learning, training data is generated using motion parallax. Specifically, in self-supervised learning, ｉ and endoscopic image Iｊ and the endoscopic image I ｉ A Depth CNN (Convolutional Neural Network) that estimates the depth from the endoscopic image I ｉ and the endoscopic image I ｊ Based on the depth and relative pose estimated by each of them, the endoscopic image I ｉ Endoscopic image I ｊ is reconstructed (this is called the "endoscopic image I"). ｉ→ｊ The reconstructed endoscopic image I ｉ→ｊ and actual endoscopic image I ｊ The model is trained using the difference between and as the loss.

[0036] The 3D reconstruction unit 23 generates a 3D model corresponding to the structure of the large intestine (intestinal tract) during endoscopic examination by performing 3D reconstruction processing based on the depth obtained from the depth estimation unit 21 and the change in relative posture of the endoscopic camera obtained from the camera posture estimation unit 22. Then, the 3D reconstruction unit 23 outputs the 3D model, the change in relative posture of the endoscopic camera, and the position of the endoscopic camera to the unobserved region detection unit 25.

[0037] That is, the three-dimensional model generating means of this embodiment includes a depth estimation unit 21, a camera attitude estimation unit 22, and a three-dimensional reconstruction unit 23.

[0038] The difficult-to-observe region detection unit 24 detects, as a difficult-to-observe region, at least one of a region in the endoscopic image where the brightness is below a predetermined value, a region where the amount of blur is above a predetermined value, and a region where residue is present. That is, the difficult-to-observe region detection unit 24 detects, based on the endoscopic image, a region inside the tubular organ that is estimated to be difficult to observe with the endoscopic camera as a difficult-to-observe region. The difficult-to-observe region detection unit 24 then outputs the detection result of the difficult-to-observe region to the unobserved region detection unit 25. Note that the difficult-to-observe region detection unit 24 may associate the detection result of the difficult-to-observe region with information indicating the factors that estimated to make observation with the endoscopic camera difficult, such as insufficient brightness, strong blur, or the presence of residue. In other words, the detection result of the difficult-to-observe region may be associated with information indicating the factors that made the observation difficult.

[0039] The unobserved region detection unit 25 detects missing regions in the three-dimensional model based on the relative posture change of the endoscopic camera, the position of the endoscopic camera, and the three-dimensional model. Specifically, the unobserved region detection unit 25 detects, for example, a region in the three-dimensional model that is hidden by an obstruction such as a fold, and / or a region where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or more as missing regions. The unobserved region detection unit 25 also detects, for example, a region in the three-dimensional model acquired from the three-dimensional reconstruction unit 23 within the last five seconds that has not been three-dimensionally reconstructed for one second or more as missing regions. The unobserved region detection unit 25 also performs processing to identify regions in the three-dimensional model generated by the three-dimensional reconstruction unit 23 that correspond to the detection results of the difficult-to-observe regions obtained from the difficult-to-observe region detection unit 24. The unobserved region detection unit 25 also detects difficult-to-observe regions and missing regions in the three-dimensional model as unobserved regions. That is, the unobserved region detection unit 25 detects, as an unobserved region, a region estimated to have not been observed by the endoscopic camera based on a three-dimensional model of the luminal organ where the endoscopic camera is located. Furthermore, the unobserved region detection unit 25 can obtain the most recent detection result based on the observation history of the large intestine (intestinal tract) by the endoscopic camera as the detection result of the unobserved region in the three-dimensional model. The unobserved region detection unit 25 then outputs the relative posture change of the endoscopic camera, the position of the endoscopic camera, the three-dimensional model, and the detection result of the unobserved region to the mapping processing unit 28. The unobserved region detection unit 25 may associate the detection result of the unobserved region with information indicating the cause of the estimation of the unobserved region, such as the presence of an obstruction, no image capture, insufficient brightness, strong blurring, or the presence of residue. In other words, the detection result of the unobserved region may be associated with information indicating the cause of the unobserved region.

[0040] The subject information acquisition unit 26 acquires, based on the endoscopic image, subject information indicating which of multiple portions of the large intestine the subject captured by the endoscopic camera corresponds to. The subject information acquisition unit 26 also acquires the subject information by processing using, for example, an image recognition model trained to output information relating to portions of the large intestine in response to input of an endoscopic image obtained by capturing an image of the inside of the large intestine. The subject information acquisition unit 26 then outputs the subject information to the mapping processing unit 28.

[0041] Lesion candidate detection unit 27 uses a trained image recognition model or the like to detect lesion candidate regions in endoscopic images that are regions that are estimated to be lesion candidates. Specifically, lesion candidate detection unit 27 detects, for example, regions that include polyps as lesion candidate regions. That is, lesion candidate detection unit 27 detects lesion candidate regions that are regions that are estimated to be lesion candidates based on endoscopic images obtained by capturing the interior of a hollow organ with an endoscopic camera. Then, lesion candidate detection unit 27 outputs the detection results of the lesion candidate regions to mapping processing unit 28 and display image generation unit 29.

[0042] The mapping processing unit 28 performs a mapping process to associate the unobserved region and the lesion candidate region in the current endoscopic examination with the large intestine model based on the relative pose change of the endoscopic camera, the position of the endoscopic camera, the 3D model obtained from the 3D reconstruction unit 23, the unobserved region detection results, the subject information, and the lesion candidate region detection results. This mapping process allows a region in the large intestine identified based on the relative pose change of the endoscopic camera, the position of the endoscopic camera, and the subject information to be excluded from the unobserved region. Hereinafter, a region that does not correspond to the unobserved region in multiple regions of the large intestine, such as a region excluded from the unobserved region by the above-described mapping process, will be referred to as an "observed region." The mapping processing unit 28 also outputs the large intestine model subjected to the above-described mapping process to the display image generation unit 29.

[0043] During the endoscopic examination, the display image generation unit 29 generates a display image based on, for example, the endoscopic image and the detection results of the lesion candidate region, and outputs the generated display image to the display device 2. During the endoscopic examination, the display image generation unit 29 acquires a large intestine model from the mapping processing unit 28 and updates the acquired large intestine model to the latest state. During the endoscopic examination, the display image generation unit 29 acquires detection results of the lesion candidate region from the lesion candidate detection unit 27 and accumulates the acquired detection results of the lesion candidate region. When the display image generation unit 29 detects that a predetermined instruction has been given after the endoscopic examination is completed, the display image generation unit 29 generates a display image including an overview of the examination results of the endoscopic examination based on the detection results of the lesion candidate region accumulated during the endoscopic examination and the latest large intestine model acquired during the endoscopic examination, and outputs the generated display image to the display device 2.

[0044] Hereinafter, the display image generated during the endoscopic examination will also be referred to as an "observation image." Also, hereinafter, the display image generated after the endoscopic examination, which includes an outline of the examination results of the endoscopic examination, will also be referred to as an "examination result image."

[0045] The observation image may include an endoscopic image, and may also include information indicating the latest detection result of the lesion candidate region.

[0046] The examination result image may include information indicating the observation achievement level for each of multiple regions of the large intestine. The observation achievement level for a region of the large intestine can be calculated, for example, by subtracting the observation incompleteness level from "1." The observation incompleteness level can be calculated by dividing the area of ​​the unobserved region mapped to that region in the large intestine model by a predetermined surface area calculated based on the standard surface area of ​​that region. The observation achievement level for a region of the large intestine may be obtained as a value ranging from 0 to 1, or from 0% to 100%. In other words, the display image generation unit 29 generates a display image including information indicating the observation achievement level for each of multiple regions of the tubular organ based on the detection results of the unobserved region associated with the large intestine model.

[0047] The examination result image may also include information indicating the positions of unobserved regions in multiple parts of the large intestine.

[0048] The inspection result image may also include information that enables identification of unobserved factors associated with the detection results of the unobserved areas. Specifically, in the inspection result image, for example, a first unobserved factor associated with the detection results of a first unobserved area may be represented in a first color, and a second unobserved factor associated with the detection results of a second unobserved area may be represented in a second color.

[0049] The test result image may also include at least one of information indicating the total number of lesion candidate areas detected based on the endoscopic image, information indicating the positions of the lesion candidate areas detected based on the endoscopic image, and information indicating the state of the lesion candidate areas detected based on the endoscopic image.

[0050] The test result image may include a large intestine model, or may include a large intestine image that is a pre-created image based on a typical large intestine structure. Hereinafter, unless otherwise specified, a large intestine image refers to an image that is created so that a cross section of the entire large intestine cut along the vertical direction of the human body can be seen from the front of the body.

[0051] According to this embodiment, information that enables identification of unobserved regions and observed regions may be added to the colon model or colon image included in the examination result image. That is, the display image generating unit 29 may generate a display image that includes information that enables identification of unobserved regions and observed regions.

[0052] Furthermore, according to this embodiment, for example, when an unobserved area in a colon model or colon image included in an examination result image is specified, an image of an observed area in the vicinity of the unobserved area may be played back.

[0053] Furthermore, according to this embodiment, for example, information that enables identification of unobserved regions on the ventral side (the inner wall of the ventral side of the large intestine) and unobserved regions on the dorsal side (the inner wall of the dorsal side of the large intestine) may be added to the large intestine model or large intestine image included in the examination result image. In such a case, for example, the unobserved regions on the ventral side and the unobserved regions on the dorsal side may be displayed in different colors.

[0054] [Display Examples] Next, a description will be given of specific examples of the observation image and the inspection result image displayed on the display device 2. Fig. 4 is a diagram for explaining a specific example of the observation image. Fig. 5 is a diagram for explaining a specific example of the inspection result image.

[0055] 4 is an image displayed on the display device 2 during an endoscopic examination. The observation image DK includes an endoscopic image 41 and a lesion candidate image 42.

[0056] The endoscopic image 41 is an image included in the endoscopic video Ic obtained during the endoscopic examination. The endoscopic image 41 includes a subject within the field of view of the endoscopic camera at its current position, and is updated in accordance with the movement of the endoscopic camera.

[0057] Lesion candidate image 42 is smaller in size than endoscopic image 41 and is positioned to the right of endoscopic image 41. Lesion candidate image 42 is an image generated by superimposing lesion position information 42A on another endoscopic image that was acquired before endoscopic image 41 was acquired.

[0058] Lesion location information 42A is displayed as information indicating the latest detection result of the lesion candidate region. In the display example of Fig. 4, lesion location information 42A is displayed as a circular marker surrounding the periphery of the lesion candidate region.

[0059] 5 is an image displayed on the display device 2 after an endoscopic examination is completed, when a predetermined instruction, such as displaying the examination results, is given. The examination result image DR includes an observation achievement level display area 51, a lesion detection count display area 52, and an observation result information display area 53. The examination result image DR also includes a lesion candidate image 54, which corresponds to information showing the state of the lesion candidate region detected during the endoscopic examination.

[0060] The observation achievement display area 51 displays values ​​indicating the observation achievement level for each of multiple regions of the large intestine. The display example of the observation achievement display area 51 in Fig. 5 allows one to confirm that there are no unobserved regions in the rectum and cecum, where the observation achievement level is 100%. Furthermore, the display example of the observation achievement display area 51 in Fig. 5 allows one to confirm that there are unobserved regions in the sigmoid colon, descending colon, transverse colon, and ascending colon, where the observation achievement level is less than 100%. Furthermore, the display example of the observation achievement display area 51 in Fig. 5 allows one to confirm that the endoscopic camera has reached the cecum, the deepest part of the large intestine, during the endoscopic examination.

[0061] Information indicating the total number of lesion candidate regions detected during endoscopic examination is displayed in the lesion detection number display area 52. According to the display example of the lesion detection number display area 52 in Fig. 5, it can be confirmed that one region containing a polyp corresponding to a lesion candidate region was detected during endoscopic examination.

[0062] A large intestine image 53A, created as a schematic diagram in which multiple regions of the large intestine are separated by dashed lines, is displayed in the observation result information display area 53. The large intestine image 53A also includes observed region information 53B, which indicates the observed regions in the multiple regions of the large intestine. The large intestine image 53A also includes unobserved region information 53C, 53D, and 53E, which indicates the positions of unobserved regions in the multiple regions of the large intestine.

[0063] In the display example of FIG. 5 , the dotted line portion in the schematic diagram of the large intestine included in the large intestine image 53A is displayed as observed region information 53B. Also, in the display example of FIG. 5 , the thick line portion in the schematic diagram of the large intestine included in the large intestine image 53A is displayed as unobserved region information 53C. Therefore, the display example of FIG. 5 makes it possible to distinguish between unobserved regions and observed regions. Also, the display examples of the observed region information 53B and unobserved region information 53C in FIG. 5 make it possible to confirm which of multiple regions of the large intestine contain unobserved regions. Note that, according to this embodiment, the observed region information 53B and the unobserved region information 53C may be displayed in different display modes in the large intestine image 53A. Specifically, for example, the observed region information 53B and the unobserved region information 53C may be displayed in different colors in the large intestine image 53A.

[0064] The unobserved region information 53D indicates the position of an unobserved region on the ventral side of the large intestine. Furthermore, the unobserved region information 53D is displayed in a different display mode from the observed region information 53B, the unobserved region information 53C, and the unobserved region information 53E in the large intestine image 53A. Specifically, the unobserved region information 53D is displayed in a different pattern or color from the observed region information 53B, the unobserved region information 53C, and the unobserved region information 53E. The display example of the unobserved region information 53D in FIG. 5 allows confirmation of the presence of an unobserved region in the ventral portion of the ascending colon. Note that, according to this embodiment, the unobserved region information 53D may be displayed in a different pattern or color depending on the cause of the unobserved region (presence of an obstruction, no image capture, insufficient brightness, strong blurring, presence of residue, etc.).

[0065] The unobserved region information 53E indicates the position of an unobserved region on the back side of the large intestine. Furthermore, the unobserved region information 53E is displayed in a different display mode from the observed region information 53B, the unobserved region information 53C, and the unobserved region information 53D in the large intestine image 53A. Specifically, the unobserved region information 53E is displayed in a different pattern or color from the observed region information 53B, the unobserved region information 53C, and the unobserved region information 53D. The display example of the unobserved region information 53E in FIG. 5 allows confirmation that an unobserved region exists in the back side of the sigmoid colon. Note that, according to this embodiment, the unobserved region information 53E may be displayed in a different pattern or color depending on the cause of the unobserved region (presence of an obstruction, no image capture, insufficient brightness, strong blurring, presence of residue, etc.).

[0066] Lesion candidate image 54 is a thumbnail image of a lesion candidate region detected during an endoscopic examination. Furthermore, lesion candidate image 54 is displayed in association with the position where the lesion candidate region was detected in observed region information 53B. According to the display example of FIG. 5 , the same number of lesion candidate images 54 as the total number (1) of lesion candidate regions displayed in lesion detection count display area 52 are displayed. According to this embodiment, clicking on a lesion candidate image 54 may display an enlarged image of the lesion candidate region or video captured around the lesion candidate region.

[0067] [Processing Flow] Next, a description will be given of the flow of processing performed in the endoscopic examination support device according to the first embodiment. Fig. 6 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the first embodiment.

[0068] First, the endoscopic examination support device 1 estimates the depth from an endoscopic image obtained during the endoscopic examination (step S11).

[0069] Next, the endoscopic examination support device 1 estimates the change in camera posture from two temporally consecutive endoscopic images obtained during the endoscopic examination (step S12).

[0070] Next, the endoscopic examination support device 1 performs a three-dimensional reconstruction process based on the depth obtained in step S11 and the camera posture change obtained in step S12, thereby generating a three-dimensional model corresponding to the structure of the large intestine (intestinal tract) during the endoscopic examination (step S13).

[0071] Next, the endoscopic examination support device 1 detects a difficult-to-observe region based on the endoscopic image obtained during the endoscopic examination (step S14).

[0072] Next, the endoscopic examination support device 1 detects missing areas in the three-dimensional model generated in step S13 (step S15).

[0073] Next, the endoscopic examination support device 1 detects, as unobserved areas, areas in the three-dimensional model generated in step S13 that correspond to the difficult-to-observe areas detected in step S14 and areas that correspond to the missing areas detected in step S15 (step S16).

[0074] Next, the endoscopic examination support device 1 acquires subject information based on the endoscopic image obtained during the endoscopic examination (step S17).

[0075] Next, the endoscopic examination support device 1 detects a lesion candidate region based on the endoscopic image obtained during the endoscopic examination (step S18). Furthermore, the endoscopic examination support device 1 accumulates the detection results of the lesion candidate region obtained in step S18 while the endoscopic examination is being performed.

[0076] Next, the endoscopic examination support device 1 performs a mapping process to associate the unobserved region and the lesion candidate region with the large intestine model based on the detection result of the unobserved region obtained in step S16, the object information obtained in step S17, and the detection result of the lesion candidate region obtained in step S18 (step S19). Furthermore, the endoscopic examination support device 1 updates the processing result of the mapping process in step S19 while the endoscopic examination is being performed, thereby obtaining the latest large intestine model.

[0077] After the endoscopic examination is completed, the endoscopic examination support device 1 generates an examination result image corresponding to the endoscopic examination based on the detection results of the lesion candidate region accumulated by performing the processing of step S18 during the endoscopic examination and the latest large intestine model obtained by performing the mapping processing of step S19 during the endoscopic examination (step S20).The examination result image generated in step S20 is then displayed on the display device 2.

[0078] In this embodiment, the process of step S12 may be executed before step S11, or the process of step S11 may be executed simultaneously with the process of step S12.

[0079] As described above, according to this embodiment, after an endoscopic examination is completed, an examination result image including information indicating the degree of observation success for each of multiple regions of the large intestine can be displayed. Therefore, according to this embodiment, the burden imposed on the operator who prepares the report related to the endoscopic examination can be reduced. This can also be used to support the user's decision-making, etc.

[0080] Furthermore, according to this embodiment, the proficiency level of the surgeon at the time of performing the endoscopic examination can be grasped based on the above-mentioned observation achievement level, which can contribute to improving the proficiency level of the surgeon performing the endoscopic examination of the large intestine.

[0081] Second Embodiment FIG. 7 is a block diagram showing the functional configuration of an endoscopic examination support device according to a second embodiment.

[0082] The endoscopic examination support device 70 according to this embodiment has the same hardware configuration as the endoscopic examination support device 1. The endoscopic examination support device 70 also includes a three-dimensional model generation unit 71, an unobserved region detection unit 72, and a display image generation unit 73.

[0083] FIG. 8 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the second embodiment.

[0084] The three-dimensional model generating means 71 generates a three-dimensional model of the hollow organ in which the endoscope camera is placed, based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscope camera (step S71).

[0085] The unobserved region detecting means 72 detects, based on the three-dimensional model, a region that is estimated not to have been observed by the endoscopic camera as an unobserved region (step S72).

[0086] The display image generating means 73 generates a display image including information indicating the degree of observation achievement for each of the multiple regions of the tubular organ based on the detection result of the unobserved region (step S73).

[0087] According to this embodiment, it is possible to reduce the burden imposed on an operator who prepares a report related to an endoscopic examination.

[0088] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0089] (Supplementary Note 1) An endoscopic examination support device comprising: a three-dimensional model generation means for generating a three-dimensional model of a hollow organ in which an endoscopic camera is placed, based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera; an unobserved area detection means for detecting, based on the three-dimensional model, an area that is estimated not to have been observed by the endoscopic camera as an unobserved area; and a display image generation means for generating, based on the detection result of the unobserved area, a display image including information indicating the degree of observation achievement for each of a plurality of areas of the hollow organ.

[0090] (Supplementary Note 2) The endoscopic examination support device according to Supplementary Note 1, wherein the display image generating means generates the display image including information indicating the position of the unobserved region in the plurality of regions.

[0091] (Supplementary Note 3) The endoscopic examination support device according to Supplementary Note 1, wherein the display image generating means generates the display image including information that can distinguish between the unobserved areas in the plurality of regions and observed areas that are areas that do not correspond to the unobserved areas in the plurality of regions.

[0092] (Supplementary Note 4) The endoscopic examination support device according to Supplementary Note 1, wherein the display image generating means generates the display image including information that can identify a cause of the unobserved area that is associated with the detection result of the unobserved area.

[0093] (Appendix 5) The endoscopic examination support device according to Appendix 1 further comprises a lesion candidate detection means for detecting lesion candidate regions, which are regions estimated to be lesion candidates, based on the endoscopic image using a machine-learned model, and the display image generation means generates the display image including at least one of information indicating the total number of the lesion candidate regions, information indicating the positions of the lesion candidate regions, and information indicating the state of the lesion candidate regions.

[0094] (Appendix 6) The endoscopic examination support device described in Appendix 1, wherein the unobserved area detection means detects at least one of a difficult-to-observe area, which is an area inside the tubular organ that is estimated to be difficult to observe using the endoscopic camera, and a missing area, which is an area where a gap has occurred in the three-dimensional model, as the unobserved area.

[0095] (Appendix 7) The difficult-to-observe area is an area in the endoscopic image that corresponds to at least one of an area where brightness is below a predetermined value, an area where the amount of blur is above a predetermined value, and an area where residue exists, and the unobserved area detection means detects an area in the three-dimensional model that corresponds to the difficult-to-observe area as the unobserved area.

[0096] (Appendix 8) The endoscopic examination support device described in Appendix 6, wherein the missing area corresponds to at least one of an area in the three-dimensional model that is hidden by an obstruction within the tubular organ and an area where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or more.

[0097] (Supplementary Note 9) An endoscopic examination support method that generates a three-dimensional model of a tubular organ in which an endoscopic camera is positioned based on endoscopic images obtained by capturing an image of the inside of the tubular organ with the endoscopic camera, detects areas that are estimated not to have been observed by the endoscopic camera based on the three-dimensional model as unobserved areas, and generates a display image that includes information indicating the degree of observation achievement for each of multiple parts of the tubular organ based on the detection results of the unobserved areas.

[0098] (Appendix 10) A recording medium having recorded thereon a program that causes a computer to execute the following processes: based on endoscopic images obtained by capturing images of the inside of a hollow organ with an endoscopic camera, generate a three-dimensional model of the hollow organ in which the endoscopic camera is located; based on the three-dimensional model, detect areas that are estimated not to have been observed by the endoscopic camera as unobserved areas; and based on the detection results of the unobserved areas, generate a display image that includes information indicating the degree of observation achievement for each of multiple parts of the hollow organ.

[0099] This application claims priority based on International Application PCT / JP2022 / 029427, filed August 1, 2022, the disclosure of which is incorporated herein in its entirety.

[0100] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0101] REFERENCE SIGNS LIST 1 Endoscopy support device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 21 Depth estimation unit 22 Camera attitude estimation unit 23 Three-dimensional reconstruction unit 24 Difficult-to-observe area detection unit 25 Unobserved area detection unit 26 Object information acquisition unit 27 Lesion candidate detection unit 28 Mapping processing unit 29 Display image generation unit 100 Endoscopy system

Claims

1. a three-dimensional model generating means for generating a three-dimensional model of the hollow organ in which the endoscope camera is disposed, based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscope camera; an unobserved region detection means for detecting a region that is estimated not to have been observed by the endoscope camera as an unobserved region based on the three-dimensional model; a display image generating means for generating a display image including information indicating an observation achievement level for each of the plurality of portions of the tubular organ based on the detection result of the unobserved region; An endoscopic examination support device comprising:

2. The endoscopic examination support device according to claim 1 , wherein the display image generating means generates the display image including information indicating the positions of the unobserved regions in the plurality of sites.

3. The endoscopic examination support device according to claim 1, wherein the display image generating means generates the display image including information capable of distinguishing between the unobserved areas in the plurality of locations and observed areas which are areas not corresponding to the unobserved areas in the plurality of locations.

4. The endoscopic examination support device according to claim 1 , wherein the display image generating means generates the display image including information capable of identifying a cause of the unobserved region that is associated with a detection result of the unobserved region.

5. The method further includes a lesion candidate detection means for detecting a lesion candidate region, which is a region estimated to be a lesion candidate, based on the endoscopic image by a machine learning model; 2. The endoscopic examination support device according to claim 1, wherein the display image generating means generates the display image including at least one of information indicating a total number of the candidate lesion regions, information indicating the positions of the candidate lesion regions, and information indicating a state of the candidate lesion regions.

6. The endoscopic examination support device according to claim 1, wherein the unobserved area detection means detects as the unobserved area at least one of a difficult-to-observe area, which is an area inside the tubular organ that is estimated to be difficult to observe using the endoscopic camera, and a missing area, which is an area where a gap has occurred in the three-dimensional model.

7. the difficult-to-observe region corresponds to at least one of a region in the endoscopic image whose brightness is equal to or lower than a predetermined value, a region whose amount of blur is equal to or higher than a predetermined value, and a region in which residue is present, 7. The endoscopic examination support device according to claim 6, wherein the unobserved region detection means detects, as the unobserved region, a region in the three-dimensional model that corresponds to the difficult-to-observe region.

8. The endoscopic examination support device according to claim 6, wherein the missing area corresponds to at least one of an area in the three-dimensional model that is hidden by an obstruction within the tubular organ and an area where imaging by the endoscopic camera has not been performed continuously for a predetermined period of time or more.

9. A computer-implemented method for assisting endoscopic examination, comprising: generating a three-dimensional model of the hollow organ in which the endoscopic camera is disposed based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera; Based on the three-dimensional model, a region that is estimated not to have been observed by the endoscope camera is detected as an unobserved region; An endoscopic examination support method for generating a display image including information indicating the observation achievement level for each of a plurality of parts of the tubular organ based on the detection result of the unobserved area.

10. generating a three-dimensional model of the hollow organ in which the endoscopic camera is disposed based on an endoscopic image obtained by capturing an image of the inside of the hollow organ with the endoscopic camera; Based on the three-dimensional model, a region that is estimated not to have been observed by the endoscope camera is detected as an unobserved region; A program that causes a computer to execute a process of generating a display image including information indicating the observation achievement level for each of a plurality of parts of the tubular organ based on the detection result of the unobserved area.