Endoscopic examination assistance device, endoscopic examination assistance method, and recording medium

Abstract

An endoscopic examination assistance device according to the present invention has an actual depth acquisition means that acquires actual depth information that indicates an actual depth at the position of each pixel in an endoscopic image on the basis of relative depth information that indicates a relative depth at the position of each pixel in the endoscopic image and reference object information that indicates the position and size of a reference object in the endoscopic image, a three-dimensional model generation means that generates a three-dimensional model that corresponds to the actual depth information, a lesion region identification means that identifies a region of the three-dimensional model that corresponds to a lesion detected from the endoscopic image as a lesion region, and an information provision means that provides information about the actual size of the lesion included in the lesion region. The endoscopic examination assistance device can be used to assist decision-making by a user that is to diagnose a legion.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

[0001] The present disclosure relates to techniques that can be used to present information to assist in endoscopic examinations.

[0002] 2. Description of the Related Art Conventionally, techniques for presenting information to assist endoscopic examinations have been known.

[0003] Specifically, for example, Patent Document 1 discloses a technology for generating a scale representing the actual size of an area to be observed by an endoscope, synthesizing the generated scale with an endoscopic image, and displaying the image on a display device.

[0004] JP 2016-182161 A

[0005] According to the technology disclosed in Patent Document 1, in order to grasp the actual size of the observation area, it is necessary to visually check the scale superimposed on the endoscopic image. Therefore, with the technology disclosed in Patent Document 1, for example, when the observation area includes a lesion, it is difficult to grasp the exact size of the lesion.

[0006] One object of the present disclosure is to provide an endoscopic examination support device that is capable of presenting information related to the exact size of a lesion discovered during an endoscopic examination.

[0007] In one aspect of the present disclosure, an endoscopic examination support device includes an actual depth acquisition means for acquiring actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image, a three-dimensional model generation means for generating a three-dimensional model according to the actual depth information, a lesion area identification means for identifying an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area, and an information presentation means for presenting information regarding the actual size of the lesion included in the lesion area.

[0008] In another aspect of the present disclosure, an endoscopic examination support method is a computer-executed endoscopic examination support method that acquires actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image, generates a three-dimensional model according to the actual depth information, identifies an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area, and presents information related to the actual size of the lesion included in the lesion area.

[0009] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of acquiring actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image, generating a three-dimensional model according to the actual depth information, identifying an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area, and presenting information related to the actual size of the lesion included in the lesion area.

[0010] According to the present disclosure, it is possible to present information regarding the exact size of a lesion discovered during an endoscopic examination.

[0011] FIG. 1 is a diagram showing a schematic configuration of an endoscopic examination system according to the present disclosure. FIG. 2 is a block diagram showing an example of the hardware configuration of an endoscopic examination support device according to the present disclosure. FIG. 3 is a block diagram showing an example of the functional configuration of an endoscopic examination support device according to the present disclosure. FIG. 4 is a diagram showing an example of an endoscopic image included in a display image generated by an endoscopic examination support device according to the present disclosure. FIG. 5 is a diagram showing another example of an endoscopic image included in a display image generated by an endoscopic examination support device according to the present disclosure. FIG. 6 is a flowchart showing an example of processing performed in an endoscopic examination support device according to the present disclosure. FIG. 7 is a diagram showing another example of an endoscopic image included in a display image generated by an endoscopic examination support device according to the present disclosure. FIG. 8 is a block diagram showing another example of the functional configuration of an endoscopic examination support device according to the present disclosure. FIG. 9 is a flowchart showing another example of processing performed in an endoscopic examination support device according to the present disclosure.

[0012] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0013] <First embodiment> [System configuration] Fig. 1 is a diagram showing a schematic configuration of an endoscopic examination system according to the present disclosure. As shown in Fig. 1, the endoscopic examination system 100 includes an endoscopic examination support device 1, a display device 2, and an endoscope 3 connected to the endoscopic examination support device 1.

[0014] The endoscopic examination support device 1 acquires, from the endoscope 3, video including time-series images obtained by capturing images of a subject during an endoscopic examination (hereinafter also referred to as endoscopic video), and displays a display image on the display device 2 for confirmation by a user, such as a doctor performing the endoscopic examination. Specifically, the endoscopic examination support device 1 acquires, from the endoscope 3, video of the interior of the large intestine obtained during the endoscopic examination as endoscopic video. Furthermore, based on an endoscopic image extracted from the endoscopic video, the endoscopic examination support device 1 estimates the distance from the current position of an endoscopic camera provided at the distal end 38 of the endoscope 3 to the position of each pixel included in the endoscopic video as a relative depth. Furthermore, the endoscopic examination support device 1 detects a reference object from the endoscopic image and, using the position and size of the reference object in the endoscopic image, acquires a depth in real space corresponding to the relative depth of each pixel included in the endoscopic image (hereinafter also referred to as actual depth). Furthermore, the endoscopic examination support device 1 generates a 3D model corresponding to the structure of the large intestine (intestinal tract) by performing 3D reconstruction based on the actual depth and camera parameters of the endoscopic camera. The endoscopic examination support device 1 also performs processing to detect lesions from endoscopic images. The endoscopic examination support device 1 also identifies, as a lesion area, an area in the three-dimensional model that corresponds to a lesion detected from the endoscopic image, and acquires, as lesion size information, information related to the actual size of the lesion contained in the lesion area. The endoscopic examination support device 1 also generates a display image based on the endoscopic image and the lesion size information, and outputs the generated display image to the display device 2. The endoscopic examination support device 1 can be used to support the decision-making of a user diagnosing a lesion.

[0015] The display device 2 has, for example, a liquid crystal monitor, etc. The display device 2 displays images, etc. output from the endoscopic examination support device 1.

[0016] The endoscope 3 mainly comprises an operation unit 36 ​​for the user to input commands such as air supply, water supply, angle adjustment, and imaging instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that incorporates an endoscopic camera such as an imaging element, and a connection unit 39 for connecting to the endoscopic examination support device 1. The endoscopic camera can obtain endoscopic images by capturing images of the inside of the large intestine, which is a hollow organ.

[0017] 2 is a block diagram showing an example of the hardware configuration of an endoscopic examination support device according to the present disclosure. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0018] The processor 11 executes predetermined processes by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer. The processor 11 also performs processes such as acquiring actual depths corresponding to the relative depths of each pixel included in an endoscopic image.

[0019] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and nonvolatile memories that store information necessary for processing by the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs for the endoscopic examination support device 1 to execute each process in this embodiment.

[0020] Furthermore, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images captured by the endoscope 3 during an endoscopic examination.

[0021] The interface 13 acts as an interface between the endoscopic examination assistance device 1 and an external device. For example, the interface 13 supplies a display image generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating an endoscopic video supplied from the endoscope 3 to the processor 11. The interface 13 also supplies an endoscopic image extracted from the endoscopic video to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0022] The input unit 14 generates an input signal in response to a user operation. The input unit 14 has at least one device selected from the group consisting of a button, a touch panel, a remote controller, a foot switch, and a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump for delivering water or air to the endoscope 3. The sound output unit 16 outputs sound under the control of the processor 11.

[0023] The DB 17 stores endoscopic images and the like acquired during past endoscopic examinations of the subject. The DB 17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory. Instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be acquired from the server via communication.

[0024] The endoscopic examination support device 1 may also be provided with a sensor, such as a magnetic sensor, that can measure the rotation and translation of the endoscopic camera.

[0025] [Functional Configuration] Fig. 3 is a block diagram showing an example of the functional configuration of the endoscopic examination support device 1 according to the present disclosure. As shown in Fig. 3, the endoscopic examination support device 1 includes a relative depth acquisition unit 21, a reference object detection unit 22, an actual depth acquisition unit 23, a three-dimensional reconstruction unit 24, a lesion detection unit 25, a lesion region identification unit 26, a lesion size acquisition unit 27, and a display image generation unit 28.

[0026] The relative depth acquisition unit 21 performs a process of estimating a relative depth SD corresponding to each pixel position included in the endoscopic image EG using a trained image recognition model, etc. Specifically, for example, among the pixels included in the endoscopic image EG, the relative depth acquisition unit 21 can acquire a relatively large value as the relative depth SD of a pixel located at a position estimated to be far from the endoscopic camera. Furthermore, for example, among the pixels included in the endoscopic image EG, the relative depth acquisition unit 21 can acquire a relatively small value as the relative depth SD of a pixel located at a position estimated to be close to the endoscopic camera.

[0027] The relative depth acquisition unit 21 acquires relative depth information SDJ indicating the relative depth SD for each pixel position of the endoscopic image EG. Specifically, the relative depth acquisition unit 21 can acquire, as the relative depth information SDJ, a monochrome image having pixel values ​​corresponding to the relative depth SD for each pixel position of the endoscopic image EG, for example, by inputting the endoscopic image EG into a trained machine learning model. In other words, the relative depth information SDJ corresponds to an output image obtained by inputting the endoscopic image EG into the trained machine learning model. Furthermore, the relative depth acquisition unit 21 outputs the relative depth information SDJ to the actual depth acquisition unit 23.

[0028] The reference object detection unit 22 detects a reference object RB included in the endoscopic image EG. Specifically, the reference object detection unit 22 detects, as the reference object RB, a jet of water ejected forward from the water outlet of the distal end portion 38 in response to, for example, the operation of a predetermined button provided on the input unit 14.

[0029] The reference object detection unit 22 acquires reference object information RBJ as information indicating the position and size of the reference object RB in the endoscopic image EG, and outputs the acquired reference object information RBJ to the actual depth acquisition unit 23.

[0030] The real depth acquisition unit 23 functions as a real depth acquisition means. Furthermore, by performing processing using the relative depth information SDJ obtained from the relative depth acquisition unit 21 and the reference object information RBJ obtained from the reference object detection unit 22, the real depth acquisition unit 23 acquires a real depth RD corresponding to each relative depth SD included in the relative depth information SDJ. Specifically, the real depth acquisition unit 23 acquires a value expressed in units of distance in real space, such as millimeters or centimeters, as the real depth RD. A specific example of a method for acquiring the real depth RD corresponding to the relative depth SD will be described later.

[0031] The actual depth acquisition unit 23 acquires actual depth information RDJ indicating the actual depth RD for each pixel position of the endoscopic image EG. Specifically, the actual depth acquisition unit 23 can acquire the actual depth information RDJ by, for example, linking the actual depth RD to each pixel of the monochrome image included in the relative depth information SDJ. The actual depth acquisition unit 23 also outputs the actual depth information RDJ to the three-dimensional reconstruction unit 24.

[0032] The 3D reconstruction unit 24 functions as a 3D model generating unit. The 3D reconstruction unit 24 performs 3D reconstruction processing based on the actual depth information RDJ obtained from the actual depth acquisition unit 23 and the camera parameters of the endoscopic camera provided at the distal end 38, thereby generating a 3D model TDM corresponding to the structure of the large intestine (intestinal tract) during endoscopic examination. The camera parameters may include, for example, a distortion coefficient and a focal length. When generating the 3D model TDM, the 3D reconstruction unit 24 may perform bundle adjustment using, for example, multiple actual depth information RDJ corresponding to each of multiple endoscopic images EG. The 3D reconstruction unit 24 outputs the 3D model TDM to the lesion region identification unit 26.

[0033] The lesion detection unit 25 performs processing to detect a lesion LE from the endoscopic image EG using a trained image recognition model, etc. The lesion detection unit 25 also acquires lesion detection information DCJ as information indicating the position and size of the lesion LE in the endoscopic image EG, and outputs the acquired lesion detection information DCJ to the lesion area identification unit 26.

[0034] The lesion area identification unit 26 functions as a lesion area identification means. Furthermore, the lesion area identification unit 26 identifies the lesion area DA by performing processing using the three-dimensional model TDM obtained from the three-dimensional reconstruction unit 24 and the lesion detection information DCJ obtained from the lesion detection unit 25. Specifically, the lesion area identification unit 26 identifies, for example, an area in the three-dimensional model TDM corresponding to the lesion LE indicated by the lesion detection information DCJ as the lesion area DA. In other words, the lesion area identification unit 26 identifies, as the lesion area DA, an area in the three-dimensional model TDM corresponding to the lesion LE detected from the endoscopic image EG. Furthermore, the lesion area identification unit 26 acquires lesion area information DAJ as information indicating the position and size of the lesion area DA, and outputs the acquired lesion area information DAJ to the lesion size acquisition unit 27.

[0035] Lesion size acquisition unit 27 performs processing using lesion area information DAJ obtained from lesion area identification unit 26 to acquire lesion size information LSJ as information indicating the actual size of lesion LE included in lesion area DA. Lesion size acquisition unit 27 also outputs lesion size information LSJ to display image generation unit 28. Specific examples of methods for acquiring lesion size information LSJ will be described later. Furthermore, lesion size information LSJ may include the same information as the information included in lesion area information DAJ.

[0036] Display image generation unit 28 functions as an information presentation means. When display image generation unit 28 is able to acquire lesion size information LSJ from lesion size acquisition unit 27, it generates a display image HG based on endoscopic image EG and the lesion size information LSJ, and outputs the generated display image HG to display device 2. When display image generation unit 28 is unable to acquire lesion size information LSJ from lesion size acquisition unit 27, for example, because a lesion cannot be detected from endoscopic image EG, it generates a display image HG including endoscopic image EG without using the lesion size information LSJ, and outputs the generated display image HG to display device 2.

[0037] [Specific Example] Next, a specific example according to the present disclosure will be described.

[0038] Fig. 4 is a diagram showing an example of an endoscopic image included in a display image generated by the endoscopic examination support device according to the present disclosure. Fig. 5 is a diagram showing another example of an endoscopic image included in a display image generated by the endoscopic examination support device according to the present disclosure. For example, when a user finds a lesion LES included in the endoscopic image EGA as shown in Fig. 4, the user presses a predetermined button on the input unit 14 to cause a jet of water to be ejected forward from the water outlet of the distal end portion 38. Based on such a user operation, the endoscopic examination support device 1 can acquire an endoscopic image EGB as shown in Fig. 5, for example. The endoscopic image EGB includes the lesion LES and the jet of water JWS. Note that in the present disclosure, for ease of illustration, the area in the endoscopic image where the jet of water JWS exists is indicated by a black pattern.

[0039] The relative depth acquisition unit 21 acquires relative depth information SDJB indicating the relative depth SDB for each pixel position of the endoscopic image EGB, and outputs the acquired relative depth information SDBJ to the actual depth acquisition unit 23. The relative depth information SDBJ includes, for example, a monochrome image having pixel values ​​corresponding to the relative depth SDB for each pixel position of the endoscopic image EGB.

[0040] The reference object detection unit 22 detects the jet water stream JWS included in the endoscopic image EGB as a reference object RBB. The reference object detection unit 22 also acquires reference object information RBBJ as information indicating the position and size of the jet water stream JWS within the endoscopic image EGB, and outputs the acquired reference object information RBJB to the actual depth acquisition unit 23.

[0041] The actual depth acquisition unit 23 performs processing using the relative depth information SDBJ and the reference object information RBBJ to acquire actual depth RDBs corresponding to the respective relative depth SDBs included in the relative depth information SDBJ.

[0042] Here, a specific example of a method for acquiring the actual depth RDB corresponding to the relative depth SDB will be described.

[0043] The actual depth acquisition unit 23 uses information included in the reference object information RBBJ to acquire a pixel position PPJ corresponding to the arrival position of the jet water stream JWS in the endoscopic image EGB and a pixel count PCJ corresponding to the width of the tip of the jet water stream JWS when it reaches the arrival position (see FIG. 5 ). In this specific example, the arrival position of the jet water stream JWS can be rephrased as the position where the tip of the jet water stream JWS contacts the inner wall of the large intestine. The actual depth acquisition unit 23 also acquires an actual depth RDBX corresponding to the relative depth SDBX of the pixel position PPJ by applying the coordinate values ​​XJ and YJ of the pixel position PPJ and the pixel count PCJ to a predetermined regression equation RE. The predetermined regression equation RE can be obtained, for example, by applying multiple regression analysis to measurement data obtained by measuring the actual depth while varying the pixel position corresponding to the arrival position of the jet water stream in the endoscopic image and the pixel count PCJ corresponding to the width of the tip of the jet water stream when it reaches the arrival position. In addition, instead of acquiring the actual depth RDBX using a predetermined regression formula RE, the actual depth acquisition unit 23 may acquire, for example, the actual depth RDBX corresponding to the combination of coordinate values ​​XJ and YJ and the number of pixels PCJ from the aforementioned measurement data.

[0044] The actual depth acquisition unit 23 uses the relationship between the relative depth SDBX and the actual depth RDBX at pixel position PPJ of the endoscopic image EGB to calculate the actual depth RDB corresponding to the relative depth SDB at each pixel position other than pixel position PPJ of the endoscopic image EGB. Specifically, for example, if SDBX = 50, RDBX = N millimeters, and the relative depth SDBY at pixel position PPY other than pixel position PPJ is 30, the actual depth acquisition unit 23 calculates the actual depth RDBY at pixel position PPY by calculating N × (30 / 50). That is, the actual depth acquisition unit 23 can calculate the actual depth RDBY by multiplying the actual depth RDBX by the ratio of the relative depth SDBY to the relative depth SDBX. The actual depth acquisition unit 23 can acquire, for example, pixel values ​​at pixel positions PPJ and PPY of the monochrome image included in the relative depth information SDBJ as relative depths SDBX and SDBY.

[0045] The actual depth acquisition unit 23 uses the method described above to calculate an actual depth RDB corresponding to each relative depth SDB included in the relative depth information SDBJ. The actual depth acquisition unit 23 also associates the actual depth RDB calculated using the method described above with each pixel of the monochrome image included in the relative depth information SDBJ, thereby acquiring actual depth information RDBJ indicating the actual depth RDB for each pixel position of the endoscopic image EGB. The actual depth acquisition unit 23 also outputs the actual depth information RDBJ to the three-dimensional reconstruction unit 24.

[0046] According to the above-described process, the actual depth acquisition unit 23 can acquire actual depth information RDBJ based on the relative depth information SDBJ and the reference object information RBJ. Furthermore, according to the above-described process, the actual depth acquisition unit 23 can acquire actual depth RDBX corresponding to the relative depth SDBX at the pixel position PPJ by performing a calculation using the pixel position PPJ and the number of pixels PCJ. Furthermore, according to the above-described process, the actual depth acquisition unit 23 can acquire actual depth RDB corresponding to the relative depth SDB at each pixel position other than the pixel position PPJ by performing a calculation using the relationship between the relative depth SDBX and actual depth RDBX at the pixel position PPJ.

[0047] The three-dimensional reconstruction unit 24 generates a three-dimensional model TDMB by performing three-dimensional reconstruction processing based on the actual depth information RDBJ and the camera parameters of the endoscopic camera provided at the tip 38, and outputs the generated three-dimensional model TDMB to the lesion area identification unit 26. In other words, the three-dimensional reconstruction unit 24 can generate a three-dimensional model according to the actual depth information RDBJ.

[0048] The lesion detection unit 25 performs processing to detect the LES lesion from the endoscopic image EGB using a trained image recognition model, etc. The lesion detection unit 25 also acquires lesion detection information DCBJ as information indicating the position and size of the LES lesion in the endoscopic image EGB, and outputs the acquired lesion detection information DCBJ to the lesion area identification unit 26.

[0049] The lesion region identification unit 26 identifies, as the lesion region DAB, an area in the three-dimensional model TDMB corresponding to the lesion LES indicated by the lesion detection information DCBJ. In other words, the lesion region identification unit 26 identifies, as the lesion region DAB, an area in the three-dimensional model TDMB corresponding to the lesion LES detected from the endoscopic image EGB. The lesion region identification unit 26 also acquires lesion region information DABJ as information indicating the position and size of the lesion region DAB, and outputs the acquired lesion region information DABJ to the lesion size acquisition unit 27.

[0050] The lesion size acquisition unit 27 acquires a line segment LN connecting the two most distant positions in the lesion area DAB indicated by the lesion area information DABJ. The lesion size acquisition unit 27 also calculates the actual length LA of the major axis of the lesion LES included in the lesion area DAB by performing an operation using the length of the line segment LN. The lesion size acquisition unit 27 also acquires lesion size information LSBJ including the position and size of the lesion area DAB, the line segment LN, and the actual length LA, and outputs the acquired lesion size information LSBJ to the display image generation unit 28.

[0051] According to the processing described above, the lesion size acquisition unit 27 can acquire, as information indicating the actual size of the lesioned LES included in the lesion area DAB, the line segment LN indicating the position of the major axis of the lesioned LES and the actual length LA of the major axis of the lesioned LES. Furthermore, according to the processing described above, the lesion size acquisition unit 27 can acquire the line segment LN indicating the position of the major axis of the lesioned LES included in the lesion area DAB.

[0052] The display image generating unit 28 superimposes the line segment LN included in the lesion size information LSJB on the lesion LES of the endoscopic image EGB. The display image generating unit 28 also generates a rectangular frame RF corresponding to the position and size of the lesion area DAB included in the lesion size information LSBJ and superimposes the frame RF around the lesion LES of the endoscopic image EGB. The display image generating unit 28 also generates a character string corresponding to the actual length LA included in the lesion size information LSJB and superimposes the character string near the frame RF.

[0053] According to the above-described processing, the display image generating unit 28 can generate a display image HGC including an endoscopic image EGC, such as that shown in FIG. 6 , and output the generated display image HGC to the display device 2. The endoscopic image EGC includes the lesion LES, the jet water stream LWS, a frame RF, a line segment LN, and a character string indicating the actual length LA. According to the above-described processing, the display image generating unit 28 can present information regarding the actual size of the lesion LES included in the lesion area DAB. According to the above-described processing, the display image generating unit 28 can present information regarding the actual size of the lesion LES by superimposing the line segment LN indicating the position of the major axis of the lesion LES and the character string indicating the actual length LA of the major axis of the lesion LES on the endoscopic image EGB. According to the above-described processing, the display image generating unit 28 can generate a frame RF corresponding to the lesion area DAS and superimpose the generated frame RF around the lesion LES in the endoscopic image EGB. FIG. 6 is a diagram showing another example of an endoscopic image included in a display image generated by the endoscopic examination support device according to the present disclosure.

[0054] The endoscopic image EGC in Fig. 6 allows the user to intuitively grasp the position of the LES lesion contained in the endoscopic image EGC and the position of the major axis of the LES lesion. Furthermore, the endoscopic image EGC in Fig. 6 allows the user to grasp that the actual length of the major axis of the LES lesion is 18 mm.

[0055] [Processing Flow] Next, a description will be given of the flow of processing performed in the endoscopic examination support device 1. Fig. 7 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the present disclosure.

[0056] First, the endoscopic examination support device 1 acquires relative depth information indicating the relative depth for each pixel position of the endoscopic image (step S11).

[0057] Next, the endoscopic examination support device 1 detects a reference object contained in the endoscopic image used in the processing of step S11, and acquires reference object information indicating the position and size of the reference object within the endoscopic image (step S12).

[0058] Next, the endoscopic examination support device 1 performs processing using the relative depth information obtained in step S11 and the reference object information obtained in step S12 to obtain actual depth information indicating the actual depth for each pixel position of the endoscopic image used in the processing of step S11 (step S13).

[0059] Next, the endoscopic examination support device 1 performs a three-dimensional reconstruction process based on the actual depth information obtained in step S13 and the camera parameters of the endoscopic camera, thereby generating a three-dimensional model corresponding to the structure of the large intestine (intestinal tract) during the endoscopic examination (step S14).

[0060] Next, the endoscopic examination support device 1 detects a lesion from the endoscopic image used in the processing of step S11, and acquires lesion detection information indicating the position and size of the lesion in the endoscopic image (step S15).

[0061] Next, the endoscopic examination support device 1 identifies the area in the three-dimensional model obtained in step S14 corresponding to the lesion indicated by the lesion detection information obtained in step S15 as the lesion area, and obtains lesion area information indicating the position and size of the lesion area (step S16).

[0062] Next, the endoscopic examination support device 1 performs processing using the lesion area information obtained in step S16 to obtain lesion size information indicating the actual size of the lesion contained in the lesion area (step S17).

[0063] Next, the endoscopic examination support device 1 generates a display image based on the endoscopic image used in the processing of step S11 and the lesion size information obtained in step S17, and outputs the generated display image to the display device 2 (step S18).

[0064] As described above, this embodiment generates a three-dimensional model based on the actual depth for each pixel position in an endoscopic image, and identifies an area in the three-dimensional model corresponding to a lesion detected in the endoscopic image as a lesion area. Furthermore, as described above, this embodiment can acquire information indicating the actual size of a lesion included in the lesion area in the three-dimensional model, and display an image based on the acquired information. Therefore, this embodiment can present information regarding the accurate size of a lesion discovered during an endoscopic examination.

[0065] [Modifications] Modifications to the present embodiment will be described below. For simplicity, specific descriptions of parts to which the above-described processes can be applied will be omitted as appropriate.

[0066] (Variation 1) The display image generating unit 28 may generate a display image including a line segment indicating the position of the tip of the jet water stream in the endoscopic image, a character string corresponding to the actual depth of the arrival position of the jet water stream in the endoscopic image, and a character string corresponding to the width of the water outlet from which the jet water stream is ejected, based on, for example, the reference object information obtained from the reference object detecting unit 22 and the actual depth information obtained from the actual depth acquiring unit 23. According to this processing, the display image generating unit 28 can generate a display image including an endoscopic image EGP as shown in FIG. 8 . Specifically, the display image generating unit 28 can generate the endoscopic image EGP by superimposing, on the endoscopic image EGC, a line segment LP indicating the position of the tip of the jet water stream JWS, a character string corresponding to the actual depth RDBX of the arrival position of the jet water stream JWS, and a character string indicating the width WP of the water outlet from which the jet water stream JWS is ejected. Furthermore, the display image generation unit 28 can generate a display image including the endoscopic image EGP generated as described above, and output the generated display image to the display device 2. Fig. 8 is a diagram showing another example of an endoscopic image included in a display image generated by the endoscopic examination support device according to the present disclosure.

[0067] (Variation 2) Instead of a water jet, the reference detection unit 22 may detect, as a reference, a treatment tool, such as biopsy forceps, protruding forward from the forceps opening of the distal end portion 38 and in contact with the inner wall of the large intestine. The reference detection unit 22 may also acquire reference information indicating the position and size of a non-movable portion of the distal end of the treatment tool within the endoscopic image. In such a case, the actual depth acquisition unit 23 can acquire an actual depth corresponding to the relative depth of the pixel position by, for example, using the pixel position and number of pixels of the non-movable portion of the distal end of the treatment tool within the endoscopic image and performing a calculation similar to that for the water jet. The non-movable portion of the distal end of the treatment tool may include, for example, the shaft of the treatment tool.

[0068] (Variation 3) Instead of a water jet, the reference object detection unit 22 may detect an artificial object, such as a hemostatic clip, that is placed in contact with the inner wall of the large intestine as the reference object RB. The reference object detection unit 22 may also acquire reference object information that indicates the position and size of a predetermined portion of the artificial object within the endoscopic image. In such a case, the actual depth acquisition unit 23 can acquire the actual depth corresponding to the relative depth of the pixel position by performing a calculation similar to that for the water jet, using, for example, the pixel position and the number of pixels of the predetermined portion of the artificial object within the endoscopic image.

[0069] Second Embodiment FIG. 9 is a block diagram showing another example of the functional configuration of the endoscopic examination support device according to the present disclosure.

[0070] The endoscopic examination support device 500 has the same hardware configuration as the endoscopic examination support device 1. The endoscopic examination support device 500 also has an actual depth acquisition means 511, a three-dimensional model generation means 512, a lesion area identification means 513, and an information presentation means 514.

[0071] The actual depth acquisition means 511 can be realized, for example, by using the function of the actual depth acquisition unit 23. The three-dimensional model generation means 512 can be realized, for example, by using the function of the three-dimensional reconstruction unit 24. The lesion area identification means 513 can be realized, for example, by using the function of the lesion area identification unit 26. The information presentation means 514 can be realized, for example, by using the function of the display image generation unit 28.

[0072] FIG. 10 is a flowchart showing another example of processing performed in the endoscopic examination support device according to the present disclosure.

[0073] The actual depth acquisition means 511 acquires actual depth information indicating the actual depth for each pixel position of the endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of the reference object within the endoscopic image (step S51).

[0074] The three-dimensional model generating means 512 generates a three-dimensional model according to the actual depth information (step S52).

[0075] The lesion area specifying means 513 specifies, as the lesion area, an area in the three-dimensional model that corresponds to the lesion detected from the endoscopic image (step S53).

[0076] The information presenting means 514 presents information relating to the actual size of the lesion contained in the lesion area (step S54).

[0077] According to this embodiment, it is possible to present information relating to the exact size of a lesion discovered during an endoscopic examination.

[0078] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0079] (Supplementary Note 1) An endoscopic examination support device comprising: an actual depth acquisition means for acquiring actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image; a three-dimensional model generation means for generating a three-dimensional model according to the actual depth information; a lesion area identification means for identifying an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area; and an information presentation means for presenting information regarding the actual size of the lesion included in the lesion area.

[0080] (Supplementary Note 2) The endoscopic examination support device of Supplementary Note 1, wherein the information presenting means presents information related to the actual size of the lesion by superimposing a first line segment indicating the position of the major axis of the lesion and a first character string indicating the actual length of the major axis of the lesion on the endoscopic image.

[0081] (Supplementary Note 3) The endoscopic examination support device according to Supplementary Note 2, wherein the information presenting means generates a frame according to the lesion area and superimposes the generated frame around the lesion on the endoscopic image.

[0082] (Appendix 4) In the endoscopic examination support device of Appendix 2, when the reference object is a jet water stream ejected from the tip of the endoscope, the information presentation means superimposes on the endoscopic image a second line segment indicating the position of the tip of the jet water stream in the endoscopic image, a second character string corresponding to the actual depth of the arrival position of the jet water stream in the endoscopic image, and a third character string corresponding to the width of the water outlet from which the jet water stream is ejected.

[0083] (Appendix 5) In the endoscopic examination support device of Appendix 1, when the reference object is a jet water stream ejected from the tip of the endoscope, the actual depth acquisition means performs a calculation using a pixel position corresponding to the arrival position of the jet water stream in the endoscopic image and the number of pixels corresponding to the width of the tip of the jet water stream when it reaches the arrival position, thereby acquiring the actual depth corresponding to the relative depth of the pixel position.

[0084] (Appendix 6) The actual depth acquisition means of the endoscopic examination support device of Appendix 5 acquires actual depths corresponding to the relative depths at each pixel position other than the pixel position of the endoscopic image by performing a calculation using the relationship between the relative depth and actual depth at the pixel position of the endoscopic image.

[0085] (Appendix 7) The actual depth acquisition means of the endoscopic examination support device of Appendix 1 acquires the actual depth corresponding to the relative depth of the pixel position by performing a calculation using the pixel position and number of pixels of the non-movable part of the tip of the treatment tool in the endoscopic image when the reference object is a treatment tool protruding from the tip of the endoscope and in contact with the inner wall of a tubular organ.

[0086] (Supplementary Note 8) The endoscopic examination support device according to Supplementary Note 1, wherein the relative depth information corresponds to an output image obtained by inputting the endoscopic image into a trained machine learning model.

[0087] (Supplementary Note 9) An endoscopic examination support method executed by a computer, comprising: acquiring actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image; generating a three-dimensional model according to the actual depth information; identifying an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area; and presenting information relating to the actual size of the lesion included in the lesion area.

[0088] (Supplementary Note 10) The endoscopic examination support method of Supplementary Note 9 presents information related to the actual size of the lesion by superimposing a first line segment indicating the position of the major axis of the lesion and a first character string indicating the actual length of the major axis of the lesion on the endoscopic image.

[0089] (Supplementary Note 11) The endoscopic examination support method according to Supplementary Note 10, further comprising generating a frame according to the lesion area and superimposing the generated frame around the lesion on the endoscopic image.

[0090] (Appendix 12) An endoscopic examination support method according to Appendix 10, in which, when the reference object is a jet water stream ejected from the tip of an endoscope, a second line segment indicating the position of the tip of the jet water stream in the endoscopic image, a second character string corresponding to the actual depth of the arrival position of the jet water stream in the endoscopic image, and a third character string corresponding to the width of the water outlet from which the jet water stream is ejected are superimposed on the endoscopic image.

[0091] (Appendix 13) An endoscopic examination support method according to Appendix 9, in which, when the reference object is a jet of water ejected from the tip of an endoscope, an actual depth corresponding to the relative depth of a pixel position is obtained by performing a calculation using a pixel position corresponding to the position where the jet of water flows in the endoscopic image and the number of pixels corresponding to the width of the tip of the jet of water when it reaches the position where the reference object is ejected from the tip of an endoscope.

[0092] (Appendix 14) An endoscopic examination support method according to Appendix 13, which obtains an actual depth corresponding to the relative depth at each pixel position other than the pixel position of the endoscopic image by performing a calculation using the relationship between the relative depth and the actual depth at the pixel position of the endoscopic image.

[0093] (Appendix 15) A recording medium having recorded thereon a program that causes a computer to execute the following processes: obtain actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image; generate a three-dimensional model according to the actual depth information; identify an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area; and present information related to the actual size of the lesion included in the lesion area.

[0094] (Appendix 16) A recording medium according to Appendix 15, having recorded thereon a program that causes a computer to execute a process of presenting information relating to the actual size of the lesion by superimposing, on the endoscopic image, a first line segment indicating the position of the major axis of the lesion and a first character string indicating the actual length of the major axis of the lesion.

[0095] (Supplementary Note 17) The recording medium of Supplementary Note 16, having recorded thereon a program for causing a computer to execute a process of generating a frame according to the lesion area and superimposing the generated frame around the lesion in the endoscopic image.

[0096] (Appendix 18) A recording medium according to Appendix 16, having recorded thereon a program that causes a computer to execute a process of superimposing, on the endoscopic image, when the reference object is a jet water stream ejected from the tip of an endoscope, a second line segment indicating the position of the tip of the jet water stream in the endoscopic image, a second character string corresponding to the actual depth of the arrival position of the jet water stream in the endoscopic image, and a third character string corresponding to the width of the water outlet from which the jet water stream is ejected.

[0097] (Appendix 19) An endoscopic examination support method according to Appendix 15, in which a program is recorded to cause a computer to execute a process of obtaining an actual depth corresponding to the relative depth of a pixel position by performing a calculation using a pixel position corresponding to the arrival position of the jet water stream in the endoscopic image and the number of pixels corresponding to the width of the tip of the jet water stream when it reaches the arrival position, when the reference object is a jet water stream ejected from the tip of an endoscope.

[0098] (Appendix 20) A recording medium according to Appendix 19, which stores a program that causes a computer to execute a process of obtaining an actual depth corresponding to the relative depth at each pixel position other than the pixel position of the endoscopic image by performing a calculation using the relationship between the relative depth and the actual depth at the pixel position of the endoscopic image.

[0099] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that would be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Each embodiment can be combined with other embodiments as appropriate. Furthermore, some or all of the configurations described in Supplements 2 to 8, which are dependent on Supplement 1, may also be dependent on Supplements 9 and 15 in the same manner as Supplements 2 to 8. Furthermore, without departing from the scope of each of the above-described embodiments, not limited to Supplement 1, Supplement 9, and Supplement 15, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording means for recording software, or systems.

[0100] REFERENCE SIGNS LIST 1 Endoscopy support device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 21 Relative depth acquisition unit 22 Reference object detection unit 23 Actual depth acquisition unit 24 Three-dimensional reconstruction unit 25 Lesion detection unit 26 Lesion area identification unit 27 Lesion size acquisition unit 28 Display image generation unit

Claims

1. An endoscopic examination support device having: an actual depth acquisition means for acquiring actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of the endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image; a three-dimensional model generation means for generating a three-dimensional model according to the actual depth information; a lesion area identification means for identifying an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area; and an information presentation means for presenting information relating to the actual size of the lesion contained in the lesion area.

2. The endoscopic examination support device of claim 1, wherein the information presentation means presents information relating to the actual size of the lesion by superimposing a first line segment indicating the position of the long diameter of the lesion and a first character string indicating the actual length of the long diameter of the lesion on the endoscopic image.

3. An endoscopic examination support device as described in claim 2, wherein the information presentation means generates a frame corresponding to the lesion area and superimposes the generated frame around the lesion in the endoscopic image.

4. The endoscopic examination support device of claim 2, wherein when the reference object is a jet water stream ejected from the tip of the endoscope, the information presentation means superimposes on the endoscopic image a second line segment indicating the position of the tip of the jet water stream in the endoscopic image, a second character string corresponding to the actual depth of the arrival position of the jet water stream in the endoscopic image, and a third character string corresponding to the width of the water outlet port from which the jet water stream is ejected.

5. The endoscopic examination support device of claim 1, wherein the actual depth acquisition means, when the reference object is a jet water stream ejected from the tip of an endoscope, acquires an actual depth corresponding to the relative depth of a pixel position by performing a calculation using a pixel position corresponding to the arrival position of the jet water stream in the endoscopic image and a number of pixels corresponding to the width of the tip of the jet water stream when it reaches the arrival position.

6. The endoscopic examination support device of claim 5, wherein the actual depth acquisition means acquires an actual depth corresponding to the relative depth at each pixel position other than the pixel position of the endoscopic image by performing a calculation using the relationship between the relative depth and the actual depth at the pixel position of the endoscopic image.

7. The endoscopic examination support device of claim 1, wherein the actual depth acquisition means, when the reference object is a treatment tool protruding from the tip of the endoscope and in contact with the inner wall of a tubular organ, acquires an actual depth corresponding to the relative depth of the pixel position by performing a calculation using the pixel position and pixel number of the non-moving part of the tip of the treatment tool in the endoscopic image.

8. An endoscopic examination support device as described in claim 1, wherein the relative depth information corresponds to an output image obtained by inputting the endoscopic image into a trained machine learning model.

9. An endoscopic examination support method executed by a computer, which acquires actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of an endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image, generates a three-dimensional model according to the actual depth information, identifies an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area, and presents information related to the actual size of the lesion contained in the lesion area.

10. An endoscopic examination support method as described in claim 9, in which information relating to the actual size of the lesion is presented by superimposing a first line segment indicating the position of the long diameter of the lesion and a first character string indicating the actual length of the long diameter of the lesion on the endoscopic image.

11. The method for supporting endoscopic examination according to claim 10, further comprising generating a frame corresponding to the lesion area and superimposing the generated frame around the lesion in the endoscopic image.

12. An endoscopic examination support method as described in claim 10, in which, when the reference object is a jet water stream ejected from the tip of an endoscope, a second line segment indicating the position of the tip of the jet water stream in the endoscopic image, a second character string corresponding to the actual depth of the arrival position of the jet water stream in the endoscopic image, and a third character string corresponding to the width of the water outlet port from which the jet water stream is ejected are superimposed on the endoscopic image.

13. An endoscopic examination support method as described in claim 9, in which, when the reference object is a jet water stream ejected from the tip of an endoscope, an actual depth corresponding to the relative depth of a pixel position is obtained by performing a calculation using a pixel position corresponding to the arrival position of the jet water stream in the endoscopic image and a number of pixels corresponding to the width of the tip of the jet water stream when it reaches the arrival position.

14. An endoscopic examination support method as described in claim 13, in which an actual depth corresponding to the relative depth at each pixel position other than the pixel position of the endoscopic image is obtained by performing a calculation using the relationship between the relative depth and the actual depth at the pixel position of the endoscopic image.

15. A recording medium having a program recorded thereon that causes a computer to execute the following processes: obtain actual depth information indicating the actual depth for each pixel position of an endoscopic image based on relative depth information indicating the relative depth for each pixel position of an endoscopic image and reference object information indicating the position and size of a reference object within the endoscopic image; generate a three-dimensional model according to the actual depth information; identify an area in the three-dimensional model corresponding to a lesion detected from the endoscopic image as a lesion area; and present information regarding the actual size of the lesion contained in the lesion area.

16. A recording medium as described in claim 15, having recorded thereon a program that causes a computer to execute a process of presenting information relating to the actual size of the lesion by superimposing a first line segment indicating the position of the long diameter of the lesion and a first character string indicating the actual length of the long diameter of the lesion on the endoscopic image.

17. A recording medium according to claim 16, having recorded thereon a program for causing a computer to execute a process of generating a frame corresponding to the lesion area and superimposing the generated frame around the lesion in the endoscopic image.

18. A recording medium as described in claim 16, having recorded thereon a program for causing a computer to execute a process of superimposing, when the reference object is a jet water stream ejected from the tip of an endoscope, a second line segment indicating the position of the tip of the jet water stream in the endoscopic image, a second character string corresponding to the actual depth of the arrival position of the jet water stream in the endoscopic image, and a third character string corresponding to the width of the water outlet port from which the jet water stream is ejected, on the endoscopic image.

19. A recording medium as described in claim 15, having recorded thereon a program for causing a computer to execute a process for obtaining an actual depth corresponding to a relative depth of a pixel position by performing a calculation using a pixel position corresponding to the arrival position of the jet water stream in the endoscopic image and a number of pixels corresponding to the width of the tip of the jet water stream when it reaches the arrival position, when the reference object is a jet water stream ejected from the tip of an endoscope.

20. A recording medium as described in claim 19, having recorded thereon a program that causes a computer to execute a process of obtaining an actual depth corresponding to the relative depth at each pixel position other than the pixel position of the endoscopic image by performing a calculation using the relationship between the relative depth and actual depth at the pixel position of the endoscopic image.

Images