Endoscopic examination assistance device, endoscopic examination assistance method, and recording medium

Abstract

In this endoscopic examination assistance device, a lesion detection means detects at least one lesion included in an endoscopic image obtained during endoscopic examination. An estimation means estimates the infiltration state of the lesion detected from the endoscopic image. A visual information generation means generates visual information indicating an estimation result of the infiltration state of the lesion. A display image generation means generates a display image including the endoscopic image and the visual information. The endoscopic examination assistance device can be used for decision-making assistance of an operator who performs treatment of the lesion discovered during endoscopic examination.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

[0001] The present disclosure relates to techniques that can be used to present information to assist in endoscopic examinations.

[0002] Techniques have been proposed to assist endoscopy.

[0003] Specifically, for example, Patent Document 1 discloses a method for supporting diagnosis of diseases using endoscopic images of the digestive organs.

[0004] Japanese Patent Application Laid-Open No. 2020-078539

[0005] However, Patent Document 1 does not specifically disclose a method for presenting information that enables an operator to intuitively grasp the infiltration state of a lesion discovered during an endoscopic examination. Therefore, the technology disclosed in Patent Document 1 has a problem in that, for example, an excessive burden may be imposed on an operator who treats a lesion discovered during an endoscopic examination.

[0006] One object of the present disclosure is to provide an endoscopic examination support device that can reduce the burden imposed on an operator who treats a lesion discovered during an endoscopic examination.

[0007] In one aspect of the present disclosure, an endoscopic examination support device includes a lesion detection means for detecting at least one lesion contained in an endoscopic image obtained during an endoscopic examination, an estimation means for estimating the infiltration state of the lesion detected from the endoscopic image, a visual information generation means for generating visual information indicating the estimation result of the infiltration state of the lesion, and a display image generation means for generating a display image including the endoscopic image and the visual information.

[0008] In another aspect of the present disclosure, an endoscopic examination support method detects at least one lesion included in an endoscopic image obtained during an endoscopic examination, estimates the infiltration state of the lesion detected from the endoscopic image, generates visual information indicating the estimated result of the infiltration state of the lesion, and generates a display image including the endoscopic image and the visual information.

[0009] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of detecting at least one lesion included in an endoscopic image obtained during an endoscopic examination, estimating the infiltration state of the lesion detected from the endoscopic image, generating visual information indicating the estimated result of the infiltration state of the lesion, and generating a display image including the endoscopic image and the visual information.

[0010] According to the present disclosure, it is possible to reduce the burden imposed on an operator who treats a lesion discovered during an endoscopic examination.

[0011] 1 is a diagram showing an example of a schematic configuration of an endoscopic examination system according to the present disclosure. FIG. 1 is a block diagram showing an example of a hardware configuration of an endoscopic examination support device according to the present disclosure. FIG. 2 is a block diagram showing an example of a functional configuration of an endoscopic examination support device according to the present disclosure. FIG. 3 is a diagram showing an example of an endoscopic image including a lesion. FIG. 4 is a diagram showing an example of a display image generated using the endoscopic image of FIG. 4. FIG. 5 is a diagram showing another example of an endoscopic image including a lesion. FIG. 6 is a diagram showing an example of a display image generated using the endoscopic image of FIG. 6. FIG. 7 is a diagram showing another example of an endoscopic image including a lesion. FIG. 8 is a diagram showing another example of a display image generated using the endoscopic image of FIG. 8. FIG. 9 is a diagram showing another example of a display image generated using the endoscopic image of FIG. 9. FIG. 10 is a diagram showing another example of an endoscopic image including a lesion. FIG. 11 is a diagram showing an example of a display image generated using the endoscopic image of FIG. 13. FIG. 12 is a diagram showing another example of a display image generated using the endoscopic image of FIG. 13. FIG. 13 is a diagram showing another example of a display image generated using the endoscopic image of FIG. 13. A flowchart showing an example of processing performed in an endoscopic examination support device according to the present disclosure. A block diagram showing another example of the functional configuration of an endoscopic examination support device according to the present disclosure. 10 is a flowchart showing another example of processing performed in the endoscopic examination support device according to the present disclosure.

[0012] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0013] <First embodiment> [System configuration] Fig. 1 is a diagram showing an example of a schematic configuration of an endoscopic examination system according to the present disclosure. As shown in Fig. 1, the endoscopic examination system 100 includes an endoscopic examination support device 1, a display device 2, and an endoscope 3 connected to the endoscopic examination support device 1.

[0014] The endoscopic examination support device 1 acquires, from the endoscope 3, video including time-series images obtained by capturing images of a subject during an endoscopic examination (hereinafter also referred to as endoscopic video), and displays a display image on the display device 2 for review by an operator, such as a doctor, performing the endoscopic examination. Specifically, the endoscopic examination support device 1 acquires, from the endoscope 3, video of the interior of the large intestine obtained during the endoscopic examination as endoscopic video. The endoscopic examination support device 1 also extracts endoscopic images from the endoscopic video and performs a process of detecting lesions from the extracted endoscopic images. In the following description, unless otherwise specified, it is assumed that a tumorous lesion is included in the endoscopic image. Furthermore, when a lesion is detected from the endoscopic image, the endoscopic examination support device 1 also performs a process of estimating the infiltration state of the lesion. Furthermore, the endoscopic examination support device 1 displays, on the display device 2, a display image including information enabling understanding of the infiltration state of the lesion detected from the endoscopic image. The endoscopic examination support device 1 can be used to support the decision-making of an operator in treating a lesion discovered during an endoscopic examination.

[0015] The display device 2 has, for example, a liquid crystal monitor, etc. The display device 2 displays images, etc. output from the endoscopic examination support device 1.

[0016] The endoscope 3 mainly has an operation unit 36 ​​that allows the surgeon to input instructions such as air supply, water supply, angle adjustment, and photography instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that has an endoscopic camera such as an ultra-small imaging element built in, and a connection unit 39 for connecting to the endoscopic examination support device 1.

[0017] 2 is a block diagram showing an example of the hardware configuration of an endoscopic examination support device according to the present disclosure. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0018] The processor 11 executes predetermined processes by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer. The processor 11 also performs processes such as generating a display image including information that enables the infiltration state of a lesion to be grasped.

[0019] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and nonvolatile memories that store information necessary for processing by the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs for the endoscopic examination support device 1 to execute each process in this embodiment.

[0020] Furthermore, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images captured by the endoscope 3 during an endoscopic examination.

[0021] The interface 13 acts as an interface between the endoscopic examination assistance device 1 and an external device. For example, the interface 13 supplies a display image generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating an endoscopic video supplied from the endoscope 3 to the processor 11. The interface 13 also supplies an endoscopic image extracted from the endoscopic video to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0022] The input unit 14 generates an input signal in response to an operation by the surgeon. The input unit 14 has at least one device selected from the group consisting of a button, a touch panel, a remote controller, a foot switch, and a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump for delivering water or air to the endoscope 3. The sound output unit 16 outputs sound under the control of the processor 11.

[0023] The DB 17 stores endoscopic images and the like acquired during past endoscopic examinations of the subject. The DB 17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory. Instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be acquired from the server via communication.

[0024] The endoscopic examination support device 1 may also be provided with a sensor, such as a magnetic sensor, that can measure the rotation and translation of the endoscopic camera.

[0025] [Functional Configuration] Fig. 3 is a block diagram showing an example of the functional configuration of the endoscopic examination support device 1 according to the present disclosure. As shown in Fig. 3, the endoscopic examination support device 1 includes a lesion detection unit 21, an infiltration state estimation unit 22, and a display image generation unit 23.

[0026] The lesion detection unit 21 functions as a lesion detection means. The lesion detection unit 21 can detect at least one lesion included in an endoscopic image obtained during an endoscopic examination. The lesion detection unit 21 also performs processing to detect a lesion TU from the endoscopic image EG. The lesion detection unit 21 also acquires position information PJ indicating the position of the lesion TU detected from the endoscopic image EG, and outputs the endoscopic image EG and the position information PJ to the infiltration state estimation unit 22. If the lesion detection unit 21 is unable to detect a lesion from the endoscopic image, it does not output information to the infiltration state estimation unit 22.

[0027] The infiltration state estimation unit 22 functions as an estimation means. The infiltration state estimation unit 22 can estimate the infiltration state of a lesion detected from an endoscopic image. When multiple lesions are detected from an endoscopic image, the infiltration state estimation unit 22 can estimate the infiltration state of each of the multiple lesions. The infiltration state estimation unit 22 also has an estimation model 22A that has been trained to estimate the infiltration distance and infiltration direction of the lesion contained in the image as the infiltration state of the lesion.

[0028] The estimation model 22A is configured as a machine learning model trained using training data that associates, for example, an image containing a lesion with the infiltration distance and infiltration direction of the lesion for each pixel constituting the lesion in the image. The infiltration distance included in the training data is preferably, for example, a measured value of the distance from the boundary between the diseased tissue and non-lesion tissue to the deepest infiltration position. Furthermore, the infiltration direction included in the training data is preferably, for example, the direction in which the measured value was measured.

[0029] The infiltration state estimation unit 22 inputs the endoscopic image EG and position information PJ obtained from the lesion detection unit 21 into the estimation model 22A, thereby estimating the infiltration distance and infiltration direction of the lesion TU for each pixel constituting the lesion TU in the endoscopic image EG. That is, the infiltration state estimation unit 22 can estimate the infiltration state of the lesion TU included in the endoscopic image EG using the estimation model 22A. The infiltration state estimation unit 22 then outputs the position information PJ obtained from the lesion detection unit 21 and the estimation result ER obtained by the above-mentioned estimation to the display image generation unit 23. Note that if the infiltration state estimation unit 22 does not obtain the endoscopic image and position information from the lesion detection unit 21, it does not perform processing related to estimating the infiltration distance and infiltration direction of the lesion.

[0030] The display image generation unit 23 functions as a display image generation means, a visual information generation means, and an infiltration state image generation means. Furthermore, the display image generation unit 23 identifies the position of the lesion TU included in the endoscopic image EG based on the position information PJ obtained from the infiltration state estimation unit 22. Furthermore, the display image generation unit 23 generates visual information VJ indicating the infiltration state of the lesion TU based on the infiltration distance and infiltration direction of the lesion TU indicated by the estimation result ER obtained from the infiltration state estimation unit 22. That is, the display image generation unit 23 can generate visual information indicating the estimation result of the infiltration state of the lesion. Furthermore, the display image generation unit 23 generates a display image HG including the endoscopic image EG and the visual information VJ, and outputs the generated display image HG to the display device 2. If the display image generation unit 23 cannot obtain position information and an estimation result from the infiltration state estimation unit 22, it outputs a display image including the endoscopic image EG but not the visual information VJ to the display device 2.

[0031] According to this embodiment, the display image generating unit 23 may generate, for example, information indicating the infiltration state in a cross section obtained by cutting the lesion TU longitudinally as the visual information VJ. Also, according to this embodiment, the display image generating unit 23 may generate, for example, information indicating the infiltration state of the lesion TU in three dimensions as the visual information VJ.

[0032] [Specific Example] Next, a specific example of processing related to generation of a display image will be described.

[0033] Specific Example 1 When an endoscopic image EGA such as that shown in Fig. 4 is input, the lesion detection unit 21 detects a lesion TUA included in the endoscopic image EGA. The lesion detection unit 21 also acquires position information PJA indicating the position of the lesion TUA in the endoscopic image EGA, and outputs the endoscopic image EGA and the position information PJA to the infiltration state estimation unit 22. Fig. 4 is a diagram showing an example of an endoscopic image including a lesion.

[0034] The infiltration state estimation unit 22 estimates the infiltration distance and infiltration direction of the lesion TUA for each pixel constituting the lesion TUA in the endoscopic image EGA by inputting the endoscopic image EGA and the position information PJA to the estimation model 22A. The infiltration state estimation unit 22 then outputs the position information PJA and the estimation result ERA obtained by the above-mentioned estimation to the display image generation unit 23.

[0035] The display image generation unit 23 identifies the position of the lesion TUA included in the endoscopic image EGA based on the position information PJA. The display image generation unit 23 also generates visual information VJA indicating the infiltration state of the lesion TUA based on the infiltration distance and infiltration direction of the lesion TUA indicated by the estimation result ERA. The display image generation unit 23 also generates a display image HGA as shown in FIG. 5 as a display image including the endoscopic image EGA and the visual information VJA, and outputs the generated display image HGA to the display device 2. FIG. 5 is a diagram illustrating an example of a display image generated using the endoscopic image of FIG. 4.

[0036] In the display image HGA of Fig. 5, the visual information VJA is superimposed on a pixel group corresponding to the infiltration distance and infiltration direction of the lesion TUA in the endoscopic image EGA. Also, in the display image HGA of Fig. 5, the area corresponding to the visual information VJA in the endoscopic image EGA is colored in a color different from the color of the biological tissue.

[0037] According to the above-described processing, the display image generating unit 23 can generate a display image including an endoscopic image and visual information superimposed on the endoscopic image at a position corresponding to the lesion. Furthermore, according to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUA included in the endoscopic image EGA by checking the visual information VJA of the endoscopic image EGA.

[0038] (Specific Example 2) When an endoscopic image EGB such as that shown in Fig. 6 is input, the lesion detection unit 21 detects a lesion TUB included in the endoscopic image EGB. The lesion detection unit 21 also acquires position information PJB indicating the position of the lesion TUB in the endoscopic image EGB, and outputs the endoscopic image EGB and the position information PJB to the infiltration state estimation unit 22. Fig. 6 is a diagram showing another example of an endoscopic image including a lesion.

[0039] The infiltration state estimation unit 22 estimates the infiltration distance and infiltration direction of the lesion TUB for each pixel constituting the lesion TUB in the endoscopic image EGB by inputting the endoscopic image EGB and the position information PJB to the estimation model 22A. Then, the infiltration state estimation unit 22 outputs the position information PJB and the estimation result ERB obtained by the above-mentioned estimation to the display image generation unit 23.

[0040] The display image generator 23 identifies the position of the lesion TUB included in the endoscopic image EGB based on the position information PJB. The display image generator 23 also generates visual information VJB indicating the infiltration state of the lesion TUB based on the infiltration distance and infiltration direction of the lesion TUB indicated by the estimation result ERB.

[0041] When the size of the lesion TUB in the endoscopic image EGB is equal to or larger than a predetermined size THS, the display image generating unit 23 generates a display image similar to that shown in Fig. 5 as a display image including the endoscopic image EGB and the visual information VJB, and outputs the generated display image to the display device 2. Note that the predetermined size THS may be set, for example, as the number of pixels, or may be set in units such as millimeters.

[0042] If the size of the lesion TUB in the endoscopic image EGB is smaller than a predetermined size THS, the display image generation unit 23 extracts a rectangular area including the lesion TUB from the endoscopic image EGB and enlarges the rectangular area to generate an enlarged image ZGB. The display image generation unit 23 also generates a display image HGB as shown in Fig. 7 as a display image including the endoscopic image EGB, visual information VJB, and the enlarged image ZGB, and outputs the generated display image HGB to the display device 2. Fig. 7 is a diagram showing an example of a display image generated using the endoscopic image of Fig. 6.

[0043] 7, the visual information VJB is superimposed on a pixel group in the enlarged image ZGB corresponding to the infiltration distance and infiltration direction of the lesion TUB. Also, in the display image HGB in FIG. 7, the area in the enlarged image ZGB corresponding to the visual information VJB is colored in a color different from the color of the living tissue.

[0044] According to the above-described processing, when the size of a lesion in an endoscopic image is less than a predetermined size, the display image generating unit 23 can generate a display image including the endoscopic image, an enlarged image of an area including the lesion, and visual information superimposed on the enlarged image at a position corresponding to the lesion. Furthermore, according to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUB included in the endoscopic image EGB by checking the visual information VJB of the enlarged image ZGB.

[0045] (Specific Example 3) When an endoscopic image EGC such as that shown in Fig. 8 is input, the lesion detection unit 21 detects lesions TUC1, TUC2, and TUC3 included in the endoscopic image EGC. The lesion detection unit 21 also acquires position information PJC that can individually identify the positions of lesions TUC1, TUC2, and TUC3 in the endoscopic image EGC. The lesion detection unit 21 also outputs the endoscopic image EGC and the position information PJC to the infiltration state estimation unit 22. Fig. 8 is a diagram showing another example of an endoscopic image including a lesion.

[0046] The infiltration state estimation unit 22 estimates the infiltration distance and infiltration direction of each pixel constituting the lesion for each of the lesions TUC1, TUC2, and TU3 in the endoscopic image EGB by inputting the endoscopic image EGC and the position information PJC to the estimation model 22A. The infiltration state estimation unit 22 then outputs the position information PJC and the estimation result ERC obtained by the above-mentioned estimation to the display image generation unit 23.

[0047] The display image generator 23 identifies the positions of the lesions TUC1, TUC2, and TUC3 included in the endoscopic image EGC based on the position information PJC. Furthermore, for each of the lesions TUC1, TUC2, and TU3 included in the endoscopic image EGB, the display image generator 23 generates visual information indicating the infiltration state of the lesion based on the infiltration distance and infiltration direction of the lesion indicated by the estimation result ERC. Through this processing, the display image generator 23 can generate visual information VJC1 indicating the infiltration state of the lesion TUC1, visual information VJC2 indicating the infiltration state of the lesion TUC2, and visual information VJC3 indicating the infiltration state of the lesion TUC3.

[0048] The display image generation unit 23 extracts rectangular regions containing the lesions TUC1, TUC2, and TU3 from the endoscopic image EGC and enlarges the rectangular regions to generate enlarged images. Through this processing, the display image generation unit 23 can generate an enlarged image ZGC1 by enlarging the rectangular region containing lesion TUC1 in the endoscopic image EGC, an enlarged image ZGC2 by enlarging the rectangular region containing lesion TUC2 in the endoscopic image EGC, and an enlarged image ZGC3 by enlarging the rectangular region containing lesion TUC3 in the endoscopic image EGC. The display image generation unit 23 also generates an infiltration status image SGC1 by combining lesion TUC1 with visual information VJC1 resized according to the size of lesion TUC1. The display image generation unit 23 also generates an infiltration status image SGC2 by combining lesion TUC2 with visual information VJC2 resized according to the size of lesion TUC2. The display image generator 23 generates an infiltration state image SGC3 by combining the lesion TUC3 with visual information VJC3 resized according to the size of the lesion TUC3. The display image generator 23 can generate the infiltration state images SGC1 to SGC3 by performing image processing such as removing the background around the lesion.

[0049] The display image generating unit 23 generates a display image HGC1 as shown in Fig. 9 as a display image including the endoscopic image EGC, an enlarged image ZGC1, and an infiltration state image SGC1. The display image generating unit 23 also generates a display image HGC2 as shown in Fig. 10 as a display image including the endoscopic image EGC, an enlarged image ZGC2, and an infiltration state image SGC2. The display image generating unit 23 also generates a display image HGC3 as shown in Fig. 11 as a display image including the endoscopic image EGC, an enlarged image ZGC3, and an infiltration state image SGC3. Fig. 9 is a diagram showing an example of a display image generated using the endoscopic image of Fig. 8. Figs. 10 and 11 are diagrams showing other examples of display images generated using the endoscopic image of Fig. 8.

[0050] The display image generation unit 23 sequentially outputs the display images HGC1, HGC2, and HGC3 to the display device 2 so that the display is switched at regular intervals. Specifically, the display image generation unit 23 switches the display images to be output to the display device 2 in the order of HGC1 → HGC2 → HGC3 → HGC1 ..., for example, every time a regular interval elapses. Furthermore, when the display image generation unit 23 outputs one of the display images HGC1, HGC2, and HGC3 to the display device 2, it generates a frame WGC indicating a lesion corresponding to the enlarged image and infiltration state image in that one display image, and superimposes the generated frame WGC around that lesion on the endoscopic image.

[0051] In the display image HGC1 of Fig. 9 , a frame WGC corresponding to the enlarged image ZGC1 and the infiltration state image SGC1 is superimposed around the lesion TUC1 in the endoscopic image EGC. Furthermore, in the display image HGC1 of Fig. 9 , the orientations of the lesion TUC1 and the visual information VJC1 in the infiltration state image SGC1 are set so that the infiltration direction of the lesion TUC1 coincides with the vertical direction of the display image HGC1. Furthermore, in the display image HGC1 of Fig. 9 , the infiltration state image SGC1 is displayed below the enlarged image ZGC1.

[0052] In the display image HGC2 of Fig. 10 , a frame WGC corresponding to the enlarged image ZGC2 and the infiltration state image SGC2 is superimposed around the lesion TUC2 in the endoscopic image EGC. Furthermore, in the display image HGC2 of Fig. 10 , the orientations of the lesion TUC2 and the visual information VJC2 in the infiltration state image SGC2 are set so that the infiltration direction of the lesion TUC2 coincides with the vertical direction of the display image HGC2. Furthermore, in the display image HGC2 of Fig. 10 , the infiltration state image SGC2 is displayed below the enlarged image ZGC2.

[0053] In the display image HGC3 of Fig. 11 , a frame WGC corresponding to the enlarged image ZGC3 and the infiltration state image SGC3 is superimposed around the lesion TUC3 in the endoscopic image EGC. Furthermore, in the display image HGC3 of Fig. 11 , the orientations of the lesion TUC3 and the visual information VJC3 in the infiltration state image SGC3 are set so that the infiltration direction of the lesion TUC3 coincides with the vertical direction of the display image HGC3. Furthermore, in the display image HGC3 of Fig. 11 , the infiltration state image SGC3 is displayed below the enlarged image ZGC3.

[0054] According to the above-described processing, the display image generating unit 23 can generate multiple pieces of visual information indicating the estimated infiltration state of each of multiple lesions. According to the above-described processing, the display image generating unit 23 can generate multiple infiltration state images by combining multiple lesions with multiple pieces of visual information. According to the above-described processing, the display image generating unit 23 can generate a display image by sequentially switching between multiple infiltration state images. According to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUC1 included in the endoscopic image EGC by checking the visual information VJC1 of the infiltration state image SGC1. According to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUC2 included in the endoscopic image EGC by checking the visual information VJC2 of the infiltration state image SGC2. According to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUC3 included in the endoscopic image EGC by checking the visual information VJC3 of the infiltration state image SGC3.

[0055] 8 is input, the lesion detection unit 21 acquires position information PJC by performing the same processing as in Specific Example 3. The lesion detection unit 21 also outputs the endoscopic image EGC and the position information PJC to the infiltration state estimation unit 22.

[0056] The infiltration state estimation unit 22 obtains the estimation result ERC by performing estimation similar to that in Specific Example 3. Furthermore, the infiltration state estimation unit 22 outputs the position information PJC and the estimation result ERC to the display image generation unit 23.

[0057] The display image generation unit 23 identifies the positions of the lesions TUC1, TUC2, and TUC3 included in the endoscopic image EGC based on the position information PJC. Furthermore, by performing the same processing as in specific example 3, the display image generation unit 23 generates visual information VJC1 indicating the infiltration state of the lesion TUC1, visual information VJC2 indicating the infiltration state of the lesion TUC2, and visual information VJC3 indicating the infiltration state of the lesion TUC3.

[0058] The display image generating unit 23 extracts rectangular regions containing the lesions from the endoscopic image EGC for each of lesions TUC1, TUC2, and TU3 contained in the endoscopic image EGC. Through this processing, the display image generating unit 23 can extract a partial image PGC1 corresponding to the rectangular region containing lesion TUC1 in the endoscopic image EGC, a partial image PGC2 corresponding to the rectangular region containing lesion TUC2 in the endoscopic image EGC, and a partial image PGC3 corresponding to the rectangular region containing lesion TUC3 in the endoscopic image EGC. The display image generating unit 23 also generates an infiltration status image SGC4 by combining lesion TUC1 with visual information VJC1 resized according to the size of lesion TUC1. The display image generating unit 23 also generates an infiltration status image SGC5 by combining lesion TUC2 with visual information VJC2 resized according to the size of lesion TUC2. The display image generator 23 generates an infiltration state image SGC6 by combining the lesion TUC3 with visual information VJC3 resized according to the size of the lesion TUC3. The display image generator 23 can generate the infiltration state images SGC4 to SGC6 by performing image processing such as removing the background around the lesion.

[0059] The display image generating unit 23 generates a display image HGC4 as shown in Fig. 12 as a display image including the endoscopic image EGC, partial images PGC1 to PGC3, and infiltration state images SGC4 to SGC6. Furthermore, when outputting the display image HGC4 to the display device 2, the display image generating unit 23 superimposes, for each of the lesions TUC1, TUC2, and TU3, an identifier indicating the correspondence between the lesion included in the endoscopic image EGC and the lesion included in the partial images PGC1 to PGC3. Fig. 12 is a diagram showing another example of a display image generated using the endoscopic image of Fig. 8.

[0060] In the display image HGC4 of Fig. 12, partial images PGC1, PGC2, and PGC3 are displayed side by side in the horizontal direction. Furthermore, in the display image HGC4 of Fig. 12, an infiltration state image SGC4 is displayed below partial image PGC1, an infiltration state image SGC5 is displayed below partial image PGC2, and an infiltration state image SGC6 is displayed below partial image PGC3. Furthermore, in the display image HGC4 of Fig. 12, an identifier "1" is superimposed near lesion TUC1 in the endoscopic image EGC and on the top of partial image PGC1. Furthermore, in the display image HGC4 of Fig. 12, an identifier "2" is superimposed near lesion TUC2 in the endoscopic image EGC and on the top of partial image PGC2. Furthermore, according to the display image HGC4 in FIG. 12, the identifier "3" is superimposed near the lesion TUC3 in the endoscopic image EGC and on the top of the partial image PGC3.

[0061] According to the above-described processing, the display image generating unit 23 can generate multiple pieces of visual information indicating the estimated infiltration state of each of multiple lesions. According to the above-described processing, the display image generating unit 23 can generate multiple infiltration state images by combining multiple lesions and multiple pieces of visual information. According to the above-described processing, the display image generating unit 23 can generate a display image including an endoscopic image and multiple infiltration state images. According to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUC1 included in the endoscopic image EGC by checking the visual information VJC1 of the infiltration state image SGC4. According to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUC2 included in the endoscopic image EGC by checking the visual information VJC2 of the infiltration state image SGC5. According to the above-described processing, the surgeon can intuitively grasp the infiltration state of the lesion TUC3 included in the endoscopic image EGC by checking the visual information VJC3 of the infiltration state image SGC6.

[0062] According to this specific example, it is desirable for the display image generation unit 23 to generate a display image that simultaneously displays multiple partial images and multiple infiltration state images. Also, according to this specific example, the display image generation unit 23 may change the number of partial images and infiltration state images simultaneously displayed in the display image HGC4, for example, depending on the display size of the display image HGC4.

[0063] (Specific Example 5) When an endoscopic image EGS such as that shown in FIG. 13 is input, the lesion detection unit 21 detects a lesion TUS included in the endoscopic image EGS. The lesion detection unit 21 also acquires position information PJS indicating the position of the lesion TUS in the endoscopic image EGS and outputs the endoscopic image EGS and the position information PJS to the infiltration state estimation unit 22. The lesion detection unit 21 also determines whether a lesion TUS is included in an endoscopic image EGV input after the endoscopic image EGS. If the lesion TUS is included in the endoscopic image EGV, the lesion detection unit 21 acquires position information PJV indicating the position of the lesion TUS in the endoscopic image EGV and outputs the endoscopic image EGV and the position information PJV to the infiltration state estimation unit 22. If the lesion TUS is not included in the endoscopic image EGV, the lesion detection unit 21 does not output information to the infiltration state estimation unit 22. FIG. 13 is a diagram showing another example of an endoscopic image including a lesion.

[0064] The infiltration state estimation unit 22 inputs the endoscopic image EGS and position information PJS into the estimation model 22A, thereby estimating the infiltration distance and infiltration direction of the lesion TUS for each pixel constituting the lesion TUS in the endoscopic image EGS. The infiltration state estimation unit 22 then outputs the position information PJS and the estimation result ERS obtained by the above-mentioned estimation to the display image generation unit 23. The infiltration state estimation unit 22 also inputs the endoscopic image EGV and position information PJV into the estimation model 22A, thereby estimating the infiltration distance and infiltration direction of the lesion TUS for each pixel constituting the lesion TUS in the endoscopic image EGV. The infiltration state estimation unit 22 then outputs the position information PJV and the estimation result ERV obtained by the above-mentioned estimation to the display image generation unit 23. If the infiltration state estimation unit 22 does not receive an endoscopic image and position information from the lesion detection unit 21, the infiltration state estimation unit 22 does not perform processing related to estimation of the infiltration distance and infiltration direction of the lesion.

[0065] Here, the endoscopic image EGS corresponds to, for example, a still image obtained in response to operation of the operation unit 36 ​​by the surgeon. The endoscopic image EGV corresponds to an endoscopic image extracted from the endoscopic video after the acquisition of the endoscopic image EGS. According to this specific example, the lesion detection unit 21 and the infiltration state estimation unit 22 do not perform processing during the period before the endoscopic image EGS is input. According to this specific example, the display image generation unit 23 performs processing to output to the display device 2 a display image that includes the endoscopic image extracted from the endoscopic video but does not include the visual information VJS and VJV described below during the period before the endoscopic image EGS is input. According to this specific example, the lesion detection unit 21 and the infiltration state estimation unit 22 perform processing during the period from the input of the endoscopic image EGS including the lesion TUS until detection of the lesion TUS in the endoscopic image EGV ceases. Furthermore, according to this specific example, during the period from when detection of a lesion TUS in the endoscopic image EGV is discontinued until the next endoscopic image EGS is input, no processing is performed by the lesion detection unit 21 and the infiltration state estimation unit 22. Furthermore, according to this specific example, during the period from when detection of a lesion TUS in the endoscopic image EGV is discontinued until the next endoscopic image EGS is input, the display image generation unit 23 performs processing to output to the display device 2 a display image that includes an endoscopic image extracted from the endoscopic video but does not include visual information VJS and VJV, which will be described later.

[0066] The display image generation unit 23 identifies the position of the lesion TUS included in the endoscopic image EGS based on the position information PJS. The display image generation unit 23 also generates visual information VJS indicating the infiltration state of the lesion TUS based on the infiltration distance and infiltration direction of the lesion TUS indicated by the estimation result ERS. The display image generation unit 23 also identifies the position of the lesion TUS included in the endoscopic image EGV based on the position information PJV. The display image generation unit 23 also generates visual information VJV indicating the infiltration state of the lesion TUS based on the infiltration distance and infiltration direction of the lesion TUS indicated by the estimation result ERV. The display image generation unit 23 also generates a display image HGV as shown in FIG. 14 as a display image including the endoscopic image EGV, the endoscopic image EGS, and the visual information VJV. In this specific example, it is desirable that the display size of the endoscopic image EGS be changed according to the display size of the display image HGV. FIG. 14 is a diagram showing an example of a display image generated using the endoscopic image of FIG.

[0067] The display image HGV of Figure 14 displays visual information VJV corresponding to the lesion TUS contained in both the endoscopic image EGV and the endoscopic image EGS. Furthermore, in the display image HGV of Figure 14, the area corresponding to the visual information VJV in the endoscopic image EGV is colored in a color different from the color of the biological tissue. Furthermore, according to this specific example, as long as the lesion TUS is contained in the image, even if an endoscopic image HGV such as that shown in Figure 15 is input, the visual information VJV corresponding to the position of the lesion TUS is displayed. In other words, when an endoscopic image EGS containing a lesion TUS is input, the display image generating unit 23 can cause the display position of the visual information VJV in the endoscopic image EGV to track the position of the lesion TUS in the endoscopic image EGV in accordance with the position of the lesion TUS in the endoscopic image EGV. Figure 15 is a diagram showing another example of a display image generated using the endoscopic image of Figure 13.

[0068] According to the processing described above, the display image generating unit 23 can generate a display image including a first endoscopic image including a lesion, a second endoscopic image including the lesion and acquired after the first endoscopic image, and visual information superimposed on the second endoscopic image at a position corresponding to the lesion. Furthermore, according to the processing described above, the surgeon can intuitively grasp the infiltration state of the lesion TUS included in the endoscopic image EGV and the endoscopic image EGS by checking the visual information VJV of the endoscopic image EGV.

[0069] According to this specific example, for example, if the endoscopic image EGS contains multiple lesions, processing by the lesion detection unit 21 and the infiltration state estimation unit 22 can be performed during the period from when the endoscopic image EGS is input until detection of each of the multiple lesions ceases.

[0070] According to this specific example, the lesion detection unit 21 may perform a discrimination process to discriminate lesions contained in an endoscopic image. The discrimination process may use, for example, a machine learning model trained using training data that associates images containing lesions with information that can identify whether the lesions are neoplastic or non-neoplastic. Furthermore, according to this specific example, the display image generation unit 23 may generate a display image that includes information indicating the processing results of the discrimination process performed by the lesion detection unit 21. According to this process, the display image generation unit 23 can generate discrimination information DJW that includes a character string indicating whether the lesion TUS is neoplastic or non-neoplastic.

[0071] According to this specific example, the lesion detection unit 21 may perform a measurement process to measure the size of a lesion included in an endoscopic image. The measurement process may use, for example, a machine learning model trained using training data that associates an image including a lesion with information capable of identifying the size of the lesion. Furthermore, according to this specific example, the display image generation unit 23 may generate a display image including information indicating the results of the measurement process performed by the lesion detection unit 21. According to this process, the display image generation unit 23 can generate size information SJW including a character string indicating the size of the lesion TUS.

[0072] The display image generating unit 23 can generate a display image HGW as shown in Fig. 16 by adding, for example, discrimination information DJW and size information SJW to the display image HGV of Fig. 14. Fig. 16 is a diagram showing another example of a display image generated using the endoscopic image of Fig. 13.

[0073] The display image generating unit 23 may generate a display image by adding either the discrimination information DJW or the size information SJW to the display image HGV of FIG. 14, for example.

[0074] [Processing Flow] Next, the flow of processing performed in the endoscopic examination support device will be described. Fig. 17 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the present disclosure. Note that the endoscopic examination support device 1 repeatedly performs the processing shown in Fig. 17 during the period from the start to the end of the endoscopic examination.

[0075] First, the endoscopic examination support device 1 detects a lesion from an endoscopic image (step S11).

[0076] Next, the endoscopic examination support device 1 estimates the infiltration state of the lesion detected by the processing of step S11 (step S12).

[0077] Next, the endoscopic examination support device 1 generates visual information indicating the infiltration state of the lesion estimated by the processing of step S12 (step S13).

[0078] Next, the endoscopic examination support device 1 generates a display image including the visual information generated by the processing of step S13 and outputs the display image to the display device 2 (step S14). In step S14, the endoscopic examination support device 1 may generate a display image in which the visual information is superimposed at a position corresponding to the lesion in the endoscopic image. Alternatively, in step S14, the endoscopic examination support device 1 may generate an infiltration state image combining the lesion and the visual information, and generate a display image including the infiltration state image.

[0079] As described above, according to this embodiment, visual information indicating the infiltration state of a lesion included in an endoscopic image obtained during an endoscopic examination can be generated, and a display image including the generated visual information can be displayed. Therefore, according to this embodiment, the burden imposed on the surgeon who treats a lesion discovered during an endoscopic examination can be reduced.

[0080] Second Embodiment FIG. 18 is a block diagram showing another example of the functional configuration of the endoscopic examination support device according to the present disclosure.

[0081] The endoscopic examination support device 500 according to this embodiment has the same hardware configuration as the endoscopic examination support device 1. The endoscopic examination support device 500 also has a lesion detection unit 501, an estimation unit 502, a visual information generation unit 503, and a display image generation unit 504.

[0082] The lesion detection means 501 can be realized, for example, by using the functions of the lesion detection unit 21 of the first embodiment. The estimation means 502 can be realized, for example, by using the functions of the infiltration state estimation unit 22 of the first embodiment. The visual information generation means 503 can be realized, for example, by using the functions of the display image generation unit 23 of the first embodiment. The display image generation means 504 can be realized, for example, by using the functions of the display image generation unit 23 of the first embodiment.

[0083] FIG. 19 is a flowchart showing another example of processing performed in the endoscopic examination support device according to the present disclosure.

[0084] The lesion detection means 501 detects at least one lesion contained in an endoscopic image obtained during an endoscopic examination (step S51).

[0085] The estimation means 502 estimates the infiltration state of the lesion detected from the endoscopic image (step S52).

[0086] The visual information generating means 503 generates visual information indicating the estimated result of the infiltration state of the lesion (step S53).

[0087] The display image generating means 504 generates a display image including the endoscopic image and visual information (step S54).

[0088] According to this embodiment, it is possible to reduce the burden imposed on the surgeon who treats a lesion discovered during an endoscopic examination.

[0089] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0090] (Supplementary Note 1) An endoscopic examination support device comprising: a lesion detection means for detecting at least one lesion included in an endoscopic image obtained during an endoscopic examination; an estimation means for estimating the infiltration state of the lesion detected from the endoscopic image; a visual information generation means for generating visual information indicating the estimation result of the infiltration state of the lesion; and a display image generation means for generating a display image including the endoscopic image and the visual information.

[0091] (Supplementary Note 2) The endoscopic examination support device of Supplementary Note 1, wherein the display image generating means generates, as the display image, an image including the endoscopic image and the visual information superimposed at a position on the endoscopic image corresponding to the lesion.

[0092] (Appendix 3) In the endoscopic examination support device of Appendix 1, when the size of the lesion in the endoscopic image is less than a predetermined size, the display image generation means generates an image including the endoscopic image, an enlarged image of an area including the lesion, and the visual information superimposed on a position on the enlarged image corresponding to the lesion, as the display image.

[0093] (Appendix 4) An endoscopic examination support device according to Appendix 1, further comprising an infiltration state image generating means for generating an infiltration state image by combining the lesion and the visual information, wherein the display image generating means generates an image including the endoscopic image and the infiltration state image as the display image.

[0094] (Appendix 5) An endoscopic examination support device according to Appendix 4, wherein the estimation means, when multiple lesions are detected from the endoscopic image, estimates the infiltration state of each of the multiple lesions; the visual information generation means generates multiple visual information indicating the estimated results of the infiltration state of each of the multiple lesions; the infiltration state image generation means generates multiple infiltration state images by combining the multiple lesions and the multiple visual information; and the display image generation means generates the display image while sequentially switching between the multiple infiltration state images.

[0095] (Appendix 6) An endoscopic examination support device according to Appendix 4, wherein the estimation means, when multiple lesions are detected from the endoscopic image, estimates the infiltration state of each of the multiple lesions; the visual information generation means generates multiple pieces of visual information indicating the estimated results of the infiltration state of each of the multiple lesions; the infiltration state image generation means generates multiple infiltration state images by combining the multiple lesions and the multiple pieces of visual information; and the display image generation means generates an image including the endoscopic image and the multiple infiltration state images as the display image.

[0096] (Appendix 7) The display image generating means of the endoscopic examination support device of Appendix 1 generates, as the display image, an image including a first endoscopic image including the lesion, a second endoscopic image including the lesion and acquired later than the first endoscopic image, and the visual information superimposed at a position corresponding to the lesion on the second endoscopic image.

[0097] (Supplementary Note 8) The endoscopic examination support device of Supplementary Note 1, wherein the estimation means estimates the infiltration distance and infiltration direction of the lesion for each pixel constituting the lesion in the endoscopic image by inputting the endoscopic image and position information indicating the position of the lesion in the endoscopic image into a machine learning model.

[0098] (Supplementary Note 9) An endoscopic examination support method that detects at least one lesion included in an endoscopic image obtained during an endoscopic examination, estimates the infiltration state of the lesion detected from the endoscopic image, generates visual information indicating the estimated result of the infiltration state of the lesion, and generates a display image including the endoscopic image and the visual information.

[0099] (Appendix 10) A recording medium having recorded thereon a program that causes a computer to execute the processes of: detecting at least one lesion included in an endoscopic image obtained during an endoscopic examination; estimating the infiltration state of the lesion detected from the endoscopic image; generating visual information indicating the estimation result of the infiltration state of the lesion; and generating a display image including the endoscopic image and the visual information.

[0100] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0101] REFERENCE SIGNS LIST 1 Endoscopy support device 2 Display device 3 Endoscope 11 Processor 21 Lesion detection unit 22 Invasion state estimation unit 23 Display image generation unit 100 Endoscopy system

Claims

1. An endoscopic examination support device having: a lesion detection means for detecting at least one lesion contained in an endoscopic image obtained during an endoscopic examination; an estimation means for estimating the infiltration state of the lesion detected from the endoscopic image; a visual information generation means for generating visual information indicating the estimation result of the infiltration state of the lesion; and a display image generation means for generating a display image including the endoscopic image and the visual information.

2. An endoscopic examination support device as described in claim 1, wherein the display image generating means generates, as the display image, an image including the endoscopic image and the visual information superimposed at a position corresponding to the lesion in the endoscopic image.

3. An endoscopic examination support device as described in claim 1, wherein, when the size of the lesion in the endoscopic image is less than a predetermined size, the display image generating means generates an image including the endoscopic image, an enlarged image of an area including the lesion, and the visual information superimposed on a position in the enlarged image corresponding to the lesion.

4. An endoscopic examination support device as described in claim 1, further comprising an infiltration state image generating means for generating an infiltration state image by combining the lesion and the visual information, wherein the display image generating means generates an image including the endoscopic image and the infiltration state image as the display image.

5. The endoscopic examination support device of claim 4, wherein the estimation means estimates the infiltration state of each of the multiple lesions when multiple lesions are detected from the endoscopic image, the visual information generation means generates multiple visual information indicating the estimated results of the infiltration state of each of the multiple lesions, the infiltration state image generation means generates multiple infiltration state images by combining the multiple lesions and the multiple visual information, and the display image generation means generates the display image while sequentially switching between the multiple infiltration state images.

6. The endoscopic examination support device of claim 4, wherein the estimation means estimates the infiltration state of each of the multiple lesions when multiple lesions are detected from the endoscopic image, the visual information generation means generates multiple visual information indicating the estimated results of the infiltration state of each of the multiple lesions, the infiltration state image generation means generates multiple infiltration state images by combining the multiple lesions and the multiple visual information, and the display image generation means generates an image including the endoscopic image and the multiple infiltration state images as the display image.

7. An endoscopic examination support device as described in claim 1, wherein the display image generating means generates, as the display image, an image including a first endoscopic image including the lesion, a second endoscopic image including the lesion and acquired after the first endoscopic image, and the visual information superimposed on a position corresponding to the lesion in the second endoscopic image.

8. The endoscopic examination support device of claim 1, wherein the estimation means estimates the infiltration distance and infiltration direction of the lesion for each pixel constituting the lesion in the endoscopic image by inputting the endoscopic image and positional information indicating the position of the lesion in the endoscopic image into a machine learning model.

9. An endoscopic examination support method comprising: detecting at least one lesion contained in an endoscopic image obtained during an endoscopic examination; estimating an infiltration state of the lesion detected from the endoscopic image; generating visual information indicating the estimated result of the infiltration state of the lesion; and generating a display image including the endoscopic image and the visual information.

10. A recording medium having recorded thereon a program that causes a computer to execute the processes of: detecting at least one lesion contained in an endoscopic image obtained during an endoscopic examination; estimating the infiltration state of the lesion detected from the endoscopic image; generating visual information indicating the estimated result of the infiltration state of the lesion; and generating a display image including the endoscopic image and the visual information.

Images