Assistance device, method for actuating assistance device, program for actuating assistance device, medical system, and learning device

Abstract

This assistance device comprises: a thermally-denatured area extraction unit that extracts a thermally-denatured area from a fluorescence image obtained by irradiating biological tissue with excitation light and imaging fluorescent light; a blood vessel area extraction unit that extracts a blood vessel area from a narrow-band light observation image imaged by irradiating the biological tissue with narrow-band light having a wavelength that is determined in accordance with the absorption ratio of hemoglobin; and an output unit that outputs information corresponding to the distance between the thermally-denatured area and the blood vessel area. Accordingly, provided is an assistance device that enables easy recognition of the distance between a thermally-treated area and a blood vessel.

Description

ASSISTANCE DEVICE, ASSISTANCE DEVICE OPERATION METHOD, ASSISTANCE DEVICE OPERATION PROGRAM, MEDICAL SYSTEM, AND LEARNING DEVICE

[0001] The present invention relates to an assistance device, an operation method for an assistance device, an operation program for an assistance device, a medical system, and a learning device.

[0002] In recent years, minimally invasive treatments using endoscopes, laparoscopes, etc. have become widely used in the medical field. For example, endoscopic submucosal dissection (ESD) is a widely used minimally invasive treatment using endoscopes, laparoscopes, etc.

[0003] In ESD, an energy device such as a high-frequency knife is used to perform thermal treatment such as ablation and coagulation of diseased tissue.

[0004] Furthermore, a technique for estimating the position of blood vessels in surgery using an endoscope is known (see, for example, Patent Document 1).

[0005] JP 2012-130506 A

[0006] However, in reality, the area where the heat treatment has been applied by the energy device is confirmed by the surgeon by visual inspection, touch, intuition, etc. Therefore, there has been a demand for a technology that enables the surgeon to recognize the distance between the area where the heat treatment has been applied and the blood vessel.

[0007] The present invention has been made in consideration of the above, and aims to provide an assistance device, an operation method for the assistance device, an operation program for the assistance device, a medical system, and a learning device that can easily recognize the distance between the area where thermal treatment has been performed and the blood vessel.

[0008] In order to solve the above-mentioned problems and achieve the object, an assistance device according to one aspect of the present invention comprises a thermally denatured region extraction unit that extracts a thermally denatured region from a fluorescence image obtained by irradiating biological tissue with excitation light and capturing the fluorescence, a vascular region extraction unit that extracts a vascular region from a narrowband light observation image obtained by irradiating biological tissue with narrowband light of a wavelength determined according to the absorption rate of hemoglobin, and an output unit that outputs information according to the distance between the thermally denatured region and the vascular region.

[0009] In addition, in the assistance device according to one aspect of the present invention, the excitation light has a wavelength that excites a substance contained in the thermally denatured region.

[0010] In addition, in the support device according to one aspect of the present invention, the narrowband light is amber light.

[0011] In addition, in the support device according to one aspect of the present invention, the narrowband light is blue-violet light.

[0012] In the support device according to one aspect of the present invention, the narrowband light is green light.

[0013] The assistance device according to an aspect of the present invention includes a calculation unit that calculates the distance between the thermally denatured region and the blood vessel region.

[0014] The support device according to an aspect of the present invention further includes an adjustment unit that aligns the fluorescence image and the narrow-band light observation image.

[0015] In addition, in an assistance device according to one aspect of the present invention, the adjustment unit extracts feature information from a first reference image captured by irradiating reference light, which is narrowband light of a different wavelength from the narrowband light, under the imaging conditions under which the fluorescence image was captured, and a second reference image captured by irradiating the reference light under the imaging conditions under which the narrowband light observation image was captured, and aligns the fluorescence image and the narrowband light observation image based on the feature information.

[0016] In addition, in an assistance device according to one aspect of the present invention, the vascular region extraction unit extracts a deep vascular region from a first narrowband light observation image captured by irradiating amber light as the narrowband light, extracts a middle vascular region from a second narrowband light observation image captured by irradiating green light as the narrowband light, and extracts a superficial vascular region from a third narrowband light observation image captured by irradiating blue-violet light as the narrowband light.

[0017] In addition, in an assistance device according to one aspect of the present invention, the output unit outputs information corresponding to two or more distances from the distance between the thermally denatured region and the deep vascular region, the distance between the thermally denatured region and the middle vascular region, or the distance between the thermally denatured region and the superficial vascular region.

[0018] In addition, in an assistance device according to one aspect of the present invention, the output unit outputs information corresponding to one distance selected from the distance between the thermally denatured region and the deep vascular region, the distance between the thermally denatured region and the middle vascular region, or the distance between the thermally denatured region and the superficial vascular region.

[0019] In addition, in an assistance device according to one aspect of the present invention, the thermally denatured region extraction unit extracts pixels deemed to be the thermally denatured region in the fluorescence image, the vascular region extraction unit extracts pixels deemed to be the vascular region in the narrowband light observation image, and the calculation unit calculates the shortest distance between the pixels deemed to be the thermally denatured region and the pixels deemed to be the vascular region.

[0020] In addition, in the support device according to one aspect of the present invention, the calculation unit calculates the depth of the thermally denatured region from the fluorescence image, extracts the depth of the vascular region from the narrowband light observation image, and calculates the distance in the depth direction between the thermally denatured region and the vascular region.

[0021] In addition, in the assistance device according to one aspect of the present invention, the output unit superimposes information according to the distance between the thermally denatured region and the blood vessel region on the display image.

[0022] In the assistance device according to one aspect of the present invention, the output unit outputs a display control signal that causes a display device to display the distance between the thermally denatured region and the blood vessel region.

[0023] In the assistance device according to one aspect of the present invention, the output unit outputs information notifying that the distance between the thermally denatured region and the blood vessel region is equal to or less than a threshold value.

[0024] Furthermore, a method of operating an assistance device according to one aspect of the present invention includes a thermally denatured region extraction unit extracting a thermally denatured region from a fluorescence image obtained by irradiating excitation light onto biological tissue and capturing fluorescence, a vascular region extraction unit extracting a vascular region from a narrowband light observation image obtained by irradiating biological tissue with narrowband light having a wavelength determined according to the absorption rate of hemoglobin, and an output unit outputting information according to the distance between the thermally denatured region and the vascular region.

[0025] In addition, an operating program for an assistance device according to one aspect of the present invention causes the assistance device to extract a thermally denatured region from a fluorescence image obtained by irradiating excitation light onto biological tissue and capturing the fluorescence, extract a vascular region from a narrowband light observation image obtained by irradiating biological tissue with narrowband light of a wavelength determined according to the absorption rate of hemoglobin, and output information according to the distance between the thermally denatured region and the vascular region.

[0026] Furthermore, a medical system according to one aspect of the present invention includes a light source device that irradiates biological tissue with excitation light and also irradiates the biological tissue with narrowband light of a wavelength determined according to the absorption rate of hemoglobin; an endoscope that generates a first imaging signal obtained by irradiating the biological tissue with the excitation light and capturing fluorescence, and a second imaging signal obtained by irradiating the biological tissue with the narrowband light; and an image processing device that generates a fluorescence image from the first imaging signal and a narrowband light observation image from the second imaging signal, wherein the image processing device has a thermally denatured region extraction unit that extracts a thermally denatured region from the fluorescence image, a vascular region extraction unit that extracts a vascular region from the narrowband light observation image, and an output unit that outputs information according to the distance between the thermally denatured region and the vascular region.

[0027] Furthermore, a learning device according to one aspect of the present invention has a learning unit that generates a trained model by machine learning using training data in which input data are a fluorescence image obtained by irradiating biological tissue with excitation light and capturing fluorescence, and a narrowband light observation image obtained by irradiating biological tissue with narrowband light having a wavelength determined according to the absorption rate of hemoglobin, and output data is information corresponding to the distance between a thermally denatured region extracted from the fluorescence image and a blood vessel region extracted from the narrowband light observation image.

[0028] According to the present invention, it is possible to realize an assistance device, an operation method for an assistance device, an operation program for an assistance device, a medical system, and a learning device that can easily recognize the distance between an area that has been subjected to thermal treatment and a blood vessel.

[0029] FIG. 1 is a diagram schematically illustrating the overall configuration of an endoscope system according to an embodiment. FIG. 2 is a block diagram illustrating the functional configuration of a main part of the endoscope system according to an embodiment. FIG. 3 is a diagram illustrating an example of biological tissue of a subject. FIG. 4 is a flowchart illustrating an outline of processing executed by a control device. FIG. 5 is a diagram illustrating an example of a narrowband light observation image. FIG. 6 is a diagram illustrating an example of a fluorescence image. FIG. 7 is a diagram illustrating an image in which a narrowband light observation image and a fluorescence image are superimposed. FIG. 8 is a diagram illustrating an example of biological tissue of a subject.

[0030] Hereinafter, an endoscopic system having an endoscope with a flexible insertion section will be described as a mode for carrying out the present disclosure (hereinafter referred to as an "embodiment"). However, the present disclosure is not limited to this and can also be applied to, for example, rigid endoscopes and surgical robots. Furthermore, the present disclosure is not limited to this embodiment. Furthermore, in the drawings, identical parts are denoted by the same reference numerals. Furthermore, it should be noted that the drawings are schematic, and the relationship between the thickness and width of each component, the ratio of each component, and the like may differ from reality. Furthermore, the drawings also include parts with different dimensions and ratios.

[0031] [Configuration of Endoscope System] Fig. 1 is a diagram showing a schematic diagram of the overall configuration of an endoscope system according to one embodiment. The endoscope system 1 shown in Fig. 1 captures images of the inside of a subject's body by inserting an insertion portion of an endoscope into a body cavity or lumen of the subject, such as a patient, and displays a display image based on the captured image signal on a display device. The endoscope system 1 includes an endoscope 2, a light source device 3, a control device 4, and a display device 5.

[0032] [Configuration of Endoscope] First, the configuration of the endoscope 2 will be described. The endoscope 2 generates an imaging signal (RAW data) obtained by imaging the inside of the subject's body and outputs the generated imaging signal to the control device 4. Specifically, the endoscope 2 generates a first imaging signal obtained by irradiating excitation light and imaging fluorescence, and a second imaging signal obtained by irradiating narrowband light. The endoscope 2 includes an insertion section 21, an operation section 22, and a universal cord 23.

[0033] The insertion section 21 is inserted into the subject and has a flexible, elongated shape. The insertion section 21 has a distal end portion 24 incorporating an imaging element (described later), a freely bendable bending section 25 composed of a plurality of bending pieces, and a flexible, elongated flexible tube portion 26 connected to the proximal end side of the bending section 25.

[0034] The tip portion 24 is made of glass fiber or the like. The tip portion 24 forms a light guide path for the illumination light supplied from the control device 4 via the universal cord 23 and the operation unit 22, and also generates an image pickup signal by capturing an image of the return light of the illumination light and outputs the image pickup signal to the control device 4.

[0035] The operation unit 22 has a bending knob 221 for bending the bending portion 25 in the up-down and left-right directions, a treatment tool insertion portion 222 for inserting a treatment tool, and a plurality of switches 223 which are an operation input portion for inputting, in addition to the control device 4, operation instruction signals for peripheral devices such as an air supply means, a water supply means, and a gas supply means, a pre-freeze signal for instructing the endoscope system 1 to take a still image, or a switching signal for switching the observation mode of the endoscope system 1. The treatment tool inserted from the treatment tool insertion portion 222 passes through a treatment tool channel (not shown) in the tip portion 24 and emerges from an opening (not shown).

[0036] The universal cord 23 incorporates at least a light guide and a light collecting cable that is a collection of one or more cables. The collecting cable is a signal line for transmitting and receiving signals between the endoscope 2 and the control device 4, and includes a signal line for transmitting and receiving an imaging signal (RAW data) and a signal line for transmitting and receiving timing signals (synchronization signals and clock signals) for driving an imaging element (described later). The universal cord 23 has a connector portion 27 that is detachable from the control device 4, and a connector portion 28 from which a coiled coil cable 27a extends, the connector portion 28 being detachable from the control device 4 at the extending end of the coil cable 27a.

[0037] [Configuration of Light Source Device] Next, the configuration of the light source device will be described. The light source device 3 irradiates biological tissue with excitation light and narrowband light with a wavelength determined according to the absorption rate of hemoglobin. The light source device 3 is connected to one end of the light guide of the endoscope 2, and supplies illumination light to the end of the light guide to irradiate the inside of the subject under the control of the control device 4. The light source device 3 is realized using one or more light sources, such as a light-emitting diode (LED) light source, a xenon lamp, or a semiconductor laser element such as a laser diode (LD), a processor that is a processing device having hardware such as a field programmable gate array (FPGA) or a central processing unit (CPU), and a memory that is a temporary storage area used by the processor. The light source device 3 and the control device 4 may be configured to communicate individually as shown in FIG. 1, or may be integrated.

[0038] [Configuration of the control device] Next, the configuration of the control device 4 will be described. The control device 4 controls each part of the endoscope system 1. The control device 4 supplies illumination light for the endoscope 2 to irradiate the subject. The control device 4 also performs various image processing on the imaging signal input from the endoscope 2 and outputs the signal to the display device 5.

[0039] [Configuration of Display Device] Next, a description will be given of the configuration of the display device 5. Under the control of the control device 4, the display device 5 displays an image based on a video signal input from the control device 4. The display device 5 is realized using a display panel such as an organic EL (Electro Luminescence) panel or a liquid crystal panel.

[0040] [Functional Configuration of Main Parts of Endoscope System] Next, a description will be given of the functional configuration of the main parts of the above-described endoscope system 1. Fig. 2 is a block diagram showing the functional configuration of the main parts of the endoscope system 1.

[0041] [Configuration of Endoscope] First, we will explain the configuration of the endoscope 2. The endoscope 2 includes an illumination optical system 201, an imaging optical system 202, a cut filter 203, an imaging element 204, an A / D conversion unit 205, a P / S conversion unit 206, an imaging and recording unit 207, and an imaging control unit 208. Note that the illumination optical system 201, the imaging optical system 202, the cut filter 203, the imaging element 204, the A / D conversion unit 205, the P / S conversion unit 206, the imaging and recording unit 207, and the imaging control unit 208 are each disposed within the tip portion 24.

[0042] The illumination optical system 201 irradiates an object (living tissue) with illumination light supplied from a light guide 231 formed of an optical fiber or the like. The illumination optical system 201 is realized using one or more lenses or the like.

[0043] The imaging optical system 202 focuses light such as reflected light from the subject, returned light from the subject, and fluorescent light emitted by the subject, thereby forming an object image (light rays) on the light receiving surface of the image sensor 204. The imaging optical system 202 is realized using one or more lenses, etc.

[0044] The cut filter 203 is disposed on an optical axis O1 between the imaging optical system 202 and the imaging element 204. The cut filter 203 blocks light in a wavelength band that is excitation light supplied from the control device 4 (described later) and that is reflected or returned from the subject, and transmits light in a wavelength band that is longer than the wavelength band of the excitation light. The cut filter 203 also transmits light in a wavelength band that is narrowband light supplied from the control device 4 (described later) and that is reflected or returned from the subject.

[0045] Under the control of the imaging control unit 208, the imaging element 204 receives the subject image (light rays) formed by the imaging optical system 202 and transmitted through the cut filter 203, performs photoelectric conversion, generates an imaging signal (RAW data), and outputs the image signal to the A / D conversion unit 205. The imaging element 204 is realized using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor in which one of color filters constituting a Bayer array (RGGB) is arranged on each of a plurality of pixels arranged in a two-dimensional matrix.

[0046] Under the control of the imaging control unit 208, the A / D conversion unit 205 performs A / D conversion processing on the analog imaging signal input from the imaging element 204 and outputs the result to the P / S conversion unit 206. The A / D conversion unit 205 is realized using an A / D conversion circuit or the like.

[0047] Under the control of the imaging control unit 208, the P / S conversion unit 206 performs parallel / serial conversion on the digital imaging signal input from the A / D conversion unit 205, and outputs the parallel / serial converted imaging signal to the control device 4 via the first transmission cable 232. The P / S conversion unit 206 is realized using a P / S conversion circuit or the like. Note that in the first embodiment, instead of the P / S conversion unit 206, an E / O conversion unit that converts the imaging signal into an optical signal may be provided so that the imaging signal is output to the control device 4 via the optical signal, or the imaging signal may be transmitted to the control device 4 via wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0048] The image capturing and recording unit 207 records various information related to the endoscope 2 (for example, pixel information of the image capturing element 204 and characteristics of the cut filter 203). The image capturing and recording unit 207 also records various setting data and control parameters transmitted from the control device 4 via the second transmission cable 233. The image capturing and recording unit 207 is configured using a non-volatile memory or a volatile memory.

[0049] The imaging control unit 208 controls the operations of the imaging element 204, the A / D conversion unit 205, and the P / S conversion unit 206 based on setting data received from the control device 4 via the second transmission cable 233. The imaging control unit 208 is realized using a TG (Timing Generator), a processor which is a processing device having hardware such as a CPU, and a memory which is a temporary storage area used by the processor.

[0050] [Configuration of Light Source Device] Next, a description will be given of the configuration of the light source device 3. The light source device 3 includes a condenser lens 30, a first light source unit 31, a second light source unit 32, and a light source control unit 33.

[0051] The condenser lens 30 condenses the light emitted by each of the first light source unit 31 and the second light source unit 32, and outputs the condensed light to the light guide 231. The condenser lens 30 is configured using one or more lenses.

[0052] The first light source unit 31 emits narrowband light under the control of the light source control unit 33, thereby supplying the narrowband light to the light guide 231. The narrowband light is, for example, amber light having a peak wavelength in a wavelength band of 580 nm to 620 nm, but may also be blue-violet light having a peak wavelength in a wavelength band of 390 nm to 430 nm, or green light having a peak wavelength in a wavelength band of 500 nm to 550 nm, or may include light of two or more wavelength bands. The first light source unit 31 is configured using a collimating lens, an LED (Light Emitting Diode) or an LD (Laser Diode), a driver, etc.

[0053] The second light source unit 32 emits excitation light having a predetermined wavelength band under the control of the light source control unit 33, thereby supplying narrowband light as illumination light to the light guide 231. Here, the excitation light has a wavelength that excites substances such as advanced glycation end products (AGEs) contained in the thermally denatured region, and has a wavelength band of, for example, 400 nm to 430 nm (center wavelength: 415 nm). The thermally denatured region is a region of living tissue that has been thermally denatured by thermal treatment using an energy device such as a high-frequency knife. The excitation light emitted by the second light source unit 32 is blocked by the cut filter 203, while the fluorescence (wavelength 540 nm) generated from the AGEs passes through the cut filter 203, allowing a fluorescent image to be captured. The second light source unit 32 is realized using a collimator lens, a semiconductor laser such as a violet LD (Laser Diode), a driver, and the like.

[0054] The light source control unit 33 is configured using a processor, which is a processing device having hardware such as an FPGA (Field-Programmable Gate Array) or a CPU (Central Processing Unit), and a memory, which is a temporary storage area used by the processor. The light source control unit 33 controls the light emission timing, light emission intensity, light emission time, etc. of each of the first light source unit 31 and the second light source unit 32 based on control data input from the control unit 405.

[0055] [Configuration of Control Device] Next, a description will be given of the configuration of the control device 4. The control device 4 includes an S / P conversion unit 401, an image processing unit 402, an input unit 403, a recording unit 404, and a control unit 405.

[0056] Under the control of the control unit 405, the S / P conversion unit 401 performs serial / parallel conversion on the imaging signal received from the endoscope 2 via the first transmission cable 232 and outputs the signal to the image processing unit 402. Note that if the endoscope 2 outputs the imaging signal as an optical signal, an O / E conversion unit that converts the optical signal into an electrical signal may be provided instead of the S / P conversion unit 401. Also, if the endoscope 2 transmits the imaging signal via wireless communication, a communication module capable of receiving wireless signals may be provided instead of the S / P conversion unit 401.

[0057] The image processing unit 402 is realized using a processor having hardware such as a CPU, a GPU (Graphics Processing Unit), or an FPGA, and a memory serving as a temporary storage area used by the processor. Under the control of the control unit 405, the image processing unit 402 performs predetermined image processing on the imaging signal input from the S / P conversion unit 401 and outputs the processed image to the display device 5. In one embodiment, the image processing unit 402 functions as both an assistance device and an image processing device. The image processing unit 402 generates a fluorescence image from the first imaging signal and generates a narrowband light observation image from the second imaging signal. The image processing unit 402 includes an image generation unit 402a, a thermally denatured region extraction unit 402b, a vascular region extraction unit 402c, an adjustment unit 402d, a calculation unit 402e, and an output unit 402f.

[0058] The image generating unit 402a generates a fluorescence image from a first imaging signal obtained by capturing fluorescence under irradiation with excitation light from the second light source unit 32. The image generating unit 402a also generates a narrowband light observation image from a second imaging signal obtained by capturing fluorescence under irradiation with narrowband light from the first light source unit 31.

[0059] The thermally degenerated region extraction unit 402b extracts a thermally degenerated region from a fluorescence image obtained by irradiating excitation light onto living tissue and capturing fluorescence. The thermally degenerated region extraction unit 402b extracts, as a thermally degenerated region, a region whose brightness is equal to or exceeds a threshold value due to fluorescence generated by AGEs from the fluorescence image obtained by irradiating excitation light onto living tissue.

[0060] The vascular region extraction unit 402c extracts a vascular region from a narrowband light observation image captured by irradiating biological tissue with narrowband light having a wavelength determined according to the absorption rate of hemoglobin. For example, amber light has a higher absorption rate by hemoglobin than red light, has a longer wavelength than green light, and reaches deeper, making it easier to observe deep blood vessels than observation with normal light. In the narrowband light observation image captured by irradiating amber light, the vascular region extraction unit 402c extracts, as deep blood vessel regions, regions where the brightness of the amber light is below a threshold due to absorption by hemoglobin.

[0061] The adjustment unit 402d aligns the fluorescence image and the narrowband light observation image. The adjustment unit 402d aligns the fluorescence image and the narrowband light observation image so that feature points (characteristic points of the image, such as the edge of a lesion or a bleeding point) in the fluorescence image correspond to the positions of feature points in the narrowband light observation image. The adjustment unit 402d may also extract feature information from a first reference image captured by irradiating the image with reference light, which is narrowband light with a wavelength different from the narrowband light, under the imaging conditions used to capture the fluorescence image, and a second reference image captured by irradiating the image with reference light under the imaging conditions used to capture the narrowband light observation image, and align the fluorescence image and the narrowband light observation image based on the feature information. The wavelength of the reference light is not particularly limited. The feature information may be, for example, position information of the feature points.

[0062] The calculation unit 402e calculates the distance between the thermally denatured region and the blood vessel region.

[0063] The output unit 402f outputs information according to the distance between the thermally denatured region and the blood vessel region. The output unit 402f outputs, for example, a display control signal superimposed on a display image to be displayed on the display device 5, indicating the distance between the thermally denatured region and the blood vessel region. The output unit 402f may also output information notifying that the distance between the thermally denatured region and the blood vessel region is equal to or less than a threshold. For example, when the distance between the thermally denatured region and the blood vessel region is equal to or less than the threshold, the output unit 402f may output a display control signal notifying by superimposing a warning using a color or mark on a display image to be displayed on the display device 5.

[0064] The input unit 403 receives inputs for various operations related to the endoscope system 1 and outputs the received operations to the control unit 405. The input unit 403 is configured using a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, and the like.

[0065] The recording unit 404 is realized using a recording medium such as a volatile memory, a non-volatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), a memory card, etc. The recording unit 404 records data including various parameters and the like necessary for the operation of the endoscope system 1. The recording unit 404 also has a program recording unit 404a that records various programs for operating the endoscope system 1.

[0066] The control unit 405 is realized using a processor having hardware such as an FPGA or a CPU, and a memory that is a temporary storage area used by the processor. The control unit 405 comprehensively controls each unit that constitutes the endoscope system 1.

[0067] [Processing of Control Device] Next, the processing executed by the control device 4 will be described. Fig. 3 is a diagram showing an example of biological tissue of a subject. As shown in Fig. 3, the subject has a vascular region A1. When a thermal treatment such as lesion resection is performed on the surface S of the biological tissue of the subject using an energy device, the resection surface is denatured by heat, forming a thermally denatured region A2. At this time, the processing executed by the control device 4 to display the distance L1 between the vascular region A1 and the thermally denatured region A2 in the horizontal direction (the direction along the surface S) on the display device 5 will be described.

[0068] 4 is a flowchart showing an outline of the processing executed by the control device. As shown in FIG. 4, first, the image generating unit 402a generates a narrowband light observation image from a second imaging signal obtained by irradiating narrowband light from the first light source unit 31 onto biological tissue and capturing an image (step S1).

[0069] Next, the vascular region extraction unit 402c extracts a vascular region from the narrowband light observation image generated by the image generation unit 402a (step S2). Fig. 5 is a diagram showing an example of a narrowband light observation image. As shown in Fig. 5, the vascular region extraction unit 402c extracts pixels regarded as a hatched vascular region B1 in the narrowband light observation image I1. The vascular region extraction unit 402c extracts pixels whose brightness is equal to or less than a threshold due to absorption by hemoglobin as the vascular region B1 in the narrowband light observation image I1.

[0070] Thereafter, the image generating unit 402a generates a fluorescence image from the first imaging signal obtained by irradiating the living tissue with excitation light from the second light source unit 32 and capturing the fluorescence (step S3).

[0071] Furthermore, the thermally altered region extraction unit 402b extracts a thermally altered region from a fluorescence image obtained by irradiating excitation light onto biological tissue and capturing fluorescence (step S4). Fig. 6 is a diagram showing an example of a fluorescence image. As shown in Fig. 6, the thermally altered region extraction unit 402b extracts pixels that are deemed to be the hatched thermally altered region B2 in the fluorescence image I2. The thermally altered region extraction unit 402b extracts pixels in the fluorescence image I2 whose brightness is equal to or exceeds a threshold due to fluorescence from AGEs as the thermally altered region B2.

[0072] Next, the adjustment unit 402d aligns the narrow-band light observation image I1 with the fluorescence image I2 (step S5). Fig. 7 shows an image in which the narrow-band light observation image and the fluorescence image are superimposed. As shown in Fig. 7, the adjustment unit 402d generates a superimposed image I3 in which the narrow-band light observation image I1 and the fluorescence image I2 are superimposed so that the positions of the feature points in the narrow-band light observation image I1 and the feature points in the fluorescence image I2 overlap.

[0073] Furthermore, the calculation unit 402e calculates the distance between the thermally denatured region B2 and the blood vessel region B1 (step S6). By detecting the distance between the tip of the endoscope 2 and the surface S of the subject's biological tissue using a distance sensor or the like, the calculation unit 402e can calculate the distance L2, which corresponds to the size of one pixel in the superimposed image I3. Then, because the distance L1 between the blood vessel region B1 and the thermally denatured region B2 corresponds to the length of two pixels, the calculation unit 402e calculates the distance L1 using the distance L2. In other words, the calculation unit 402e estimates the actual distance between the subject's thermally denatured region and the blood vessel from the distance L1 between the blood vessel region B1 and the thermally denatured region B2 in the superimposed image I3.

[0074] The output unit 402f then outputs information corresponding to the distance between the thermally denatured region B2 and the blood vessel region B1 (step S7). The output unit 402f outputs, for example, a display control signal that causes the display device 5 to display the distance between the thermally denatured region B2 and the blood vessel region B1.

[0075] According to the endoscopic system 1 described above, information corresponding to the distance between the thermally denatured region B2 and the vascular region B1 is output based on the narrowband light observation image I1 and the fluorescence image I2, so that the surgeon can easily recognize the distance between the region that has been subjected to thermal treatment and the blood vessel.

[0076] [Modification] Fig. 8 is a diagram showing an example of biological tissue of a subject. As shown in Fig. 8, a blood vessel region A11 is present deep within the subject. When a thermal treatment such as lesion resection is performed on the surface S of the biological tissue of the subject using an energy device, the resection surface is denatured by heat, forming a thermally denatured region A12. At this time, the control device 4 may cause the display device 5 to display the depthwise distance L11 between the blood vessel region A11 and the thermally denatured region A12. The depthwise direction refers to the direction perpendicular to the surface S of the biological tissue.

[0077] The calculation unit 402e calculates the depth of the thermally denatured region A12 from the fluorescent image. Note that because there is a correlation between the depth of the thermally denatured region A12 and the brightness of the fluorescent image, the calculation unit 402e can estimate the depth of the thermally denatured region A12 from the brightness of the fluorescent image based on the correlation obtained in advance by measurement.

[0078] The calculation unit 402e also extracts the depth of the vascular region A11 from the narrowband light observation image. The calculation unit 402e extracts a deep vascular region from a first narrowband light observation image captured using amber light as the narrowband light, extracts a middle vascular region from a second narrowband light observation image captured using green light as the narrowband light, and extracts a superficial vascular region from a third narrowband light observation image captured using blue-violet light as the narrowband light. The calculation unit 402e can then estimate the depth of the vascular region A11 from the deep to superficial vascular regions.

[0079] Then, the calculation unit 402e calculates the distance in the depth direction between the thermally denatured region A12 and the blood vessel region A11.

[0080] Furthermore, the output unit 402f outputs information corresponding to the distance L11 between the thermally denatured region A12 and the blood vessel region A11. The output unit 402f outputs, for example, a display control signal that causes the display device 5 to display the depthwise distance L11 between the thermally denatured region A12 and the blood vessel region A11.

[0081] According to the modified example described above, information corresponding to the depthwise distance L11 between the thermally denatured area A12 and the vascular area A11 is output based on the narrowband light observation image and the fluorescence image, so that the surgeon can easily recognize the distance between the area subjected to thermal treatment and the blood vessel.

[0082] The adjuster 402d may extract feature information from a first reference image captured by irradiating the narrowband light with reference light, which is narrowband light with a wavelength different from the narrowband light, under the imaging conditions for capturing the fluorescent image, and a second reference image captured by irradiating the narrowband light with reference light under the imaging conditions for capturing the narrowband light observation image, and align the fluorescent image and the narrowband light observation image based on the feature information. The wavelength of the reference light is not particularly limited. The feature information may be, for example, position information of feature points. By aligning the fluorescent image and the narrowband light observation image using the image captured by irradiating the reference light, the accuracy of alignment can be improved.

[0083] Furthermore, the vascular region extraction unit 402c may extract a deep vascular region from a first narrowband light observation image captured by irradiating amber light as the narrowband light, extract a middle vascular region from a second narrowband light observation image captured by irradiating green light as the narrowband light, and extract a superficial vascular region from a third narrowband light observation image captured by irradiating blue-violet light as the narrowband light.

[0084] At this time, the output unit 402f outputs information corresponding to two or more of the distances between the thermally denatured region and the deep layer vascular region, the distance between the thermally denatured region and the middle layer vascular region, and the distance between the thermally denatured region and the superficial layer vascular region, allowing the surgeon to recognize the distances between the thermally denatured region and the vascular region in two or more selected layers from the deep layer to the superficial layer.

[0085] The output unit 402f may output information corresponding to one distance selected from the distance between the thermally denatured region and the deep vascular region, the distance between the thermally denatured region and the middle vascular region, or the distance between the thermally denatured region and the superficial vascular region, thereby enabling the surgeon to recognize the distance between the thermally denatured region and the vascular region in one selected layer from the deep to superficial layers.

[0086] The control unit 405 may also function as a learning unit of a learning device. The control unit 405 may generate a trained model by machine learning using training data, which takes as input data a fluorescence image obtained by irradiating biological tissue with excitation light and a narrowband light observation image obtained by irradiating biological tissue with narrowband light having a wavelength determined according to the hemoglobin absorption rate, and output data corresponding to the distance between a thermally denatured region extracted from the fluorescence image and a vascular region extracted from the narrowband light observation image. Here, the trained model is composed of a neural network, each layer of which has one or more nodes. The type of machine learning is not particularly limited, but may involve, for example, preparing training data and training data that associate multiple fluorescence images and narrowband light observation images of a subject with the distance between a thermally denatured region and a vascular region calculated from the multiple fluorescence images and narrowband light observation images, and inputting the training data and training data into a computational model based on a multilayer neural network for training. Furthermore, as a machine learning technique, for example, a technique based on a deep neural network (DNN), which is a multilayer neural network such as a convolutional neural network (CNN) or a 3D-CNN, may be used. Furthermore, as a machine learning technique, a technique based on a recurrent neural network (RNN) or long short-term memory units (LSTM), which are an extension of an RNN, may also be used. Note that these functions may be executed by a learning unit of a learning device different from the control device 4.

[0087] Further advantages and modifications will readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.

[0088] REFERENCE SIGNS LIST 1 Endoscope system 2 Endoscope 3 Display device 4 Control device 5 Laser irradiation device 7 Display device 21 Insertion section 22 Operation section 23 Universal cord 24 Tip section 25 Bending section 26 Flexible tube section 27 Connector section 27a Coil cable 28 Connector section 30 Condenser lens 31 First light source section 32 Second light source section 33 Light source control section 201 Illumination optical system 202 Imaging optical system 203 Cut filter 204 Imaging element 205 A / D conversion section 206 P / S conversion section 207 Imaging recording section 208 Imaging control section 221 Bending knob 222 Treatment tool insertion section 223 Switch 231 Light guide 232 First transmission cable 233 Second transmission cable 401 S / P conversion section 402 Image processing section 402a Image generating unit 402b Thermally denatured region extracting unit 402c Blood vessel region extracting unit 402d Adjusting unit 402e Calculating unit 402f Output unit 403 Input unit 404 Recording unit 404a Program recording unit 405 Control unit I1 Narrow band light observation image I2 Fluorescence image I3 Superimposed image

Claims

1. A thermally denatured region is extracted from a fluorescence image obtained by irradiating excitation light onto biological tissue and capturing the fluorescence. extracting a blood vessel region from a narrowband light observation image captured by irradiating biological tissue with narrowband light having a wavelength determined according to the absorption rate of hemoglobin; aligning the fluorescent image with the narrow-band light observation image; Outputting information according to the distance between the thermally denatured region and the blood vessel region An assistance device comprising a processor configured to:

2. The assistance device according to claim 1 , wherein the excitation light has a wavelength that excites a substance contained in the thermally denatured region.

3. The assistive device of claim 1 , wherein the narrow band light is amber light.

4. The support device according to claim 1 , wherein the narrow-band light is blue-violet light.

5. The support device of claim 1 , wherein the narrow-band light is green light.

6. An assistance device as described in claim 1, wherein the processor calculates the distance between the thermally denatured region and the vascular region.

7. The support device described in claim 1, wherein the processor extracts feature information from a first reference image captured by irradiating reference light, which is narrow-band light of a different wavelength from the narrow-band light, under the imaging conditions under which the fluorescent image was captured, and a second reference image captured by irradiating the reference light under the imaging conditions under which the narrow-band light observation image was captured, and aligns the fluorescent image and the narrow-band light observation image based on the feature information.

8. The processor: extracting a deep blood vessel region from a first narrowband light observation image captured by irradiating the subject with amber light as the narrowband light; extracting a middle layer blood vessel region from a second narrowband light observation image captured by irradiating the subject with green light as the narrowband light; 8. The support device according to claim 7, wherein a superficial blood vessel region is extracted from a third narrowband light observation image captured by irradiating blue-violet light as the narrowband light.

9. The assistance device described in Claim 8, wherein the processor outputs information corresponding to two or more of the distances between the thermally denatured region and the deep vascular region, the distance between the thermally denatured region and the middle vascular region, or the distance between the thermally denatured region and the superficial vascular region.

10. The assistance device described in Claim 8, wherein the processor outputs information corresponding to one distance selected from the distance between the thermally denatured region and the deep vascular region, the distance between the thermally denatured region and the middle vascular region, or the distance between the thermally denatured region and the superficial vascular region.

11. The processor, extracting pixels regarded as the thermally denatured region from the fluorescence image; extracting pixels regarded as the blood vessel region from the narrow-band light observation image; The support device according to claim 6 , wherein the shortest distance between the pixel regarded as the thermally denatured region and the pixel regarded as the blood vessel region is calculated.

12. The processor, Calculating the depth of the thermally denatured region from the fluorescent image; extracting a depth of the blood vessel region from the narrow-band light observation image; The assistance device according to claim 6 , wherein the distance in the depth direction between the thermally denatured region and the blood vessel region is calculated.

13. The processor, The assistance device according to claim 1 , wherein information according to the distance between the thermally denatured region and the blood vessel region is superimposed on a displayed image.

14. The processor, The assistance device according to claim 1, further comprising a display control signal for causing a display device to display the distance between the thermally denatured region and the blood vessel region.

15. The processor, The assistance device according to claim 1 , wherein the assistance device outputs information informing that the distance between the thermally denatured region and the blood vessel region is equal to or smaller than a threshold value.

16. A processor comprising: A thermally denatured region is extracted from a fluorescence image obtained by irradiating excitation light onto biological tissue and capturing the fluorescence. extracting a blood vessel region from a narrowband light observation image captured by irradiating biological tissue with narrowband light having a wavelength determined according to the absorption rate of hemoglobin; aligning the fluorescent image with the narrow-band light observation image; A method for operating an assistance device, comprising outputting information according to the distance between the thermally denatured region and the blood vessel region.

17. A thermally denatured region is extracted from a fluorescence image obtained by irradiating excitation light onto biological tissue and capturing the fluorescence. extracting a blood vessel region from a narrowband light observation image captured by irradiating biological tissue with narrowband light having a wavelength determined according to the absorption rate of hemoglobin; aligning the fluorescent image with the narrow-band light observation image; An operating program for an assistance device that causes a processor included in the assistance device to output information according to the distance between the thermally denatured region and the blood vessel region.

18. a light source device that irradiates the biological tissue with excitation light and also irradiates the biological tissue with narrowband light having a wavelength determined according to the absorption rate of hemoglobin; an endoscope that generates a first imaging signal obtained by irradiating living tissue with the excitation light and capturing fluorescence, and a second imaging signal obtained by irradiating living tissue with the narrowband light and capturing fluorescence; a processor configured to generate a fluorescence image from the first imaging signal and a narrowband light observation image from the second imaging signal; Equipped with The processor: extracting a thermally denatured region from the fluorescent image; extracting a blood vessel region from the narrow-band light observation image; aligning the fluorescent image with the narrow-band light observation image; outputting information according to the distance between the thermally denatured region and the blood vessel region; Healthcare system.

19. A learning device including a processor configured to generate a trained model by machine learning using training data that takes as input data a fluorescence image obtained by irradiating biological tissue with excitation light and capturing fluorescence, and a narrow-band light observation image obtained by irradiating biological tissue with narrow-band light having a wavelength determined according to the absorption rate of hemoglobin, aligns the fluorescence image with the narrow-band light observation image, and outputs information corresponding to the distance between a thermally denatured region extracted from the fluorescence image and a blood vessel region extracted from the narrow-band light observation image.