Medical device, endoscope system, control method, control program, and learning device

Abstract

A medical device (9) is provided with a processor (92) for processing a captured image obtained by capturing fluorescent light generated from biological tissue by irradiating the biological tissue with excitation light. A processor (92) performs: a process for extracting, as a thermal-denaturation-deficient region in which thermal denaturation is deficient, a region comprising pixels having a fluorescence intensity equal to or less than a first fluorescence intensity among all pixels in a captured image; and generating a display image in which the region with insufficient thermal denaturation can be distinguished from other regions in all pixels of the captured image.

Description

Technical Field

[0001] The present invention relates to a medical device, an endoscope system, a control method, a control program, and a learning device. Background Art

[0002] Conventionally, there is known a technique for visualizing the thermal denaturation state of living tissue when the living tissue is treated with an energy device or the like (for example, see Patent Documents 1 and 2).

[0003] The techniques described in Patent Documents 1 and 2 visualize the thermal denaturation state of biological tissue based on a captured image of fluorescence generated by irradiating the biological tissue with excitation light. Specifically, the technique described in Patent Document 1 displays regions of all pixels in the captured image with fluorescence intensities higher than a preset intensity as regions of high thermal denaturation. Conversely, the technique described in Patent Document 2 displays regions of the captured image with fluorescence intensities lower than a preset intensity as regions of high thermal denaturation.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: International Publication No. 2020 / 054723

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2017-23604 Summary of the Invention

[0008] Problems to be solved by the invention

[0009] Furthermore, if the thermal denaturation of biological tissue is low when it is treated with an energy device, the treatment may be insufficient. Therefore, if the operator or other user can be made aware of areas of biological tissue with insufficient thermal denaturation, convenience can be improved.

[0010] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a medical device, an endoscope system, a control method, a control program, and a learning device that can improve convenience.

[0011] Solutions for solving problems

[0012] In order to solve the above-mentioned problems and achieve the purpose, the medical device involved in the present invention includes a processor that processes a video image captured by fluorescence generated from biological tissue by irradiating the biological tissue with excitation light, wherein the processor performs the following processing: extracting an area consisting of pixels whose fluorescence intensity is less than a first fluorescence intensity from all pixels of the video image as an insufficient thermal denaturation area; and generating a display image that can distinguish the insufficient thermal denaturation area from other areas in all pixels of the video image.

[0013] The endoscope system involved in the present invention comprises: a light source device that irradiates excitation light; an endoscope that can be inserted into a subject and outputs a video image that captures fluorescence generated from the biological tissue in the subject by irradiating the excitation light to the biological tissue; and a medical device having a processor that processes the video image, wherein the processor performs the following processing: extracting an area composed of pixels whose fluorescence intensity is less than a first fluorescence intensity from all pixels of the video image as an insufficient thermal denaturation area; and generating a display image that can distinguish the insufficient thermal denaturation area from other areas in all pixels of the video image.

[0014] The control method according to the present invention is a control method executed by a medical device, wherein a region consisting of pixels having a fluorescence intensity of a first fluorescence intensity or less from all pixels in a captured image capturing fluorescence generated from living tissue by irradiating the living tissue with excitation light is extracted as a region insufficiently thermally denatured by thermal denaturation; and a display image is generated that can distinguish the region insufficiently thermally denatured from other regions in all pixels of the captured image.

[0015] A control program according to the present invention is a control program executed by a medical device, the control program being configured to instruct the medical device to perform the following processing: extracting, as a thermally insufficient region, a region consisting of pixels having a fluorescence intensity of a first fluorescence intensity or less from all pixels in a captured image in which fluorescence generated from living tissue by irradiating the living tissue with excitation light; and generating a display image capable of distinguishing the thermally insufficient region from other regions in all pixels of the captured image.

[0016] The learning device involved in the present invention includes a learning unit that generates a learned model by performing machine learning using training data, wherein the training data includes a fluorescence image captured by irradiating biological tissue with excitation light as input data, and information related to insufficiently thermally denatured areas extracted from the fluorescence image as output data.

[0017] Effects of the Invention

[0018] According to the medical device, endoscope system, control method, control program, and learning device according to the present invention, convenience can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram showing the overall configuration of an endoscope system according to an embodiment.

[0020] Figure 2 This is a block diagram showing the functional configuration of a main portion of an endoscope system according to an embodiment.

[0021] Figure 3 Graph showing the wavelength characteristics of the excitation light emitted by the second light source unit.

[0022] Figure 4 It is a graph showing the transmission characteristics of the cut filter.

[0023] Figure 5 A diagram illustrating the principle of observation in the fluorescence observation mode.

[0024] Figure 6 This is a diagram explaining the observation principle in the normal light observation mode.

[0025] Figure 7 is a flowchart illustrating a control method executed by the control device.

[0026] Figure 8 This is a diagram illustrating the control method.

[0027] Figure 9 This is a diagram illustrating the control method.

[0028] Figure 10 This is a diagram illustrating the control method.

[0029] Figure 11 This is a diagram illustrating the control method.

[0030] Figure 12 This is a diagram illustrating the control method. DETAILED DESCRIPTION

[0031] Hereinafter, the mode for implementing the present invention (hereinafter referred to as embodiment) will be described with reference to the accompanying drawings. In addition, the present invention is not limited to the embodiment described below. In addition, in the description of the accompanying drawings, the same reference numerals are used for the same parts.

[0032] [Overall structure of the endoscope system]

[0033] Figure 1 1 is a diagram showing the overall configuration of an endoscope system 1 according to the embodiment.

[0034] The endoscope system 1 according to this embodiment is used in holmium laser nucleation of the prostate (HoLEP), a surgical treatment for benign prostatic hyperplasia (BPH). Specifically, HoLEP is a surgical treatment that irradiates the boundary between the inner and outer glands of an enlarged prostate with a holmium YAG laser to remove the prostate.

[0035] like Figure 1 As shown, the endoscope system 1 includes an insertion portion 2 , a light source device 3 , a light guide 4 , a camera 5 , a first transmission cable 6 , a display device 7 , a second transmission cable 8 , a control device 9 , and a third transmission cable 10 .

[0036] The insertion portion 2 is rigid or at least partially flexible and has an elongated shape and is inserted into the subject (the bladder). An optical system such as a lens for forming an image of the subject is provided in the insertion portion 2 .

[0037] The light source device 3 is connected to one end of the light guide 4. Under the control of the control device 9, the light source device 3 supplies illumination light to the one end of the light guide 4 to irradiate the subject. The light source device 3 is implemented using the following components: any one or more light sources such as LED (Light Emitting Diode: Light Emitting Diode) light sources, xenon lamps and LD (Laser Diode: Laser Diode) semiconductor laser elements; a processor as a processing device having hardware such as FPGA (Field Programmable Gate Array: Field Programmable Gate Array), CPU (Central Processing Unit: Central Processing Unit); and a memory as a temporary storage area used by the processor. In addition, the light source device 3 and the control device 9 can be as follows: Figure 1 Although the structure is set as a structure in which communication is performed individually as shown, it can also be an integrated structure.

[0038] One end of the light guide 4 is detachably connected to the light source device 3 , and the other end is detachably connected to the insertion portion 2 . The light guide 4 guides illumination light supplied from the light source device 3 from one end to the other end, thereby supplying the illumination light to the insertion portion 2 .

[0039] The camera 5 is detachably connected to the eyepiece portion 21 of the insertion portion 2. Furthermore, under the control of the control device 9, the camera 5 receives the subject image formed by the insertion portion 2 and performs photoelectric conversion to generate image data (RAW data), and outputs the image data to the control device 9 via the first transmission cable 6.

[0040] The insertion portion 2 and the camera 5 described above correspond to the endoscope according to the present invention.

[0041] One end of the first transmission cable 6 is detachably connected to the control device 9 via a video connector 61, and the other end is detachably connected to the camera 5 via a camera connector 62. Furthermore, the first transmission cable 6 transmits image data output from the camera 5 to the control device 9, and transmits setting data and power output from the control device 9 to the camera 5. Here, the setting data includes control signals, synchronization signals, and clock signals for controlling the camera 5.

[0042] The display device 7 is composed of a display monitor such as liquid crystal or organic EL (Electro Luminescence), and displays images based on image data processed by the control device 9 and various information related to the endoscope system 1 under the control of the control device 9 .

[0043] One end of the second transmission cable 8 is detachably connected to the display device 7 , and the other end is detachably connected to the control device 9 . The second transmission cable 8 transmits image data processed by the control device 9 to the display device 7 .

[0044] The control device 9 corresponds to the medical device involved in the present invention. The control device 9 is implemented using the following components: a processor as a processing device having hardware such as a GPU (Graphics Processing Unit), an FPGA, or a CPU; and a memory as a temporary storage area used by the processor. Moreover, the control device 9 comprehensively controls the operation of the light source device 3, the camera 5, and the display device 7 through each of the first transmission cable 6, the second transmission cable 8, and the third transmission cable 10 according to the program recorded in the memory. In addition, the control device 9 performs various image processing on the image data input via the first transmission cable 6 and outputs it to the second transmission cable 8.

[0045] One end of the third transmission cable 10 is detachably connected to the light source device 3 , and the other end is detachably connected to the control device 9 . The third transmission cable 10 transmits control data from the control device 9 to the light source device 3 .

[0046] [Functional structure of the main parts of the endoscope system]

[0047] Next, the functional configuration of the main parts of the endoscope system 1 will be described.

[0048] Figure 2 1 is a block diagram showing the functional configuration of the main parts of the endoscope system 1 .

[0049] Hereinafter, the insertion portion 2 , the light source device 3 , the camera 5 , and the control device 9 will be described in this order.

[0050] [Structure of the insertion part]

[0051] First, the structure of the insertion portion 2 will be described.

[0052] like Figure 2 As shown, the insertion portion 2 includes an optical system 22 and an illumination optical system 23 .

[0053] The optical system 22 is composed of one or more lenses, and forms an image of the subject by converging reflected light from the subject, return light from the subject, excitation light from the subject, and fluorescence emitted by the subject.

[0054] The illumination optical system 23 is composed of one or more lenses and the like, and irradiates the illumination light supplied from the light guide 4 toward the subject.

[0055] [Structure of light source device]

[0056] Next, the structure of the light source device 3 will be described.

[0057] like Figure 2 As shown, the light source device 3 includes a condenser lens 30 , a first light source unit 31 , a second light source unit 32 , and a light source control unit 33 .

[0058] The condenser lens 30 condenses the light emitted by the first light source unit 31 and the second light source unit 32 and emits the light toward the light guide 4 .

[0059] The first light source unit 31 emits white light (normal light) as visible light under the control of the light source control unit 33, and supplies the white light as illumination light to the light guide 4. The first light source unit 31 is configured using a collimating lens, a white LED lamp, and a driver.

[0060] Alternatively, the first light source unit 31 may be configured to simultaneously emit red, green, and blue LEDs to supply visible white light. Alternatively, the first light source unit 31 may be configured with a halogen lamp, a xenon lamp, or the like.

[0061] The second light source unit 32 emits excitation light having a predetermined wavelength range under the control of the light source control unit 33 , thereby supplying the excitation light to the light guide 4 as illumination light.

[0062] Figure 3 is a diagram showing the wavelength characteristics of the excitation light emitted by the second light source unit 32. Specifically, Figure 3 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents wavelength characteristics. Figure 3 In the middle, curve L V represents the wavelength characteristics of the excitation light emitted by the second light source unit 32. Figure 3 In the middle, curve L B Indicates the wavelength range of blue, curve L G Indicates the wavelength range of green, curve L R Indicates the wavelength range of red.

[0063] Here, if Figure 3 As shown, the second light source unit 32 emits excitation light having a central wavelength (peak wavelength) of 415 nm and a wavelength range of 400 nm to 430 nm. The second light source unit 32 is constructed using a collimating lens, a semiconductor laser such as a violet LD, and a driver.

[0064] The light source control unit 33 is implemented using a processor (e.g., a hardware processor such as an FPGA or CPU) as a processing device and a memory (e.g., a temporary storage area) used by the processor. Furthermore, the light source control unit 33 controls the timing and duration of light emission of each of the first light source unit 31 and the second light source unit 32 based on control data input from the control device 9.

[0065] [Camera structure]

[0066] Next, the structure of the camera 5 will be described.

[0067] like Figure 2 As shown, the camera 5 includes an optical system 51 , a driving unit 52 , a cut filter 53 , an imaging element 54 , an A / D converter 55 , a P / S converter 56 , an imaging and recording unit 57 , an imaging control unit 58 , and an operating unit 59 .

[0068] The optical system 51 forms an image of the subject focused by the optical system 22 of the insertion portion 2 on the light receiving surface of the imaging element 54. The optical system 51 is composed of a plurality of lenses 511 ( Figure 2 ) is configured to be able to change the focal length and focus position. Specifically, the optical system 51 is configured such that each lens in the plurality of lenses 511 is driven by the drive unit 52 to move along the optical axis L1 ( Figure 2 ) to change the focal length and focus position.

[0069] The drive unit 52 is configured using a motor such as a stepping motor, a DC motor, or a voice coil motor, and a transmission mechanism such as gears that transmits the rotation of the motor to the optical system 51. Furthermore, the drive unit 52 moves the multiple lenses 511 of the optical system 51 along the optical axis L1 under the control of the imaging control unit 58.

[0070] The cut filter 53 is disposed on the optical axis L1 between the optical system 51 and the imaging element 54. The cut filter 53 blocks light in a predetermined wavelength range and transmits other light.

[0071] Figure 4 is a diagram showing the transmission characteristics of the cut filter 53. Specifically, Figure 4 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents wavelength characteristics. Figure 4 In the middle, curve L F The curve L represents the transmission characteristics of the cutoff filter 53. V represents the wavelength characteristics of the excitation light. Figure 4 In the middle, curve L NG This indicates the wavelength characteristics of fluorescence generated by irradiating excitation light onto advanced glycation end products generated by laser irradiation (heat treatment) of biological tissue using an energy device such as a holmium yttrium aluminum garnet laser.

[0072] Here, if Figure 4 As shown, the cutoff filter 53 blocks a portion of the excitation light reflected from the biological tissue in the observation area, while transmitting light in other wavelength ranges including fluorescent components. More specifically, the cutoff filter 53 blocks a portion of light in the wavelength range shorter than 400 nm to 430 nm including the excitation light, while transmitting light in the wavelength range longer than 430 nm including fluorescence generated by irradiating the excitation light onto the advanced glycation end products produced by heat treatment.

[0073] The imaging element 54 is constructed using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor, in which color filters forming a Bayer pattern (RGGB) are arranged in each of a plurality of pixels arranged in a two-dimensional matrix. Furthermore, under the control of the imaging control unit 58, the imaging element 54 receives the subject image formed by the optical system 51 and passed through the cut filter 53, performs photoelectric conversion on the image, generates image data (RAW data), and outputs it to the A / D converter 55.

[0074] The A / D conversion section 55 is configured using an A / D conversion circuit or the like, and performs A / D conversion processing on analog image data input from the imaging element 54 under the control of the imaging control section 58 , and outputs the converted data to the P / S conversion section 56 .

[0075] The P / S conversion unit 56 is constructed using a P / S conversion circuit, etc., and under the control of the camera control unit 58, performs parallel / serial conversion on the digital image data (equivalent to the camera image involved in the present invention) input from the A / D conversion unit 55, and outputs the image data after parallel / serial conversion to the control device 9 via the first transmission cable 6.

[0076] In addition, the following structure can also be set: an E / O conversion unit that converts image data into an optical signal is provided instead of the P / S conversion unit 56, and the image data is output to the control device 9 through the optical signal. In addition, the following structure can also be set: the image data is sent to the control device 9 through wireless communications such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0077] The image recording unit 57 is composed of a nonvolatile memory or a volatile memory and is used to record various information related to the camera 5 (for example, pixel information of the image sensor 54 and the characteristics of the cut filter 53). In addition, the image recording unit 57 records various setting data and control parameters transmitted from the control device 9 via the first transmission cable 6.

[0078] The image capture control unit 58 is implemented using the following components: a TG (Timing Generator); a processor (processing device) including hardware such as a CPU; and a memory serving as a temporary storage area used by the processor. Furthermore, the image capture control unit 58 controls the operation of the drive unit 52, the image sensor 54, the A / D converter 55, and the P / S converter 56 based on setting data received from the control device 9 via the first transmission cable 6.

[0079] The operation unit 59 is composed of buttons, switches, etc., and is used to receive user operations performed by a user such as an operator, and outputs an operation signal corresponding to the user operation to the control device 9. As an example of such user operation, an operation of switching the observation mode of the endoscope system 1 to a normal light observation mode or a fluorescence observation mode can be exemplified.

[0080] [Structure of control device]

[0081] Next, the configuration of the control device 9 will be described.

[0082] like Figure 2 As shown, the control device 9 includes an S / P conversion unit 91 , an image processing unit 92 , an input unit 93 , a recording unit 94 , and a control unit 95 .

[0083] Under the control of the control unit 95 , the S / P conversion unit 91 performs serial / parallel conversion on the image data received from the camera 5 via the first transmission cable 6 and outputs the result to the image processing unit 92 .

[0084] Furthermore, when the camera 5 outputs image data via optical signals, an O / E converter that converts the optical signal into an electrical signal may be provided in place of the S / P converter 91. Furthermore, when the camera 5 transmits image data via wireless communication, a communication module capable of receiving wireless signals may be provided in place of the S / P converter 91.

[0085] The image processing unit 92 corresponds to the processor involved in the present invention. This image processing unit 92 is implemented using the following components: a processor, comprising hardware such as a GPU or FPGA, as a processing device; and a memory, serving as a temporary storage area used by the processor. Furthermore, under the control of the control unit 95, the image processing unit 92 performs predetermined image processing on the image data input as parallel data from the S / P conversion unit 91, and then outputs the data to the display device 7. Examples of the predetermined image processing include demosaicing, white balance, gain adjustment, gamma correction, and format conversion.

[0086] The input unit 93 is composed of a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, etc., receives user operations performed by a user such as an operator, and outputs operation signals corresponding to the user operations to the control unit 95.

[0087] The recording unit 94 is configured using a recording medium such as a volatile memory, a nonvolatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory card. The recording unit 94 records data including various parameters required for the operation of the endoscope system 1. The recording unit 94 also includes a program recording unit 941 for recording various programs used to operate the endoscope system 1.

[0088] The control unit 95 is realized using a processor as a processing device including hardware such as an FPGA or a CPU, and a memory as a temporary storage area used by the processor. Furthermore, the control unit 95 comprehensively controls the various components constituting the endoscope system 1 .

[0089] [Observation Principles of the Endoscope System in Observation Mode]

[0090] Next, the observation principle in the observation mode of the endoscope system 1 will be described.

[0091] The following describes the fluorescence observation mode and the normal light observation mode in this order.

[0092] [Observation principle in fluorescence observation mode]

[0093] First, the principle of observation in the fluorescence observation mode will be described.

[0094] Figure 5 A diagram illustrating the principle of observation in the fluorescence observation mode.

[0095] like Figure 5 As shown in the diagram G11, first, the light source device 3, under the control of the control device 9, emits the excitation light (central wavelength 415 nm) by making the second light source unit 32 emit light to irradiate the biological tissue O10 (heat-treated area) that has been laser-irradiated (heat-treated) by the holmium yttrium aluminum garnet laser. In this case, as Figure 5 As shown in the graph G12, at least the components of the excitation light reflected by the biological tissue O10 (heat treatment area) and the reflected light of the return light (hereinafter referred to as reflected light W10) are shielded by the cutoff filter 53 and the intensity is reduced. On the other hand, a part of the components on the long wavelength side longer than the shielded wavelength range is incident on the imaging element 54 without reducing the intensity.

[0096] More specifically, if Figure 5 As shown in the graph G12 of FIG. 1 , the cut filter 53 blocks most of the reflected light W10 in the wavelength range on the short wavelength side including the wavelength range of the excitation light incident on the G pixel of the imaging element 54, and transmits the reflected light W10 in the wavelength range on the long wavelength side longer than the blocked wavelength range. Figure 5 As shown in graph G12, the cutoff filter 53 transmits fluorescence WF10 emitted by the advanced glycation end products themselves in the biological tissue O10 (heat-treated area). Therefore, the reflected light W10 with reduced intensity and the fluorescence WF10 enter each of the R, G, and B pixels of the imaging element 54.

[0097] Here, the G pixel in the imaging element 54 is sensitive to the fluorescence WF10. Figure 5 The fluorescence characteristic curve L in the graph G12 NG As shown, the fluorescence response is very small. Therefore, the output value corresponding to the fluorescence WF10 in the G pixel is small.

[0098] The image processing unit 92 then acquires image data (RAW data) from the imaging element 54, performs image processing on the output values ​​of the G and B pixels contained in this image data, and generates a fluorescence image. In this case, the output values ​​of the G pixels include fluorescence information corresponding to the fluorescence WF10 emitted from the thermally treated area. Furthermore, the output values ​​of the B pixels include background information from the subject's biological tissue, including the thermally treated area. Furthermore, displaying this fluorescence image on the display device 7 allows observation of the biological tissue (thermally treated area) thermally treated with a holmium yttrium aluminum garnet laser or the like.

[0099] [Observation principle in normal light observation mode]

[0100] Next, the observation principle in the normal light observation mode will be described.

[0101] Figure 6 This is a diagram explaining the observation principle in the normal light observation mode.

[0102] like Figure 6 As shown in the diagram G21, first, the light source device 3, under the control of the control device 9, emits white light by causing the first light source unit 31 to illuminate the living tissue O10. In this case, a portion of the reflected light and the return light (hereinafter referred to as reflected light WR30, WG30, and WB30) reflected by the living tissue O10 is shielded by the cutoff filter 53, and the remaining portion is incident on the imaging element 54. Specifically, as shown in FIG. Figure 6 As shown in graph G22, the cutoff filter 53 blocks reflected light in the wavelength range shorter than the wavelength range of the excitation light. Therefore, the component of light in the blue wavelength range incident on the B pixel in the imaging element 54 is smaller than when the cutoff filter 53 is not provided.

[0103] The image processing unit 92 then acquires image data (RAW data) from the imaging element 54, performs image processing on the output values ​​of the R, G, and B pixels contained in this image data, and generates an observation image (white light image). In this case, the image processing unit 92 performs white balance adjustment processing. Since the blue component contained in the image data is smaller than when the cut filter 53 is not configured, the white balance is adjusted to maintain a constant ratio among the red, green, and blue components. Furthermore, when this observation image (white light image) is displayed on the display device 7, a natural observation image (white light image) can be observed even when the cut filter 53 is configured.

[0104] [Control Method]

[0105] Next, the control method executed by the control device 9 will be described.

[0106] Figure 7 1 is a flowchart showing a control method executed by the control device 9 . Figures 8 to 12 is a diagram illustrating the control method. Specifically, Figure 8 The correlation between the fluorescence intensity of the fluorescence emitted by the advanced glycation end products in the biological tissue and the degree of invasion (depth and area) of the biological tissue caused by heat treatment (straight line L Y ). In addition, Figure 8 In the figure, the vertical axis represents the fluorescence intensity, and the horizontal axis represents the degree of invasion into the biological tissue caused by heat treatment. Figure 9 FIG. 1 is a diagram showing the fluorescent image F1 generated in step S3 . Figure 10 is with Figure 9 The corresponding figure is a figure for explaining step S4. Figure 11 is with Figure 9 The corresponding figure is a figure for explaining step S5. Figure 12 is with Figure 9 The corresponding figure shows the display image F2 generated in step S6.

[0107] The following describes a control method executed by the control device 9 during holmium laser prostatectomy. Specifically, the insertion portion 2 is inserted into the subject (the bladder), and the observation area of ​​the endoscope system 1 becomes the living tissue (heat-treated area) that has been thermally treated by the holmium yttrium aluminum garnet laser.

[0108] First, the control unit 95 switches the observation mode to the fluorescence observation mode in response to an operation of “switching the observation mode of the endoscope system 1 to the fluorescence observation mode” performed on the operation unit 59 by a user such as an operator (step S1 ).

[0109] After step S1 , the control unit 95 controls the light source control unit 33 to cause the second light source unit 32 to emit excitation light (step S2 ).

[0110] After step S2 , the image processing unit 92 generates a fluorescence image based on the image data generated by the imaging element 54 (step S3 ).

[0111] In addition, the fluorescence intensity of the fluorescence emitted by the advanced glycation end products in the biological tissue is related to the degree of invasion of the biological tissue caused by heat treatment (the degree of thermal denaturation). Figure 8 Specifically, Figure 8 The straight line L Y As shown, the higher the degree of thermal denaturation (the greater the degree of invasion of biological tissues due to heat treatment), the higher the fluorescence intensity.

[0112] Here, in Figure 9 In the fluorescent image F1 shown, the area ArF1 filled with white is the area with the fluorescence intensity of the first fluorescence intensity Th1 ( Figure 8 ) or less, and the degree of thermal denaturation is low, resulting in insufficient thermal denaturation. In such an insufficient thermal denaturation area ArF1, due to insufficient thermal denaturation, hemostasis is insufficient, and bleeding may occur after surgery. In addition, in the fluorescence image F1, the area ArF2 filled with black is composed of pixels with a fluorescence intensity of the second fluorescence intensity Th2 ( Figure 8 ) or more, and has a high degree of thermal denaturation, resulting in excessive thermal denaturation. In this excessive thermal denaturation region ArF2, excessive thermal denaturation may lead to perforation after surgery. Furthermore, in the fluorescence image F1, region ArF3, excluding regions ArF1 and ArF2, is a moderate thermal denaturation region, consisting of pixels having a fluorescence intensity greater than the first fluorescence intensity Th1 and less than the second fluorescence intensity Th2, and has an appropriate degree of thermal denaturation.

[0113] Furthermore, if the operator or other user can be aware of the insufficiently thermally denatured region ArF1 and the excessively thermally denatured region ArF2, they can implement corresponding countermeasures to reduce the risk of postoperative bleeding and perforation. However, even if the fluorescence image F1 is displayed on the display device 7, it is difficult for the operator or other user to identify the regions ArF1 to ArF3 based on the fluorescence image F1 because the fluorescence intensity changes gradually.

[0114] Therefore, in this embodiment, by executing steps S4 to S6 described below, a display image capable of distinguishing the regions ArF1 to ArF3 is generated.

[0115] Specifically, the image processing unit 92 is as follows Figure 10As shown, the insufficient thermal denaturation region ArF1 consisting of pixels having a fluorescence intensity equal to or less than the first fluorescence intensity Th1 among all pixels of the fluorescence image F1 generated in step S3 is extracted (step S4 ).

[0116] In addition, the image processing unit 92 is as follows Figure 11 As shown, the thermal denaturation transition region ArF2 consisting of pixels having a fluorescence intensity equal to or greater than the second fluorescence intensity Th2 among all pixels of the fluorescence image F1 generated in step S3 is extracted (step S5 ).

[0117] Furthermore, steps S4 and S5 may be executed in the order of steps S4 and S5, or in the order of steps S5 and S4, or may be executed in parallel and substantially simultaneously.

[0118] Here, as examples of the fluorescence intensity used in steps S4 and S5, the output value of the G pixel in the imaging element 54, at least the g value among the pixel values ​​(r, g, b) of each pixel after de-mosaicing the image data acquired from the imaging element 54, or the brightness value corresponding to the Y signal (brightness signal), etc. can be exemplified.

[0119] Then, the image processing unit 92 Figure 12 As shown, a display image F2 capable of distinguishing the regions ArF1 to ArF3 is generated (step S6).

[0120] Here, the phrase "being able to distinguish each region ArF1 to ArF3" means that the insufficiently heat-denatured region ArF1, the excessively heat-denatured region ArF2, and the appropriately heat-denatured region ArF3 may be made different colors. Alternatively, the entire regions ArF1 to ArF3 may be made different colors, or the frame portions forming the outer edges of the regions ArF1 and ArF2 may be made different colors. Figure 12 , the frame portions forming the outer edges of the regions ArF1 and ArF2 are shown in different colors. In addition to the above-described distinction using colors, the regions ArF1 to ArF3 may also be distinguished using annotations (characters).

[0121] According to the present embodiment described above, the following effects are achieved.

[0122] In the control device 9 of this embodiment, the image processing unit 92 extracts, as the insufficient thermal denaturation region ArF1, a region consisting of pixels having a fluorescence intensity of less than or equal to the first fluorescence intensity Th1 from all pixels of the captured image generated by the imaging element 54. The image processing unit 92 then generates a display image F2 that can distinguish the insufficient thermal denaturation region ArF1 from other regions in all pixels of the captured image.

[0123] Therefore, the operator or other user can be made aware of the insufficient thermal denaturation region ArF1 where the body tissue is insufficiently thermally denatured and hemostasis is insufficient, which may cause bleeding after surgery, thereby improving convenience.

[0124] Furthermore, in the control device 9 according to the present embodiment, the image processing unit 92 extracts, as the excessive thermal denaturation region ArF2, a region consisting of pixels having a fluorescence intensity greater than or equal to a second fluorescence intensity Th2, which is greater than the first fluorescence intensity Th1, from the captured image generated by the imaging element 54. The image processing unit 92 then generates a display image F2 that can distinguish the excessive thermal denaturation region ArF2 from other regions in all pixels of the captured image.

[0125] Therefore, the user such as the operator can be made aware of the excessive thermal denaturation region ArF2 where the body tissue is excessively thermally denatured and may perforate after surgery, thereby further improving convenience.

[0126] (Other embodiments)

[0127] Although the modes for carrying out the present invention have been described above, the present invention should not be limited to the above-described embodiments.

[0128] In the above-described embodiment, the medical device according to the present invention is mounted on an endoscope system used in holmium laser prostatectomy. However, the present invention is not limited thereto and may be mounted on an endoscope system used in other procedures.

[0129] In the above-described embodiment, the medical device according to the present invention is mounted on an endoscope system using a rigid endoscope. However, the present invention is not limited thereto and may be mounted on an endoscope system using a flexible endoscope or an endoscope system using a medical surgical robot.

[0130] In the above embodiment, a display image F2 is generated that can distinguish both the insufficiently thermally denatured region ArF1 and the excessively thermally denatured region ArF2 from other regions in all pixels of the captured image generated by the imaging element 54. However, the present invention is not limited to this embodiment. For example, a display image F2 can be generated that can distinguish only the insufficiently thermally denatured region ArF1 from other regions in all pixels of the captured image.

[0131] In the above embodiment, a display image F2 is generated that distinguishes the insufficiently thermally denatured region ArF1 and the excessively thermally denatured region ArF2 from other regions in all pixels of the fluorescence image F1. However, the present invention is not limited to this embodiment. For example, the fluorescence image F1 and the observation image (white light image) can be generated in a time-division manner by alternating between the fluorescence observation mode and the normal light observation mode. Alternatively, a display image can be generated that distinguishes the region corresponding to the insufficiently thermally denatured region ArF1 and the region corresponding to the excessively thermally denatured region ArF2 from other regions in all pixels of the observation image (white light image) generated at approximately the same time as the fluorescence image F1.

[0132] In the above embodiment, the control unit 95 may also function as a learning unit of the learning device according to the present invention. In this case, the control unit 95 corresponds to the learning device according to the present invention.

[0133] Specifically, the control unit 95 generates a learned model by performing machine learning using training data. The training data includes a fluorescence image captured by irradiating biological tissue with excitation light, which serves as input data, and information regarding insufficiently thermally denatured regions, excessively thermally denatured regions, and appropriately thermally denatured regions extracted from the fluorescence image, serving as output data. The output data only needs to include information regarding insufficiently thermally denatured regions and may not include information regarding excessively thermally denatured regions or appropriately thermally denatured regions.

[0134] Here, the learned model is composed of a neural network with one or more nodes in each layer. The type of machine learning is not particularly limited. For example, training data and learning data are prepared by pairing fluorescence images of multiple subjects with information extracted from these fluorescence images, at least related to areas of insufficient thermal denaturation, and then inputting these training data and learning data into a computational model based on a multi-layer neural network to perform learning. Furthermore, machine learning techniques include, for example, techniques based on multi-layer neural networks such as CNN (Convolutional Neural Network) and 3D-CNN (Deep Neural Network). Furthermore, techniques based on recurrent neural networks (RNN) or LSTM (Long Short-Term Memory Units), which are extensions of RNNs, may also be used. Furthermore, a learning unit of a learning device separate from the control device 9 may perform these functions.

[0135] Description of Reference Numerals

[0136] 1: Endoscope system; 2: Insertion unit; 3: Light source device; 4: Light guide; 5: Camera; 6: First transmission cable; 7: Display device; 8: Second transmission cable; 9: Control device; 10: Third transmission cable; 21: Eyepiece; 22: Optical system; 23: Illumination optical system; 30: Condenser lens; 31: First light source; 32: Second light source; 33: Light source control unit; 51: Optical system; 52: Drive unit; 53: Cutoff filter; 54: Image sensor; 55: A / D converter; 56: P / S converter Switching unit; 57: Video recording unit; 58: Video control unit; 59: Operation unit; 61: Video connector; 62: Camera connector; 91: S / P conversion unit; 92: Image processing unit; 93: Input unit; 94: Recording unit; 95: Control unit; 511: Lens; 941: Program recording unit; ArF1: Insufficient thermal denaturation area; ArF2: Excessive thermal denaturation area; ArF3: Appropriate thermal denaturation area; G11 to G13, G21 to G23: Graphs; F1: Fluorescent image; F2: Display image; L1: Optical axis; L B , L G , L R , L V : curve; L F : Transmission characteristics of the cutoff filter; L NG : wavelength characteristics of fluorescence; L Y : straight line; O10: biological tissue; Th1: first fluorescence intensity; Th2: second fluorescence intensity; W10, WB30, WG30, WR30: reflected light; WF10: fluorescence.

Claims

1. A medical device comprising a processor for processing a captured image of fluorescence generated from living tissue by irradiating the living tissue with excitation light, wherein: The processor performs the following processing: extracting a region consisting of pixels having a fluorescence intensity of less than or equal to a first fluorescence intensity from among all pixels of the captured image as an insufficient thermal denaturation region; as well as A display image is generated that can distinguish the thermal denaturation insufficient region from other regions in all pixels of the captured image.

2. The medical device according to claim 1, wherein The fluorescence is generated from advanced glycation end products produced by heat treatment of the biological tissue.

3. The medical device according to claim 1, wherein The processor performs the following processing: extracting, from all pixels of the captured image, a region consisting of pixels having a fluorescence intensity greater than or equal to a second fluorescence intensity greater than the first fluorescence intensity as an excessive thermal denaturation region; as well as The display image is generated so as to distinguish the excessive thermal denaturation region from other regions in all pixels of the captured image.

4. The medical device according to claim 1, wherein The display image is an image capable of distinguishing the insufficiently thermally denatured region from other regions on the captured image.

5. The medical device according to claim 1, wherein The processor acquires a white light image obtained by photographing the biological tissue irradiated with white light, The display image is an image capable of distinguishing the insufficient thermal denaturation region from other regions on the white light image.

6. An endoscope system comprising: a light source device for irradiating excitation light; an endoscope that is insertable into a subject and outputs a captured image of fluorescence generated from the living tissue within the subject by irradiating the living tissue with the excitation light; as well as A medical device having a processor for processing the captured image, wherein: The processor performs the following processing: extracting a region consisting of pixels having a fluorescence intensity equal to or less than a first fluorescence intensity from among all pixels of the captured image as a region having insufficient thermal denaturation; as well as A display image is generated that can distinguish the thermal denaturation insufficient region from other regions in all pixels of the captured image.

7. A control method performed by a medical device, wherein: A method for extracting a region consisting of pixels having a fluorescence intensity of less than or equal to a first fluorescence intensity from all pixels of a captured image in which fluorescence generated from living tissue by irradiating the living tissue with excitation light is captured as a region with insufficient thermal denaturation; and generating a display image capable of distinguishing the region with insufficient thermal denaturation from other regions in all pixels of the captured image.

8. A control program for causing a medical device to execute a control program, wherein: The control program is used to instruct the medical device to perform the following processing: A method for extracting a region consisting of pixels having a fluorescence intensity of less than or equal to a first fluorescence intensity from all pixels of a captured image in which fluorescence generated from living tissue by irradiating the living tissue with excitation light is captured as a region with insufficient thermal denaturation; and generating a display image capable of distinguishing the region with insufficient thermal denaturation from other regions in all pixels of the captured image.

9. A learning device comprising a learning unit, wherein: The learning unit generates a learned model by performing machine learning using training data, wherein the training data includes a fluorescence image captured of fluorescence generated from biological tissue by irradiating the biological tissue with excitation light as input data, and information related to an insufficiently thermally denatured region extracted from the fluorescence image as output data.

Images