Medical device, endoscope system, control method, control program, and learning device

Abstract

A medical device (4) is provided with a processor (409) for processing a captured image in which fluorescence generated from biological tissue by irradiating the biological tissue with excitation light has been captured. The processor (409) determines a change in the state of the thermal denaturation on the basis of the captured image, and, on the basis of the determination result of the change in the state of the thermal denaturation, transmits a control signal for controlling the operation of the perfusion device (6) to the perfusion device (6) that perfuses the perfusion liquid.

Description

Technical Field

[0001] The present invention relates to a medical device, an endoscope system, a control method, a control program, and a learning device. Background Art

[0002] Conventionally, there is known a technique for visualizing the state of thermal denaturation of living tissue when the living tissue is treated with an energy device or the like (for example, see Patent Document 1).

[0003] The technology described in Patent Document 1 visualizes the state of thermal denaturation of biological tissue based on a captured image of fluorescence generated from biological tissue by irradiating the tissue with excitation light. Specifically, the technology described in Patent Document 1 detects regions with fluorescence intensities exceeding a predetermined intensity across all pixels in the captured image as regions of high thermal denaturation.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: International Publication No. 2020 / 174666 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] In addition, a surgical method known as transurethral ureterolithotripsy (TUL) is conventionally known. Specifically, transurethral ureterolithotripsy is a surgical method in which an insertion portion of an endoscope is inserted through the urethra, the distal end of the insertion portion is guided to the stone in the ureter or renal pelvis, the stone is observed while being crushed by laser irradiation, and the crushed stone is removed from the body. Furthermore, during transurethral ureterolithotripsy, when the stone is crushed by laser irradiation, if the laser irradiating the stone hits the living tissue, or if the high-temperature stone irradiated by the laser comes into contact with the living tissue, the effect may spread to the living tissue.

[0009] Here, when the technology described in Patent Document 1 is applied, although the state of thermal denaturation in living tissue can be visualized, it is difficult to appropriately cope with the effects of laser irradiation on the living tissue.

[0010] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a medical device, an endoscope system, a control method, a control program, and a learning device capable of appropriately controlling perfusion in response to the effects of laser irradiation on living tissue.

[0011] Solutions for solving problems

[0012] To solve the above-mentioned problems and achieve the object, the medical device according to the present invention includes a processor that processes a captured image of fluorescence generated from living tissue by irradiating the living tissue with excitation light. The processor determines a change in the state of thermal denaturation based on the captured image, and transmits a control signal for controlling the operation of a perfusion device that performs perfusion of a perfusion fluid based on the result of the determination of the change in the state of thermal denaturation.

[0013] An endoscope system according to the present invention comprises: a light source device for irradiating excitation light; an endoscope capable of being inserted into a subject and outputting a camera image capturing fluorescence generated from living tissue within the subject by irradiating the living tissue with the excitation light; and a medical device having a processor for processing the camera image, wherein the processor performs the following processing: determining a change in state of thermal denaturation based on the camera image; and transmitting a control signal for controlling the operation of a perfusion device that performs perfusion of a perfusion fluid based on the result of the determination of the change in state of thermal denaturation.

[0014] A control method according to the present invention is a control method executed by a medical device, wherein a change in the state of thermal denaturation is determined based on a captured image of fluorescence generated from living tissue by irradiating the living tissue with excitation light, and a control signal for controlling the operation of the perfusion device is transmitted to the perfusion device that performs perfusion of a perfusion fluid based on the result of the determination of the change in the state of thermal denaturation.

[0015] A control program according to the present invention is a control program executed by a medical device, wherein the control program instructs the medical device to perform the following processing: determining a change in the state of thermal denaturation based on a captured image of fluorescence generated from living tissue by irradiating the living tissue with excitation light; and transmitting a control signal for controlling the operation of a perfusion device that performs perfusion of a perfusion fluid based on the result of the determination of the change in the state of thermal denaturation.

[0016] A learning device according to the present invention includes a learning unit that generates a learned model by performing machine learning using training data, wherein the training data includes, as input data, a fluorescence image obtained by capturing fluorescence generated from living tissue by irradiating the tissue with excitation light, and output data including, as output data, information corresponding to a control signal for controlling the operation of a perfusion device that performs perfusion of a perfusion fluid based on a change in the state of thermal denaturation extracted from the fluorescence image.

[0017] Effects of the Invention

[0018] According to the medical device, endoscope system, control method, control program, and learning device according to the present invention, it is possible to appropriately control perfusion in response to the effect of laser irradiation on living tissue. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram showing the overall configuration of an endoscope system according to an embodiment.

[0020] Figure 2 This is a block diagram showing the functional configuration of a main portion of an endoscope system according to an embodiment.

[0021] Figure 3 : is a diagram showing the wavelength characteristics of the excitation light emitted by the second light source unit.

[0022] Figure 4 It is a graph showing the transmission characteristics of the cut filter.

[0023] Figure 5 A diagram illustrating the principle of observation in the fluorescence observation mode.

[0024] Figure 6 A diagram illustrating the principle of observation in the normal light observation mode.

[0025] Figure 7 is a flowchart illustrating a control method executed by the control device.

[0026] Figure 8 This is a diagram illustrating the control method.

[0027] Figure 9 This is a diagram illustrating the control method.

[0028] Figure 10 This is a diagram illustrating the control method. DETAILED DESCRIPTION

[0029] Hereinafter, the mode for implementing the present invention (hereinafter referred to as embodiment) will be described with reference to the accompanying drawings. In addition, the present invention is not limited to the embodiment described below. In addition, in the description of the accompanying drawings, the same reference numerals are used for the same parts.

[0030] [Overall structure of the endoscope system]

[0031] Figure 1 1 is a diagram showing the overall configuration of an endoscope system 1 according to the embodiment.

[0032] The endoscope system 1 involved in this embodiment is an endoscope system used in transurethral ureterolithotripsy (TUL). Specifically, in transurethral ureterolithotripsy, the insertion portion 21 of the endoscope 2 is inserted into the urinary tract to capture the interior of the subject, and a display image based on the captured image data is displayed on the display device 3. Moreover, while confirming the displayed image, the operator irradiates the stone in the subject with a laser from the laser irradiation device 5 to break up the stone, removes the broken stone using a disposal instrument such as a basket catheter, and leaves a medical device in the urinary tract until a specified period of time has passed. Here, the medical device is any one of a stent, a catheter, and an indwelling needle.

[0033] like Figure 1 As shown, the endoscope system 1 includes an endoscope 2 , a display device 3 , a control device 4 , a laser irradiation device 5 , and an irrigation device 6 .

[0034] The endoscope 2 generates image data (RAW data) obtained by capturing the interior of the subject, and outputs the image data to the control device 4. Figure 1 As shown, the endoscope 2 includes an insertion portion 21 , an operation portion 22 , and a universal cable 23 .

[0035] At least a portion of the insertion portion 21 is flexible, and the insertion portion 21 is inserted into the subject. Figure 1 As shown, the insertion portion 21 includes: a front end portion 24, which is arranged at the front end of the insertion portion 21; a bending portion 25, which is connected to the base end side (operation portion 22 side) of the front end portion 24 and is configured to be bendable; and a long flexible tube portion 26, which is connected to the base end side of the bending portion 25 and is flexible.

[0036] The operation part 22 is connected to the base end portion of the insertion part 21. The operation part 22 receives various operations for the endoscope 2. Figure 1 As shown, a bending knob 221 , an insertion port 222 , and a plurality of operating members 223 are provided.

[0037] The bending knob 221 is configured to be rotatable by a user operation such as an operator. The bending knob 221 is rotated to activate a bending mechanism (not shown) such as a metal or resin wire disposed within the insertion portion 21. As a result, the bending portion 25 bends.

[0038] The insertion port 222 communicates with a treatment instrument channel (not shown) which is a conduit extending from the distal end of the insertion portion 21 and is used to insert a treatment instrument or the like from the outside of the endoscope 2 into the treatment instrument channel.

[0039] The plurality of operating members 223 are composed of buttons and the like that accept various operations by a user such as an operator, and output operation signals corresponding to the various operations to the control device 4 via the universal cable 23. Examples of the various operations include switching the observation mode of the endoscope system 1 to a normal light observation mode or a fluorescence observation mode.

[0040] The universal cable 23 extends from the operation portion 22 in a direction different from the extension direction of the insertion portion 21 and is provided with a light guide 231 (see FIG. Figure 2 ), the first signal line 232 for transmitting the above-mentioned image data (refer to Figure 2 ), a second signal line 233 (refer to Figure 2 ) etc. Moreover, at the base end of the universal cable 23, as Figure 1 As shown, a first connector portion 27, a second connector portion 28 and a cable 27a are provided.

[0041] The first connector portion 27 is detachably connected to the control device 4 .

[0042] The cable 27 a is a coiled cable extending from the first connector portion 27 .

[0043] The second connector portion 28 is provided at the front end of the cable 27 a and is detachably connected to the control device 4 .

[0044] The display device 3 is composed of a display monitor such as liquid crystal or organic EL (Electro Luminescence), and displays images based on image data processed by the control device 4 and various information related to the endoscope system 1 under the control of the control device 4 .

[0045] The control device 4 corresponds to the medical device of the present invention. This control device 4 is implemented using a processor (a processing device) comprising hardware such as a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or a CPU (Central Processing Unit); and a memory serving as a temporary storage area for the processor. Furthermore, the control device 4 comprehensively controls the operation of each component of the endoscope system 1 according to the program stored in the memory.

[0046] The laser irradiation device 5 irradiates a high-output infrared laser, such as a holmium YAG laser, under the control of the control device 4. Specifically, the laser irradiation device 5 is inserted through the insertion port 222 and into the urinary tract (e.g., the kidney, ureter, bladder, and urethra) via the treatment instrument channel within the insertion portion 21. Furthermore, the laser irradiation device 5 irradiates the laser toward the calculi formed within the subject in response to user manipulation by the operator, thereby fragmenting the calculi.

[0047] The perfusion device 6 is composed of a tube, a pump, etc. Figure 1 As shown, the insertion port 222 communicates with the treatment instrument channel in the insertion portion 21 .

[0048] Here, the urinary tract is filled with a perfusion fluid such as physiological saline. Furthermore, under the control of the control device 4, the perfusion device 6 delivers the perfusion fluid into the urinary tract from the insertion port 222 through the treatment instrument channel in the insertion portion 21, and discharges the perfusion fluid in the urinary tract to the outside of the urinary tract.

[0049] [Functional structure of the main parts of the endoscope system]

[0050] Next, the functional configuration of the main parts of the endoscope system 1 will be described.

[0051] Figure 2 1 is a block diagram showing the functional configuration of the main parts of the endoscope system 1 .

[0052] Next, the endoscope 2 and the control device 4 will be described in this order.

[0053] [Structure of an endoscope]

[0054] First, the structure of the endoscope 2 will be described.

[0055] like Figure 2 As shown, the endoscope 2 includes an illumination optical system 201 , an imaging optical system 202 , a cut filter 203 , an imaging element 204 , an A / D converter 205 , a P / S converter 206 , an imaging and recording unit 207 , and an imaging control unit 208 .

[0056] Here, the illumination optical system 201 , the imaging optical system 202 , the cut filter 203 , the imaging element 204 , the A / D converter 205 , the P / S converter 206 , the imaging and recording unit 207 , and the imaging control unit 208 are each arranged in the distal end portion 24 .

[0057] The illumination optical system 201 is composed of one or a plurality of lenses and the like, and irradiates the subject with illumination light supplied from the light guide 231 .

[0058] The imaging optical system 202 is composed of one or more lenses, and forms an image of the subject on the light receiving surface of the imaging element 204 by converging reflected light from the subject, return light from the subject, fluorescence emitted from the subject, and other light.

[0059] The cut filter 203 is disposed on the optical axis L1 of the imaging optical system 202, between the imaging optical system 202 and the imaging element 204. The cut filter 203 blocks light in a predetermined wavelength range and transmits other light.

[0060] Note that the transmission characteristics of the cut filter 203 will be described in the “Configuration of the Control Device” to be described later.

[0061] The imaging element 204 is constructed using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor, with one of the color filters forming a Bayer array (RGGB) arranged at each pixel in a two-dimensional matrix. Under the control of the imaging control unit 208, the imaging element 204 receives the subject image formed by the imaging optical system 202 and passed through the cut filter 203, performs photoelectric conversion to generate image data (RAW data), and outputs it to the A / D converter 205.

[0062] The A / D conversion section 205 is configured using an A / D conversion circuit or the like, and performs A / D conversion processing on analog image data input from the imaging element 204 under the control of the imaging control section 208 , and outputs the result to the P / S conversion section 206 .

[0063] The P / S conversion unit 206 is constructed using a P / S conversion circuit, etc., and under the control of the camera control unit 208, performs parallel / serial conversion on the digital image data (equivalent to the camera image involved in the present invention) input from the A / D conversion unit 205, and outputs it to the control device 4 via the first signal line 232.

[0064] Alternatively, an E / O converter that converts image data into an optical signal may be provided in place of the P / S converter 206, and the image data may be output via the optical signal to the controller 4. Alternatively, the image data may be transmitted to the controller 4 via wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0065] The image recording unit 207 is composed of a nonvolatile memory or a volatile memory and records various information related to the endoscope 2 (for example, pixel information of the image sensor 204 and characteristics of the cutoff filter 203). In addition, the image recording unit 207 records various setting data and control parameters transmitted from the control device 4 via the second signal line 233.

[0066] The image capture control unit 208 is implemented using the following components: a TG (Timing Generator); a processor (a processing device comprising hardware such as a CPU); and a memory serving as a temporary storage area used by the processor. Furthermore, the image capture control unit 208 controls the operation of the image sensor 204, the A / D converter 205, and the P / S converter 206 based on setting data received from the control device 4 via the second signal line 233.

[0067] [Structure of control device]

[0068] Next, the configuration of the control device 4 will be described.

[0069] like Figure 2 As shown, the control device 4 includes a condenser lens 401 , a first light source unit 402 , a second light source unit 403 , a light source control unit 404 , an S / P conversion unit 405 , an image processing unit 406 , an input unit 407 , a recording unit 408 , and a control unit 409 .

[0070] The condenser lens 401 condenses the light emitted by the first light source unit 402 and the second light source unit 403 and emits the light toward the light guide 231 .

[0071] The first light source unit 402 emits white light (normal light), which is visible light, under the control of the light source control unit 404, and supplies the white light as illumination light to the light guide 231. The first light source unit 402 is composed of a collimating lens, a white LED (Light Emitting Diode) lamp, and a driver.

[0072] Alternatively, a red LED, a green LED, and a blue LED may be used as the first light source unit 402 to emit light simultaneously, thereby supplying visible white light. Alternatively, the first light source unit 402 may be composed of a halogen lamp, a xenon lamp, or the like.

[0073] Under the control of the light source control unit 404 , the second light source unit 403 emits excitation light having a predetermined wavelength range and supplies the excitation light as illumination light to the light guide 231 .

[0074] Figure 3is a diagram showing the wavelength characteristics of the excitation light emitted by the second light source unit 403. Specifically, Figure 3 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents wavelength characteristics. Figure 3 In the middle, curve L V represents the wavelength characteristics of the excitation light emitted by the second light source unit 403. Figure 3 In the middle, curve L B Indicates the wavelength range of blue, curve L G Indicates the wavelength range of green, curve L R Indicates the wavelength range of red.

[0075] In this embodiment, if Figure 3 As shown, the second light source unit 403 emits excitation light with a central wavelength (peak wavelength) of 415 nm and a wavelength range of 400 nm to 430 nm. The second light source unit 403 is composed of a collimating lens, a semiconductor laser such as a violet LD (laser diode), and a driver.

[0076] Here, the transmission characteristics of the cut filter 203 will be described.

[0077] Figure 4 is a diagram showing the transmission characteristics of the cut filter 203. Specifically, Figure 4 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents wavelength characteristics. Figure 4 In the middle, curve L F The curve L represents the transmission characteristics of the cutoff filter 203. V represents the wavelength characteristics of the excitation light. Figure 4 In the middle, curve L NG This indicates the wavelength characteristics of fluorescence generated by irradiating the advanced glycation products produced by heat treatment of living tissue with excitation light. In this embodiment, heat treatment of living tissue refers to a state in which the laser light irradiated from the laser irradiation device 5 toward the stone reaches the living tissue, or a state in which the high-temperature stone irradiated by the laser light comes into contact with the living tissue.

[0078] In this embodiment, the cut-off filter 203 is as follows: Figure 4 As shown, the cut filter 203 blocks a portion of the excitation light reflected from the biological tissue in the observation area, while transmitting light in other wavelength ranges including fluorescent components. More specifically, the cut filter 203 blocks a portion of the light in the wavelength range shorter than 400 nm to 430 nm including the excitation light, while transmitting light in the wavelength range longer than 430 nm including the fluorescence generated by irradiating the advanced glycation products produced by heat treatment with the excitation light.

[0079] The light source control unit 404 is implemented using a processor (e.g., a hardware processor such as an FPGA or CPU) as a processing device and a memory (e.g., a temporary storage area) used by the processor. Furthermore, the light source control unit 404 controls the timing and duration of light emission of the first and second light source units 402 and 403, respectively, based on control data input from the control unit 409.

[0080] Under the control of the control unit 409 , the S / P conversion unit 405 performs serial / parallel conversion on the image data received from the endoscope 2 via the first signal line 232 , and outputs the result to the image processing unit 406 .

[0081] Furthermore, when the endoscope 2 outputs image data as an optical signal, an O / E converter that converts the optical signal into an electrical signal may be provided in place of the S / P converter 405. Furthermore, when the endoscope 2 transmits image data via wireless communication, a communication module capable of receiving wireless signals may be provided in place of the S / P converter 405.

[0082] The image processing unit 406 is implemented using a processor as a processing device, such as a GPU or FPGA, and a memory as a temporary storage area used by the processor. Under the control of the control unit 409, the image processing unit 406 performs predetermined image processing on the parallel image data input from the S / P conversion unit 405 and outputs the data to the display device 3. Examples of the predetermined image processing include demosaicing, white balance, gain adjustment, gamma correction, and format conversion.

[0083] The input unit 407 is configured using a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, etc., receives user operations from a user such as a surgeon, and outputs an operation signal corresponding to the user operation to the control unit 409 .

[0084] The recording unit 408 is configured using a recording medium such as a volatile memory, a nonvolatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory card. The recording unit 408 records data including various parameters required for the operation of the endoscope system 1. The recording unit 408 also includes a program recording unit 408a for recording various programs used to operate the endoscope system 1.

[0085] The control unit 409 corresponds to the processor involved in the present invention. This control unit 409 is implemented using the following components: a processor as a processing device, including hardware such as an FPGA or CPU; and a memory as a temporary storage area used by the processor. Furthermore, the control unit 409 comprehensively controls the various components that make up the endoscope system 1.

[0086] [Observation Principles in Observation Mode of Endoscope Systems]

[0087] Next, the observation principle in the observation mode of the endoscope system 1 will be described.

[0088] The following describes the fluorescence observation mode and the normal light observation mode in this order.

[0089] [Observation Principles in Fluorescence Observation Mode]

[0090] First, the principle of observation in the fluorescence observation mode will be described.

[0091] Figure 5 A diagram illustrating the principle of observation in the fluorescence observation mode.

[0092] like Figure 5 As shown in the graph G11 of FIG, first, the control device 4 irradiates the biological tissue O10 with the excitation light (central wavelength 415 nm) by making the second light source unit 403 emit light. Figure 5 As shown in the graph G12, at least the components of the excitation light reflected by the biological tissue O10 and the reflected light of the return light (hereinafter referred to as reflected light W10) are shielded by the cutoff filter 203 and the intensity is reduced. On the other hand, a part of the components on the long wavelength side longer than the shielded wavelength range is incident on the imaging element 204 without reducing the intensity.

[0093] More specifically, if Figure 5 As shown in the graph G12 of FIG. 1 , the cut filter 203 blocks most of the reflected light W10 in the wavelength range on the short wavelength side including the wavelength range of the excitation light incident on the G pixel of the imaging element 204, and transmits the wavelength range on the long wavelength side longer than the blocked wavelength range. Figure 5 As shown in graph G12, the cut filter 203 transmits fluorescence WF10 emitted by the end-glycation products themselves, resulting from the heat treatment of the biological tissue O10. Therefore, the fluorescence WF10 and the reflected light W10 of reduced intensity are incident on each of the R, G, and B pixels of the imaging element 204.

[0094] Here, the G pixel of the imaging element 204 is sensitive to the fluorescence WF10. Figure 5 The fluorescence characteristic curve L in the graph G12 NGAs shown, fluorescence is a minute reaction, so the output value corresponding to the fluorescence WF10 at the G pixel is a small value.

[0095] The image processing unit 406 then acquires image data (RAW data) from the imaging element 204 and performs image processing on the output values ​​of the G and B pixels contained in this image data to generate a fluorescence image. In this case, the output values ​​of the G pixels include fluorescence information corresponding to fluorescence WF10 emitted from the heat-treated region (end-glycation products) of the biological tissue O10 where the heat treatment was performed. Furthermore, the output values ​​of the B pixels include background information from the subject's biological tissue O10, including the heat-treated region. Displaying this fluorescence image on the display device 3 allows observation of the heat-treated region of the biological tissue O10.

[0096] [Observation principle in normal light observation mode]

[0097] Next, the principle of observation in the normal light observation mode will be described.

[0098] Figure 6 A diagram illustrating the principle of observation in the normal light observation mode.

[0099] like Figure 6 As shown in the diagram G21, the control device 4 first illuminates the biological tissue O10 with white light by causing the first light source unit 402 to emit light. In this case, a portion of the reflected light and the return light (hereinafter referred to as reflected light WR30, WG30, and WB30) reflected by the biological tissue O10 is shielded by the cut filter 203, and the remaining portion is incident on the imaging element 204. Specifically, Figure 6 As shown in graph G22, the cut filter 203 blocks reflected light in the wavelength range shorter than the wavelength range of the excitation light. Therefore, the blue wavelength component incident on the B pixel of the imaging element 204 is reduced compared to a state where the cut filter 203 is not provided.

[0100] The image processing unit 406 then acquires image data (RAW data) from the imaging element 204 and performs image processing on the output values ​​of the R, G, and B pixels contained in this image data to generate an observation image (white light image). In this case, the blue component contained in the image data is smaller than when the cut filter 203 is not configured. Therefore, the image processing unit 406 performs white balance adjustment processing to adjust the white balance so that the ratio of the red, green, and blue components is constant. Furthermore, by displaying this observation image (white light image) on the display device 3, a natural observation image (white light image) can be observed even when the cut filter 203 is configured.

[0101] [Control Method]

[0102] Next, a control method executed by the control device 4 will be described.

[0103] Figure 7 1 is a flowchart showing a control method executed by the control device 4 . Figures 8 to 10 is a diagram illustrating the control method. Specifically, Figure 8 The correlation between the fluorescence intensity of the advanced glycation products themselves in the biological tissue and the degree of invasion (depth and area) caused by heat treatment of the biological tissue (straight line L Y ). In addition, Figure 8 In the graph, the vertical axis represents the fluorescence intensity, and the horizontal axis represents the degree of invasion caused by heat treatment of biological tissue. Figure 9 3 is a diagram showing the fluorescent image (first fluorescent image F1 ) generated in step S3 . Figure 10 is a diagram showing the fluorescence image (second fluorescence image F2) generated in step S5. Figure 9 and Figure 10 In FIG. 1 , for the sake of convenience, illustration of stones captured in the first fluorescent image F1 and the second fluorescent image F2 is omitted.

[0104] The following describes the control method executed by the control device 4 during transurethral ureterolithotripsy. Specifically, the insertion portion 21 is inserted into the urinary tract, and the observation area of ​​the endoscope system 1 becomes an area encompassing the calculi within the urinary tract. Furthermore, the laser irradiation device 5 is inserted into the urinary tract from the insertion port 222 via the treatment instrument channel within the insertion portion 21, and is in a state capable of irradiating the calculi with laser light. Furthermore, the control unit 409 performs normal control of the irrigation device 6, filling the urinary tract with irrigation fluid from the insertion port 222 via the treatment instrument channel within the insertion portion 21, and entering a normal state in which the irrigation fluid is irrigated at a normal rate (discharging the irrigation fluid into the urinary tract while simultaneously draining the irrigation fluid from the urinary tract).

[0105] First, the control unit 409 switches the observation mode to the fluorescence observation mode in response to an operation of “switching the observation mode of the endoscope system 1 to the fluorescence observation mode” performed by a user such as an operator on the operation member 223 (step S1 ).

[0106] After step S1 , the control unit 409 controls the light source control unit 404 to start irradiating the excitation light from the second light source unit 403 (step S2 ).

[0107] After step S2, the image processing unit 406 generates a fluorescence image (a first fluorescence image F1 ( Figure 9 ))(Step S3).

[0108] After step S3, the control unit 409 controls the operation of the laser irradiation device 5 to irradiate the laser beam (step S4). The stones in the urinary tract are broken up by the irradiation of the laser beam.

[0109] After step S4, the image processing unit 406 generates a fluorescence image (a second fluorescence image F2 ( Figure 10 ))(Step S5).

[0110] In addition, there is a correlation between the fluorescence intensity of the final glycation product itself generated by heat treatment of biological tissue and the degree of invasion (degree of thermal denaturation) caused by heat treatment of the biological tissue. Figure 8 Specifically, Figure 8 The straight line L Y As shown, the higher the degree of thermal denaturation (the greater the degree of invasion caused by heat treatment of biological tissue), the higher the fluorescence intensity.

[0111] Here, in Figure 10 In the second fluorescent image F2 shown, the shaded area Ar1 is a thermally denatured area where thermal denaturation has increased due to the thermal treatment of the living tissue. In other words, the thermally denatured area Ar1 corresponds to the area where the laser light irradiated on the stone in step S4 was irradiated onto the living tissue and thus thermally treated, or the area where the high-temperature stone irradiated by the laser light came into contact with the living tissue and thus thermally treated. Figure 9 The first fluorescent image F1 shown shows the same observation area as that in the second fluorescent image F2, and was captured before laser irradiation in step S4. Therefore, in the first fluorescent image F1, there is no region with increased thermal denaturation corresponding to the thermally denatured region Ar1 in the second fluorescent image F2. In other words, the first fluorescent image F1 corresponds to the first captured image of the present invention. Furthermore, the second fluorescent image F2 corresponds to the second captured image of the present invention.

[0112] Furthermore, depending on the degree of change in the state of thermal denaturation, there is a possibility that the effect may affect living tissues.

[0113] Therefore, in this embodiment, the effect of laser irradiation on living tissue is appropriately addressed by executing steps S6 and S7 described below.

[0114] Specifically, after step S5, the control unit 409 determines the change in the state of thermal denaturation based on the first fluorescent image F1 and the second fluorescent image F2 (step S6). In this embodiment, the control unit 409 determines the change in the state of thermal denaturation based on the difference in fluorescence intensity between corresponding pixels in the first fluorescent image F1 and the second fluorescent image F2.

[0115] Here, as the fluorescence intensity used in step S6 , at least the g value of the pixel values ​​(r, g, b) of each pixel in the demosaiced first fluorescent image F1 or the second fluorescent image F2 or the luminance value corresponding to the Y signal (luminance signal) can be exemplified.

[0116] Furthermore, while the change in the state of thermal denaturation is determined in step S6 based on the first and second fluorescent images F1 and F2, this is not limiting. For example, the change in the state of thermal denaturation may be determined based on two pieces of image data (captured images according to the present invention) that precede and follow each other in time before image processing by the image processing unit 406. In this case, the output value of the G pixel in the imaging element 204 can be used as an example of fluorescence intensity.

[0117] After step S6, the control unit 409 executes perfusion control (operation control of the perfusion device 6) based on the result of the determination of the state change of thermal denaturation in step S6 (step S7). Thereafter, the control device 4 returns to step S3.

[0118] Specifically, the perfusion control performed in step S7 includes the following examples (1) to (3). Furthermore, the state change amount described below indicates the extent of the region where the state change has occurred due to thermal denaturation, or the intensity of the thermal denaturation in the region where the state change has occurred due to thermal denaturation. For example, the state change amount determined based on the first fluorescent image F1 and the second fluorescent image F2 indicates the extent of the thermal denaturation region Ar1, which is the region where the state change has occurred due to thermal denaturation, or the intensity of the fluorescence intensity in the thermal denaturation region Ar1.

[0119] (1) When the control unit 409 determines in step S6 that the state change amount indicating the state change of thermal denaturation is within the first state change amount, it continues normal control of the perfusion device 6 and maintains the normal state of perfusion of the perfusion fluid at a normal speed.

[0120] In the example of the first fluorescent image F1 and the second fluorescent image F2 , the perfusion control of (1) is executed when the extent of the thermally denatured region Ar1 or the fluorescence intensity in the thermally denatured region Ar1 is within the first state change amount.

[0121] (2) When the control unit 409 determines in step S6 that the state change amount indicating the state change of thermal denaturation exceeds the first state change amount and is within a second state change amount that is larger than the first state change amount, the control unit 409 sends a control signal to the perfusion device 6 to change the perfusion control of the perfusion device 6 from the normal control to the first control. The first control is a control that increases the perfusion speed of the perfusion fluid from the normal speed to a first speed that is higher than the normal speed during a first time period. Furthermore, when the first time period has elapsed, the control unit 409 sends a control signal to the perfusion device 6 to return the perfusion control of the perfusion device 6 from the first control to the normal control.

[0122] In the example of the first fluorescent image F1 and the second fluorescent image F2, the perfusion control of (2) is executed when the extent of the thermally denatured region Ar1 or the fluorescence intensity in the thermally denatured region Ar1 exceeds the first state change amount and is within the second state change amount.

[0123] (3) If the control unit 409 determines in step S6 that the state change amount indicating the state change of thermal denaturation exceeds the second state change amount, the control unit 409 sends a control signal to the perfusion device 6 to change the perfusion control of the perfusion device 6 from the normal control to the second control. The second control is a control that increases the perfusion speed of the perfusion fluid from the normal speed to the first speed during the second time period that is longer than the first time period. Furthermore, if the second time period has elapsed, the control unit 409 sends a control signal to the perfusion device 6 to return the perfusion control of the perfusion device 6 from the second control to the normal control.

[0124] In the example of the first fluorescent image F1 and the second fluorescent image F2 , the perfusion control of (3) is executed when the extent of the thermally denatured region Ar1 or the intensity of the fluorescence in the thermally denatured region Ar1 exceeds the second state change amount.

[0125] In addition, the following perfusion control (4) may be executed instead of the above perfusion control (2), and the following perfusion control (5) may be executed instead of the above perfusion control (3).

[0126] (4) When the control unit 409 determines in step S6 that the state change amount indicating the state change of thermal denaturation exceeds the first state change amount and is within the second state change amount that is greater than the first state change amount, the control unit 409 sends a control signal to the perfusion device 6 to change the perfusion control of the perfusion device 6 from the normal control to the third control. The third control is a control that increases the perfusion speed of the perfusion fluid from the normal speed to the second speed during the third time period. Furthermore, when the third time period has elapsed, the control unit 409 sends a control signal to the perfusion device 6 to return the perfusion control of the perfusion device 6 from the third control to the normal control.

[0127] (5) When the control unit 409 determines in step S6 that the state change amount indicating the state change of thermal denaturation exceeds the second state change amount, the control unit 409 sends a control signal to the perfusion device 6 to change the perfusion control of the perfusion device 6 from the normal control to the fourth control. The fourth control is a control for increasing the perfusion speed of the perfusion fluid from the normal speed to a third speed higher than the second speed during the third time period. Furthermore, when the third time period has elapsed, the control unit 409 sends a control signal to the perfusion device 6 to return the perfusion control of the perfusion device 6 from the fourth control to the normal control.

[0128] According to the present embodiment described above, the following effects are achieved.

[0129] In the control device 4 according to the present embodiment, the control unit 409 determines a change in the state of thermal denaturation based on the fluorescence image, and changes the perfusion control of the perfusion device 6 based on the determination result of the change in the state of thermal denaturation.

[0130] Therefore, during transurethral ureterolithotripsy, when laser light irradiating a stone hits living tissue, or when a high-temperature stone irradiated by the laser light comes into contact with living tissue, and a change in the state of thermal denaturation of the living tissue occurs, perfusion control such as (2) to (5) above can be performed. The living tissue that has undergone a change in state of thermal denaturation is then immediately cooled by the perfusion fluid based on the perfusion control. Therefore, according to the control device 4 of this embodiment, perfusion can be appropriately controlled in response to the effect of laser irradiation on the living tissue.

[0131] In the control device 4 according to the present embodiment, the control unit 409 determines a change in the state of thermal denaturation based on the difference in fluorescence intensity between corresponding pixels in the first and second fluorescence images F1 and F2, which are temporally preceding and following each other. Furthermore, the control unit 409 performs the perfusion control described in (1) to (5) above based on the extent of the region indicating a change in the state of thermal denaturation or the amount of change in the intensity of thermal denaturation in the region indicating a change in the state of thermal denaturation.

[0132] Therefore, when cooling of living tissue is required, the perfusion control described in (2) to (5) can be performed, and the living tissue can be appropriately cooled.

[0133] (Other embodiments)

[0134] Although the modes for carrying out the present invention have been described above, the present invention should not be limited to the above-described embodiments.

[0135] In the above-described embodiment, the medical device according to the present invention is mounted on an endoscope system used for transurethral ureterolithotripsy. However, the present invention is not limited thereto and may be mounted on an endoscope system used for other surgeries.

[0136] In the above-described embodiment, the medical device according to the present invention is mounted on an endoscope system using a flexible endoscope. However, the present invention is not limited thereto and may be mounted on an endoscope system using a rigid endoscope or an endoscope system using a medical surgical robot.

[0137] In the above embodiment, the control unit 409 may also function as a learning unit of the learning device according to the present invention. In this case, the control device 4 corresponds to the learning device according to the present invention.

[0138] Specifically, the control unit 409 generates a learned model by performing machine learning using training data. In this training data, a fluorescence image obtained by capturing fluorescence generated from living tissue by irradiating the living tissue with excitation light is used as input data, and information corresponding to a control signal for controlling the operation of the perfusion device 6 based on a change in the state of thermal denaturation extracted from the fluorescence image is used as output data.

[0139] Here, the learned model is composed of a neural network with one or more nodes in each layer. The type of machine learning is not particularly limited. For example, training data and learning data corresponding to a plurality of fluorescence images of a subject and information corresponding to a control signal for controlling the operation of the perfusion device 6 based on changes in the state of thermal denaturation extracted from the plurality of fluorescence images can be prepared. This training data and learning data can then be input into a computational model based on a multi-layer neural network to perform learning. Furthermore, as machine learning techniques, techniques based on multi-layer neural networks such as CNN (Convolutional Neural Network) and 3D-CNN (Deep Neural Network) can be used. Furthermore, techniques based on recurrent neural networks (RNN) or LSTM (Long Short-Term Memory Units) that are extensions of RNNs can also be used. Furthermore, a learning unit of a learning device separate from the control device 4 can perform these functions.

[0140] Description of Reference Numerals

[0141] 1: Endoscope system; 2: Endoscope; 3: Display device; 4: Control device; 5: Laser irradiation device; 6: Irrigation device; 21: Insertion portion; 22: Operation portion; 23: Universal cable; 24: Distal end portion; 25: Bend portion; 26: Flexible tube portion; 27: First connector portion; 27a: Cable; 28: Second connector portion; 201: Illumination optical system; 202: Imaging optical system; 203: Cutoff filter; 204: Imaging element; 205: A / D converter; 206: P / S converter; 207: Image recording unit; 208: Image control unit; 221 : Bending knob; 222: Insertion port; 223: Operating member; 231: Light guide; 232: First signal line; 233: Second signal line; 401: Condensing lens; 402: First light source unit; 403: Second light source unit; 404: Light source control unit; 405: S / P conversion unit; 406: Image processing unit; 407: Input unit; 408: Recording unit; 408a: Program recording unit; 409: Control unit; Ar1: Thermal denaturation area; F1: First fluorescent image; F2: Second fluorescent image; G11 to G13, G21 to G23: Graphs; L1: Optical axis; L B , L G , L R , L V : curve; L F : Transmission characteristics of the cutoff filter; L NG : wavelength characteristics of fluorescence; L Y : straight line; O10: biological tissue; W10, WB30, WG30, WR30: reflected light; WF10: fluorescence.

Claims

1. A medical device comprising a processor for processing a captured image of fluorescence generated from living tissue by irradiating the living tissue with excitation light, wherein: The processor performs the following processing: determining a change in the state of thermal denaturation based on the captured image; and Based on the determination result of the change in the state of the thermal denaturation, a control signal for controlling the operation of the perfusion device is sent to the perfusion device that performs perfusion of the perfusion fluid.

2. The medical device according to claim 1, wherein The processor transmits the control signal for increasing the perfusion speed of the perfusion solution to the perfusion device based on the determination result of the change in the state of the thermal denaturation.

3. The medical device according to claim 1, wherein The processor transmits the control signal for increasing the perfusion rate of the perfusion solution during a first period of time to the perfusion device based on the determination result of the change in the state of the thermal denaturation.

4. The medical device according to claim 1, wherein The processor determines a change in the state of the thermal denaturation based on the two captured images that are temporally adjacent to each other.

5. The medical device according to claim 4, wherein The two captured images that are temporally preceding and following each other are a first captured image captured before laser light is irradiated and a second captured image captured after laser light is irradiated.

6. The medical device according to claim 4, wherein: The processor determines a change in the state of the thermal denaturation based on a difference in fluorescence intensity between corresponding pixels in the two captured images that are temporally preceding and following each other.

7. The medical device according to claim 1, wherein The processor transmits the control signal for increasing the perfusion speed of the perfusion solution to the first speed during a first time period to the perfusion device when it is determined that the state change amount indicating the state change of the thermal denaturation exceeds a first state change amount and is within a second state change amount that is larger than the first state change amount.

8. The medical device according to claim 7, wherein: When the processor determines that the state change amount indicating the state change of the thermal denaturation exceeds the second state change amount, the processor transmits the control signal to the perfusion device for increasing the perfusion speed of the perfusion solution to the first speed for a second time period longer than the first time period.

9. The medical device according to claim 1, wherein When the processor determines that the state change amount indicating the state change of the thermal denaturation exceeds a first state change amount and is within a second state change amount that is greater than the first state change amount, the processor transmits the control signal for increasing the perfusion speed of the perfusion solution to the second speed during a third time period to the perfusion device.

10. The medical device according to claim 9, wherein When the processor determines that the state change amount indicating the state change of the thermal denaturation exceeds the second state change amount, the processor transmits the control signal to the perfusion device for increasing the perfusion speed of the perfusion solution to a third speed higher than the second speed during the third time period.

11. The medical device according to claim 7, wherein: The state change amount indicating the state change of the thermal denaturation indicates the extent of a region where the state change of the thermal denaturation has occurred or the intensity of the thermal denaturation in the region where the state change of the thermal denaturation has occurred.

12. The medical device according to claim 1, wherein The fluorescence is generated from the terminal glycation products produced by heat treatment of the biological tissue.

13. An endoscope system comprising: a light source device for irradiating excitation light; an endoscope that is insertable into a subject and outputs a captured image of fluorescence generated from living tissue within the subject by irradiating the living tissue with the excitation light; and A medical device having a processor for processing the captured image, wherein: The processor performs the following processing: determining a change in the state of thermal denaturation based on the captured image; and Based on the determination result of the change in the state of the thermal denaturation, a control signal for controlling the operation of the perfusion device is sent to the perfusion device that performs perfusion of the perfusion fluid.

14. A control method executed by a medical device, wherein: determining a change in the state of thermal denaturation based on a captured image of fluorescence generated from the living tissue by irradiating the living tissue with excitation light, Based on the determination result of the change in the state of the thermal denaturation, a control signal for controlling the operation of the perfusion device is sent to the perfusion device that performs perfusion of the perfusion fluid.

15. A control program for causing a medical device to execute a control program, wherein: The control program instructs the medical device to execute the following processing: determining a change in the state of thermal denaturation based on a captured image of fluorescence generated from the living tissue by irradiating the living tissue with excitation light; and Based on the determination result of the change in the state of the thermal denaturation, a control signal for controlling the operation of the perfusion device is sent to the perfusion device that performs perfusion of the perfusion fluid.

16. A learning device, wherein: The learning device includes a learning unit that generates a learned model by performing machine learning using training data, wherein the training data includes, as input data, a fluorescence image obtained by capturing fluorescence generated from living tissue by irradiating the living tissue with excitation light, and output data including, as output data, information corresponding to a control signal for controlling the operation of a perfusion device that performs perfusion of a perfusion fluid based on a change in a state of thermal denaturation extracted from the fluorescence image.

Images