Medical device, endoscope system, control method, and control program

Abstract

The medical device (4) is provided with processors (406, 409) that process captured images in which fluorescent light generated from biological tissue by irradiating the biological tissue with excitation light has been captured. A processor (406, 409) performs: a process for acquiring a tissue image including a site to be thermally treated in biological tissue; acquiring range information indicating a treatment target range of thermal treatment in the tissue image; determining a state of thermal denaturation caused by the thermal treatment on the basis of the captured image; and causing a notification unit (3) to notify thermal denaturation information indicating the state of thermal denaturation within the treatment target range.

Description

Technical Field

[0001] The present invention relates to a medical device, an endoscope system, a control method, and a control program. Background Art

[0002] Conventionally, there is known a technique for visualizing the state of thermal denaturation of living tissue when the living tissue is subjected to heat treatment by an energy device or the like (for example, see Patent Document 1).

[0003] The technology described in Patent Document 1 visualizes the state of thermal denaturation of biological tissue based on a captured image of fluorescence generated from biological tissue by irradiating the tissue with excitation light. Specifically, the technology described in Patent Document 1 displays regions of all pixels in the captured image where the fluorescence intensity exceeds a predetermined intensity as regions of high thermal denaturation.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: International Publication No. 2020 / 054723 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] Furthermore, in surgical procedures such as transesophageal endoscopic myotomy (POEM), the target area for thermal treatment of biological tissue using an energy device is relatively large. Therefore, it is difficult for the operator to determine whether the thermal treatment has been properly performed throughout the entire target area.

[0009] Here, when the technology described in Patent Document 1 is applied, the state of thermal denaturation in biological tissue can be visualized, but it is difficult for a user such as a surgeon to determine whether heat treatment has been appropriately performed over the entire wide treatment target area.

[0010] Therefore, a technology that enables a user such as a surgeon to determine whether heat treatment has been appropriately performed over the entire wide treatment target area and that can improve convenience is desired.

[0011] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide a medical device, an endoscope system, a control method, and a control program that can improve convenience.

[0012] Solutions for solving problems

[0013] In order to solve the above-mentioned problems and achieve the purpose, the medical device involved in the present invention includes a processor, which processes a camera image that captures fluorescence, wherein the fluorescence is fluorescence generated from the living tissue by irradiating the living tissue with excitation light, and the processor performs the following processing: obtaining a tissue image including a target area for heat treatment in the living tissue; obtaining range information indicating a treatment target range of the heat treatment in the tissue image; judging a state of thermal denaturation caused by the heat treatment based on the camera image; and causing a notification unit to notify thermal denaturation information indicating the state of the thermal denaturation within the treatment target range.

[0014] The endoscope system involved in the present invention comprises: a light source device that irradiates excitation light; an endoscope that can be inserted into a subject and outputs a video image that captures fluorescence, wherein the fluorescence is fluorescence generated from the biological tissue in the subject by irradiating the biological tissue in the subject with excitation light; and a medical device having a processor that processes the video image, wherein the processor performs the following processing: obtaining a tissue image including a target area for heat treatment in the biological tissue; obtaining range information representing a treatment target range of the heat treatment in the tissue image; judging a state of thermal denaturation caused by the heat treatment based on the video image; and causing a notification unit to notify thermal denaturation information representing the state of the thermal denaturation within the treatment target range.

[0015] The control method involved in the present invention is a control method performed by a medical device, wherein a tissue image including a target site for heat treatment in living tissue is obtained, range information indicating a treatment target range of the heat treatment in the tissue image is obtained, a state of thermal denaturation caused by the heat treatment is determined based on a camera image in which fluorescence is captured, the fluorescence being generated from the living tissue by irradiating the living tissue with excitation light, and a notification unit is caused to notify the patient of thermal denaturation information indicating the state of thermal denaturation within the treatment target range.

[0016] The control program involved in the present invention is a control program executed by a medical device, wherein the control program instructs the medical device to perform the following processing: obtaining a tissue image including a target area for heat treatment in biological tissue; obtaining range information indicating a treatment target range of the heat treatment in the tissue image; judging a state of thermal denaturation caused by the heat treatment based on a camera image in which fluorescence is captured, the fluorescence being fluorescence generated from the biological tissue by irradiating the biological tissue with excitation light; and causing a notification unit to notify thermal denaturation information indicating the state of the thermal denaturation within the treatment target range.

[0017] Effects of the Invention

[0018] According to the medical device, endoscope system, control method, and control program according to the present invention, convenience can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram showing the overall configuration of an endoscope system according to an embodiment.

[0020] Figure 2 This is a block diagram showing the functional configuration of a main portion of an endoscope system according to an embodiment.

[0021] Figure 3 : is a diagram showing the wavelength characteristics of the excitation light emitted by the second light source unit.

[0022] Figure 4 It is a graph showing the transmission characteristics of the cut filter.

[0023] Figure 5 A diagram illustrating the principle of observation in the fluorescence observation mode.

[0024] Figure 6 This is a diagram explaining the observation principle in the normal light observation mode.

[0025] Figure 7 This is a flowchart showing a control method before heat treatment.

[0026] Figure 8 This is a diagram explaining the control method before heat treatment.

[0027] Figure 9 This is a diagram explaining the control method before heat treatment.

[0028] Figure 10 This is a flowchart showing a control method when performing heat treatment.

[0029] Figure 11 This is a diagram explaining a control method when performing heat treatment.

[0030] Figure 12 This is a diagram explaining a control method when performing heat treatment.

[0031] Figure 13 This is a diagram explaining a control method when performing heat treatment.

[0032] Figure 14 This is a diagram explaining a control method when performing heat treatment.

[0033] Figure 15 This is a diagram illustrating a control method during heat treatment. DETAILED DESCRIPTION

[0034] Below, the mode for implementing the present invention (hereinafter referred to as embodiment) is described with reference to the accompanying drawings. In addition, the present invention is not limited to the embodiment described below. In addition, in the description of the accompanying drawings, the same reference numerals are used for the same parts.

[0035] [Overall structure of the endoscope system]

[0036] Figure 1 1 is a diagram showing the overall configuration of an endoscope system 1 according to the embodiment.

[0037] The endoscope system 1 of this embodiment is used for transesophageal endoscopic myotomy (POEM). Specifically, during POEM, the insertion portion 21 of the endoscope 2 is inserted into the esophagus of a subject through the oral cavity to image the subject's interior. An image based on the captured image data is then displayed on a display device 3. The operator then incises (heat-treats) the overdeveloped esophageal and cardiac muscles using a heat treatment device 5 while confirming the displayed image.

[0038] like Figure 1 As shown, the endoscope system 1 includes an endoscope 2 , a display device 3 , a control device 4 , and a thermal treatment device 5 .

[0039] The endoscope 2 generates image data (RAW data) captured inside the subject and outputs the image data to the control device 4. Figure 1 As shown, the endoscope 2 includes an insertion portion 21 , an operation portion 22 , and a universal cord 23 .

[0040] At least a portion of the insertion portion 21 has flexibility, and the insertion portion 21 is inserted into the subject. Figure 1 As shown, the insertion portion 21 includes a front end portion 24 arranged at the front end of the insertion portion 21, a bending portion 25 connected to the base end side (operation portion 22 side) of the front end portion 24 and configured to be bendable, and a long flexible tube portion 26 connected to the base end side of the bending portion 25 and having flexibility.

[0041] The operation part 22 is connected to the base end portion of the insertion part 21. The operation part 22 receives various operations for the endoscope 2. Figure 1 As shown, the operating portion 22 is provided with a bending knob 221 , an insertion port 222 , and a plurality of operating members 223 .

[0042] The bending knob 221 is configured to be rotatable in response to user operations performed by a surgeon or other user. Rotation of the bending knob 221 activates a bending mechanism (not shown) such as a metal or resin wire disposed within the insertion portion 21. As a result, the bending portion 25 bends.

[0043] The insertion port 222 communicates with a treatment instrument channel (not shown) that is a conduit extending from the distal end of the insertion portion 21 and is used to insert a treatment instrument or the like into the treatment instrument channel from the outside of the endoscope 2 .

[0044] The plurality of operating members 223 are composed of buttons and the like for receiving various operations performed by a user such as a surgeon, and output operation signals corresponding to the various operations to the control device 4 via the universal cable 23. Examples of the various operations include operations for switching the observation mode of the endoscope system 1 to a normal light observation mode, a fluorescence observation mode, or a special observation mode.

[0045] The universal cable 23 extends from the operation portion 22 in a direction different from the extension direction of the insertion portion 21 and is provided with a light guide 231 composed of an optical fiber or the like (see Figure 2 ), a first signal line 232 for transmitting the above-mentioned image data (refer to Figure 2 ), and a second signal line 233 (see Figure 2 ) and other cables. Moreover, Figure 1 As shown, a first connector portion 27 , a second connector portion 28 , and a cable 27 a are provided at a base end of the universal cable 23 .

[0046] The first connector portion 27 is detachably connected to the control device 4 .

[0047] The cable 27 a is a spiral cable extending from the first connector portion 27 .

[0048] The second connector portion 28 is provided at the front end of the cable 27 a and is detachably connected to the control device 4 .

[0049] The display device 3 is composed of a display monitor such as liquid crystal or organic EL (Electro Luminescence), and displays images based on image data processed by the control device 4 and various information related to the endoscope system 1 under the control of the control device 4 .

[0050] The control device 4 corresponds to the medical device of the present invention. This control device 4 is implemented using a processor, which is a processing device comprising hardware such as a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or a CPU (Central Processing Unit), and a memory serving as a temporary storage area for the processor. Furthermore, the control device 4 centrally controls the operation of each component of the endoscope system 1 according to a program stored in the memory.

[0051] The heat treatment device 5 is, for example, an energy device such as a high-frequency electric knife that applies high-frequency current to biological tissue to perform heat treatment on the tissue, or a laser irradiation device that irradiates biological tissue with high-power infrared laser light to perform heat treatment on the tissue. Specifically, the heat treatment device 5 is inserted into the esophagus through the insertion port 222 via the treatment instrument channel within the insertion portion 21. The heat treatment device 5 then performs heat treatment on the esophagus and cardiac muscles in response to user operations performed by a user such as a surgeon.

[0052] [Functional structure of main parts of endoscope system]

[0053] Next, the functional configuration of the main parts of the endoscope system 1 will be described.

[0054] Figure 2 1 is a block diagram showing the functional configuration of the main parts of the endoscope system 1 .

[0055] Next, the endoscope 2 and the control device 4 will be described in sequence.

[0056] [Structure of an endoscope]

[0057] First, the structure of the endoscope 2 will be described.

[0058] like Figure 2 As shown, the endoscope 2 includes an illumination optical system 201 , an imaging optical system 202 , a cut filter 203 , an imaging element 204 , an A / D converter 205 , a P / S converter 206 , an imaging and recording unit 207 , an imaging control unit 208 , and a sensor unit 209 .

[0059] Here, the illumination optical system 201 , imaging optical system 202 , cut filter 203 , imaging element 204 , A / D converter 205 , P / S converter 206 , imaging and recording unit 207 , imaging control unit 208 , and sensor unit 209 are each arranged in the distal end portion 24 .

[0060] The illumination optical system 201 is composed of one or more lenses and the like, and irradiates the illumination light supplied from the light guide 231 toward the subject.

[0061] The imaging optical system 202 is composed of one or more lenses, and forms an image of the subject on the light receiving surface of the imaging element 204 by converging reflected light from the subject, return light from the subject, fluorescence emitted by the subject, and the like.

[0062] The cut filter 203 is disposed between the imaging optical system 202 and the imaging element 204 on the optical axis L1 of the imaging optical system 202. The cut filter 203 blocks light in a predetermined wavelength range and transmits light other than the wavelength range.

[0063] Note that the transmission characteristics of the cutoff filter 203 will be described in the “Configuration of the Control Device” to be described later.

[0064] The imaging element 204 is constructed using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor, in which one of the color filters forming a Bayer array (RGGB) is arranged in each of a plurality of pixels arranged in a two-dimensional matrix. Furthermore, under the control of the imaging control unit 208, the imaging element 204 receives the subject image formed by the imaging optical system 202 and passed through the cutoff filter 203, and performs photoelectric conversion on this subject image to generate a captured image (analog signal). In this embodiment, the imaging element 204 is constructed by integrating this image sensor with a Time of Flight (TOF) sensor that acquires subject distance information (hereinafter referred to as depth map information) using a TOF method. This depth map information is information obtained by detecting the subject distance from the position of the imaging element 204 (the position of the tip 24) to the corresponding position on the observation object corresponding to the pixel position in the captured image, for each pixel position.

[0065] Furthermore, the structure for generating depth map information is not limited to the aforementioned TOF sensor, and a phase difference sensor, a stereo camera, or the like may also be used.

[0066] Hereinafter, the depth map information and the captured image are described together as image data.

[0067] Then, the imaging element 204 outputs the image data to the A / D conversion unit 205 .

[0068] The A / D conversion unit 205 is configured using an A / D conversion circuit and the like, and performs A / D conversion processing on analog image data input from the imaging element 204 under the control of the imaging control unit 208 , and outputs the processed image data to the P / S conversion unit 206 .

[0069] The P / S conversion unit 206 is constructed using a P / S conversion circuit, etc., and under the control of the camera control unit 208, performs parallel / serial conversion on the digital image data input from the A / D conversion unit 205, and outputs the converted image data to the control device 4 via the first signal line 232.

[0070] Alternatively, an E / O converter that converts image data into an optical signal may be provided in place of the P / S converter 206, and the image data may be output via the optical signal to the controller 4. Alternatively, the image data may be transmitted to the controller 4 via wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0071] The image recording unit 207 is composed of a nonvolatile memory or a volatile memory and records various information related to the endoscope 2 (for example, pixel information of the image sensor 204 and the characteristics of the cutoff filter 203). In addition, the image recording unit 207 records various setting data and control parameters transmitted from the control device 4 via the second signal line 233.

[0072] The imaging control unit 208 is implemented using a TG (Timing Generator), a processor (a processing device including hardware such as a CPU), and a memory serving as a temporary storage area for the processor. Furthermore, the imaging control unit 208 controls the operation of the imaging element 204, the A / D converter 205, and the P / S converter 206 based on setting data received from the control device 4 via the second signal line 233.

[0073] The sensor unit 209 is a sensor for calculating the position of the distal end (distal end portion 24) of the insertion unit 21 and the direction (the imaging field of the distal end) of the insertion unit 21. In this embodiment, the sensor unit 209 is composed of a plurality of magnetic coils that generate magnetic force.

[0074] [Structure of control device]

[0075] Next, the configuration of the control device 4 will be described.

[0076] like Figure 2As shown, the control device 4 includes a condenser lens 401, a first light source unit 402, a second light source unit 403, a light source control unit 404, an S / P conversion unit 405, an image processing unit 406, an input unit 407, a recording unit 408, a control unit 409, a communication unit 410 and a receiving unit 411.

[0077] The condenser lens 401 condenses the light emitted from the first light source unit 402 and the second light source unit 403 and emits the light toward the light guide 231 .

[0078] The first light source unit 402 emits white light (normal light) as visible light under the control of the light source control unit 404 and supplies the white light as illumination light to the light guide 231. The first light source unit 402 is configured using a collimating lens, a white LED (Light Emitting Diode) lamp, and a driver.

[0079] Alternatively, the first light source unit 402 may provide white light as visible light by using a red LED lamp, a green LED lamp, and a blue LED lamp to emit light simultaneously. Alternatively, the first light source unit 402 may be composed of a halogen lamp, a xenon lamp, or the like.

[0080] Under the control of the light source control unit 404 , the second light source unit 403 emits excitation light having a predetermined wavelength range and supplies the excitation light as illumination light to the light guide 231 .

[0081] Figure 3 is a diagram showing the wavelength characteristics of the excitation light emitted by the second light source unit 403. Specifically, Figure 3 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents wavelength characteristics. Figure 3 In the middle, curve L V represents the wavelength characteristics of the excitation light emitted from the second light source unit 403. Figure 3 In the middle, curve L B Indicates the wavelength range of blue, curve L G Indicates the wavelength range of green, curve L R Indicates the wavelength range of red.

[0082] In this embodiment, if Figure 3 As shown, the second light source unit 403 emits excitation light with a center wavelength (peak wavelength) of 415 nm and a wavelength range of 400 nm to 430 nm. The second light source unit 403 is composed of a collimating lens, a semiconductor laser such as a violet LD (laser diode), and a driver.

[0083] Here, the transmission characteristics of the cutoff filter 203 will be described.

[0084] Figure 4 is a diagram showing the transmission characteristics of the cutoff filter 203. Specifically, Figure 4 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents wavelength characteristics. Figure 4 In the middle, curve L F The curve L represents the transmission characteristics of the cutoff filter 203. V represents the wavelength characteristics of the excitation light. Figure 4 In the middle, curve L NG This indicates the wavelength characteristics of fluorescence generated by irradiating excitation light onto advanced glycation end products produced by heat treatment of biological tissue.

[0085] In this embodiment, if Figure 4 As shown, the cutoff filter 203 blocks a portion of the excitation light reflected from the biological tissue in the observation area, while transmitting light in other wavelength ranges including fluorescent components. More specifically, the cutoff filter 203 blocks a portion of light in the short wavelength range of less than 400 nm to 430 nm, including the excitation light, while transmitting light in the long wavelength range longer than 430 nm, including fluorescence generated by irradiating the excitation light onto advanced glycation end products produced by heat treatment.

[0086] The light source control unit 404 is implemented using a processor as a processing device having hardware such as an FPGA or a CPU, and a memory as a temporary storage area used by the processor. Furthermore, the light source control unit 404 controls the light emission timing and light emission duration of each of the first light source unit 402 and the second light source unit 403 based on control data input from the control unit 409.

[0087] Under the control of the control unit 409 , the S / P conversion unit 405 performs serial / parallel conversion on the image data received from the endoscope 2 via the first signal line 232 , and outputs the converted image data to the image processing unit 406 .

[0088] Furthermore, when the endoscope 2 outputs image data in the form of an optical signal, an O / E converter that converts the optical signal into an electrical signal may be provided in place of the S / P converter 405. Furthermore, when the endoscope 2 transmits image data via wireless communication, a communication module capable of receiving wireless signals may be provided in place of the S / P converter 405.

[0089] The image processing unit 406 corresponds to the processor involved in the present invention. This image processing unit 406 is implemented using a processor, which is a processing device comprising hardware such as a GPU or FPGA, and a memory serving as a temporary storage area for the processor. Furthermore, under the control of the control unit 409, the image processing unit 406 performs predetermined image processing on the parallel image data input from the S / P conversion unit 405 and outputs the processed image data to the display device 3. Examples of the predetermined image processing include demosaicing, white balance, gain adjustment, gamma correction, and format conversion.

[0090] The input unit 407 is configured using a mouse, foot switch, keyboard, buttons, switches, and touch panel, and receives user operations performed by a user such as a surgeon and outputs operation signals corresponding to the user operations to the control unit 409 .

[0091] The recording unit 408 is configured using a recording medium such as a volatile memory, a nonvolatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory card. Furthermore, the recording unit 408 records data including various parameters required for the operation of the endoscope system 1. The recording unit 408 also includes a program recording unit 408a for recording various programs used to operate the endoscope system 1, and a learning model recording unit 408b described below.

[0092] The learning model recording unit 408b records a learning model used in image recognition performed by the control unit 409. This learning model is, for example, a model generated by machine learning using artificial intelligence (AI).

[0093] Specifically, the learning model is obtained by using image data of a treatment target area during thermal treatment in transesophageal endoscopic myotomy (POEM) as training data and performing machine learning (eg, deep learning) on ​​the treatment target area based on the training data.

[0094] The control unit 409 corresponds to the processor involved in the present invention. The control unit 409 is implemented using a processor as a processing device having hardware such as an FPGA or a CPU, and a memory as a temporary storage area used by the processor. Furthermore, the control unit 409 comprehensively controls the various components that constitute the endoscope system 1.

[0095] Communication unit 410 is an interface for communicating various data with an external tomography device (not shown), such as a CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) device, according to a prescribed protocol. Furthermore, under the control of control unit 409, communication unit 410 acquires tissue images, which are tomographic images captured by the tomography device.

[0096] Furthermore, communication between the communication unit 410 and the external tomography apparatus may be wireless or wired. Alternatively, a configuration may be employed in which tomographic images (tissue images) captured by the tomography apparatus are stored in advance on a server, etc., and the communication unit 410 obtains the tomographic images (tissue images) from the server.

[0097] The receiving unit 411 receives the magnetic force generated by the sensor unit 209. The receiving unit 411 then outputs a signal corresponding to the received magnetic force to the control unit 409.

[0098] [Observation principle of the endoscope system in observation mode]

[0099] Next, the observation principle in the observation mode of the endoscope system 1 will be described.

[0100] Next, the fluorescence observation mode and the normal light observation mode are described in sequence.

[0101] [Observation principle in fluorescence observation mode]

[0102] First, the principle of observation in the fluorescence observation mode will be described.

[0103] Figure 5 A diagram illustrating the principle of observation in the fluorescence observation mode.

[0104] like Figure 5 As shown in the graph G11 of FIG. 1 , first, the control device 4 irradiates the biological tissue O10 with excitation light (central wavelength 415 nm) by making the second light source unit 403 emit light. In this case, as shown in FIG. Figure 5 As shown in the graph G12, at least the components of the excitation light reflected by the biological tissue O10 and the reflected light of the return light (hereinafter referred to as reflected light W10) are shielded by the cutoff filter 203 and the intensity is reduced. On the other hand, a part of the components on the long wavelength side longer than the shielded wavelength range is incident on the imaging element 204 without reducing the intensity.

[0105] More specifically, if Figure 5As shown in the graph G12 of FIG. 2 , the cutoff filter 203 blocks most of the reflected light W10 incident on the G pixel in the imaging element 204. This reflected light W10 is light in the wavelength range on the short wavelength side including the wavelength range of the excitation light, and the cutoff filter 203 transmits the wavelength range on the long wavelength side longer than the blocked wavelength range. Figure 5 As shown in graph G12, the cutoff filter 203 transmits fluorescence WF10 emitted by the advanced glycation end products themselves, which are generated by heat treatment of the biological tissue O10. Therefore, the fluorescence WF10 and the reflected light W10 with reduced intensity enter each of the R, G, and B pixels of the imaging element 204.

[0106] Here, the G pixel in the imaging element 204 is sensitive to the fluorescence WF10. Figure 5 The fluorescence characteristic curve L in the graph G12 NG As shown, fluorescence is a small reaction. Therefore, the output value corresponding to the fluorescence WF10 in the G pixel is a small value.

[0107] The image processing unit 406 then acquires image data (RAW data) from the imaging element 204 and performs image processing on the output values ​​of the G and B pixels included in the image data to generate a fluorescence image. In this case, the output values ​​of the G pixels include fluorescence information corresponding to fluorescence WF10 emitted from the heat-treated region (advanced glycation end products) of the biological tissue O10 after heat treatment. Furthermore, the output values ​​of the B pixels include background information from the subject's biological tissue O10, including the heat-treated region. Displaying this fluorescence image on the display device 3 allows observation of the heat-treated region of the biological tissue O10 after heat treatment.

[0108] [Observation principle in normal light observation mode]

[0109] Next, the observation principle in the normal light observation mode will be described.

[0110] Figure 6 This is a diagram explaining the observation principle in the normal light observation mode.

[0111] like Figure 6 As shown in the diagram G21, the control device 4 first emits white light to the biological tissue O10 by making the first light source unit 402 emit light. In this case, part of the reflected light and return light (hereinafter referred to as reflected light WR30, WG30, and WB30) reflected by the biological tissue O10 is blocked by the cut filter 203, and the remaining part is incident on the imaging element 204. Specifically, as shown in FIG. Figure 6As shown in graph G22, cutoff filter 203 blocks reflected light in a wavelength range shorter than the excitation light wavelength range. Therefore, compared to a state without cutoff filter 203, the component of light in the blue wavelength range incident on the B pixel of imaging element 204 is reduced.

[0112] Next, the image processing unit 406 acquires image data (RAW data) from the imaging element 204 and performs image processing on the output values ​​of the R, G, and B pixels included in this image data to generate an observation image (white light image). In this case, since the blue component included in the image data is smaller than when the cutoff filter 203 is not configured, the image processing unit 406 performs white balance adjustment processing to maintain a constant ratio of red, green, and blue components. Furthermore, when this observation image (white light image) is displayed on the display device 3, a natural observation image (white light image) can be viewed even when the cutoff filter 203 is configured.

[0113] [Control method]

[0114] Next, the control method executed by the control device 4 will be described.

[0115] Below, the control method executed by the control device 4 before the heat treatment device 5 performs heat treatment on the biological tissue (hereinafter referred to as the control method before heat treatment) and the control method executed by the control device 4 when performing the heat treatment (hereinafter referred to as the control method during heat treatment) are described in sequence.

[0116] [Control method before heat treatment]

[0117] First, a control method executed by the control device 4 before the thermal treatment device 5 performs thermal treatment on living tissue will be described.

[0118] Figure 7 This is a flowchart showing a control method before heat treatment. Figure 8 and Figure 9 This is a diagram illustrating the control method before heat treatment. Specifically, Figure 8 3 is a diagram showing the tissue image FA acquired in step S1A. Figure 9 is with Figure 8 The corresponding figure is a figure explaining step S1C.

[0119] The control unit 409 controls the operation of the communication unit 410 in response to the user's operation on the input unit 407, and acquires the tissue image FA ( Figure 8 )(Step S1A).

[0120] Here, the tissue image FA is a tomographic image of the same subject as the subject to be thermally treated by the thermal treatment apparatus 5, captured by an external tomography apparatus. In the tissue image FA, each pixel is annotated with the three-dimensional coordinates of the corresponding position on the observation object corresponding to the pixel position. These three-dimensional coordinates are based on a specific coordinate system and are calculated by the tomography apparatus.

[0121] After step S1A, the control unit 409 acquires range information indicating the treatment target range of the thermal treatment in the tissue image FA (step S1B).

[0122] Specifically, in step S1B, the control unit 409 acquires (extracts) the treatment target range ArT ( ) representing the thermal treatment in the tissue image FA by performing image recognition using the learning model recorded in the learning model recording unit 408b. Figure 9 ) range information.

[0123] After step S1B, the image processing unit 406, under the control of the control unit 409, performs the following operations: Figure 9 As shown, the treatment target area ArT is superimposed on the tissue image FA (step S1C). Then, the control unit 409 records the data of the tissue image FA on which the treatment target area ArT is superimposed in the recording unit 408.

[0124] Through the above steps S1A to S1C, the control method before heat treatment is completed.

[0125] [Control method during heat treatment]

[0126] Next, a control method executed by the control device 4 when the thermal treatment device 5 performs thermal treatment on living tissue will be described.

[0127] Figure 10 This is a flowchart showing a control method when performing heat treatment. Figures 11 to 15 This is a diagram illustrating a control method for heat treatment. Specifically, Figure 11 The straight line L shows the correlation between the fluorescence intensity of the fluorescence emitted by the advanced glycation end products in the biological tissue and the degree of invasion (depth and area) of the biological tissue by heat treatment ( Y ). Figure 11 In the figure, the vertical axis represents the fluorescence intensity, and the horizontal axis represents the degree of invasion of the biological tissue by heat treatment. Figure 12 is with Figure 8 and Figure 9 The corresponding figure shows the display image DI generated in step S2B. Figures 13 to 15 is with Figure 12 The corresponding figures are figures explaining steps S2D to S2H.

[0128] Furthermore, the endoscope system 1 has already been set to the following state.

[0129] Specifically, the insertion unit 21 is inserted from the subject's oral cavity into the esophagus, and the observation area of ​​the endoscope system 1 is the area within the esophagus. Furthermore, the thermal treatment device 5 is inserted into the esophagus through the insertion port 222 via the treatment instrument channel within the insertion unit 21, thereby enabling thermal treatment. Furthermore, the observation mode is switched to the specific observation mode in response to a user, such as a surgeon, operating the operating member 223 to "switch the observation mode of the endoscope system 1 to the specific observation mode."

[0130] First, the image processing unit 406 generates a thermal denaturation image FB ( ) capable of identifying thermally denatured regions Ar1 to Ar3 in the biological tissue that have undergone thermal denaturation, under the control of the control unit 409. Figures 12 to 15 ) (Step S2A). Here, the thermal denaturation region Ar1 is an insufficient thermal denaturation region with insufficient thermal denaturation. In addition, the thermal denaturation region Ar2 is an excessive thermal denaturation region with excessive thermal denaturation. Furthermore, the thermal denaturation region Ar3 is an appropriate thermal denaturation region with appropriate thermal denaturation. Figures 13 to 15 In FIG. 1 , for convenience of explanation, the thermal denaturation appropriate region Ar3 in the thermal denaturation image FB is omitted from illustration.

[0131] Specifically, in the specific observation mode, the fluorescence observation mode and the normal light observation mode are alternately switched to generate a fluorescence image and an observation image (white light image) in a time-sharing manner. Then, in step S2A, the image processing unit 406 performs a superposition process to generate a thermal denaturation image FB by superimposing the fluorescence image and the observation image (white light image) generated at approximately the same timing as the fluorescence image.

[0132] Here, as the superimposition processing executed by the image processing unit 406 , the following first superimposition processing and the second superimposition processing can be exemplified.

[0133] The first superimposition process is a process of replacing the regions at the same pixel positions as the thermally denatured regions Ar1 to Ar3 in the fluorescence image in the observation image (white light image) with the images of the thermally denatured regions Ar1 to Ar3 in the fluorescence image.

[0134] The second superposition processing is the following processing (so-called alpha blending processing): based on the fluorescence intensity at each pixel position in the thermally denatured areas Ar1 to Ar3 of the fluorescence image, the brightness of the color of the fluorescence assigned to each pixel in the area of ​​the observation image (white light image) that is the same pixel position as the thermally denatured areas Ar1 to Ar3 is changed.

[0135] After step S2A, the image processing unit 406 reads the data of the tissue image FA recorded in the recording unit 408 in step S1C, and generates the following image: Figure 12 As shown, the tissue image FA and the thermal denaturation image FB generated in step S2A are arranged together in a display image DI (step S2B).

[0136] After step S2B, the control section 409 starts a determination process for determining the correspondence relationship between the tissue image FA and the thermally denatured image FB for each pixel (step S2C).

[0137] Specifically, in step S2C, the control unit 409 estimates the shape of the insertion portion 21 based on the magnetic force generated from the sensor unit 209 and received by the receiving unit 411, and calculates the position information of the three-dimensional coordinates representing the position of the front end (front end portion 24) of the insertion portion 21 and the direction information representing the camera field of view of the front end. The three-dimensional coordinates are coordinates in the same coordinate system as the three-dimensional coordinates assigned to each pixel in the tissue image FA. In addition, the control unit 409 calculates the three-dimensional coordinates of the corresponding position on the observation object corresponding to the pixel position for each pixel of the thermal denaturation image FB based on the calculated position information and direction information and the depth map information included in the image data used to generate the thermal denaturation image FB. Then, the control unit 409 starts a determination process of comparing the three-dimensional coordinates assigned to each pixel in the tissue image FA with the three-dimensional coordinates calculated for each pixel in the thermal denaturation image FB, and determining the correspondence between each pixel of the tissue image FA and the thermal denaturation image FB. In addition, based on the result of the determination process, the image processing unit 406, under the control of the control unit 409, as Figure 12 As shown, a position P1 is superimposed on the tissue image FA in the display image DI. The position P1 is a position on the tissue image FA corresponding to the position of the region of the subject in the thermally denatured image FB currently being observed by the endoscope 2 .

[0138] After step S2C, the control unit 409 performs notification control (step S2D).

[0139] Specifically, in step S2D, the control unit 409 controls the operation of the image processing unit 406 to Figure 12 The display image DI shown is displayed on the display device 3. That is, the display device 3 corresponds to the notification unit according to the present invention.

[0140] In addition, there is a correlation between the fluorescence intensity of the advanced glycation end products themselves in the biological tissue and the degree of invasion of the biological tissue by heat treatment (the degree of thermal denaturation). Figure 11 Specifically, Figure 11 The straight line L YAs shown, the higher the degree of thermal denaturation (the greater the degree of invasion of the biological tissue by the heat treatment), the higher the fluorescence intensity.

[0141] Furthermore, if a user, such as a surgeon, can identify the insufficiently thermally denatured region Ar1 and the excessively thermally denatured region Ar2 within the treatment target area ArT, measures can be taken to reduce the risk of postoperative recurrence and perforation. Therefore, in this embodiment, steps S2E to S2H described below are executed to allow a user, such as a surgeon, to identify the insufficiently thermally denatured region Ar1 and the excessively thermally denatured region Ar2 within the treatment target area ArT.

[0142] Specifically, the image processing unit 406 extracts pixels having a fluorescence intensity of the first fluorescence intensity Th1 ( Figure 11 ) or less as the thermal denaturation insufficient region Ar1 (step S2E).

[0143] Furthermore, the image processing unit 406 extracts the second fluorescence intensity Th2 ( Th1 ) having a fluorescence intensity greater than the first fluorescence intensity Th1 from all pixels of the fluorescence image used to generate the thermally denatured image FB. Figure 11 ) or more pixels as the thermal denaturation transition area Ar2 (step S2F).

[0144] Then, the image processing unit 406 extracts, as the thermal denaturation appropriate region Ar3, a region consisting of pixels having a fluorescence intensity greater than the first fluorescence intensity Th1 and less than the second fluorescence intensity Th2 from among all pixels of the fluorescence image used to generate the thermal denaturation image FB (step S2G).

[0145] Furthermore, steps S2E to S2G may be executed in the order of steps S2E to S2G, or in another order, or may be executed in parallel substantially at the same time.

[0146] Here, as examples of the fluorescence intensity used in steps S2E to S2G, the output value of the G pixel in the imaging element 204, at least the g value among the pixel values ​​(r, g, b) of each pixel after de-mosaicing the image data acquired from the imaging element 204, or the brightness value corresponding to the Y signal (brightness signal), etc. can be exemplified.

[0147] Then, the image processing unit 406 updates the display image DI (step S2H). Thereafter, the control device 4 returns to step S2D.

[0148] Specifically, in step S2H, the image processing section 406 executes the following processing based on the result of the determination processing under the control of the control section 409 .

[0149] Specifically, the image processing unit 406 superimposes the insufficient thermal denaturation region Ar1 on the tissue image FA, which corresponds to the insufficient thermal denaturation region Ar1 on the fluorescence image extracted in step S2E; the excessive thermal denaturation region Ar2 on the tissue image FA, which corresponds to the excessive thermal denaturation region Ar2 on the fluorescence image extracted in step S2F; and the appropriate thermal denaturation region Ar3 on the tissue image FA, which corresponds to the appropriate thermal denaturation region Ar3 on the fluorescence image extracted in step S2G. In other words, the insufficient thermal denaturation region Ar1, the excessive thermal denaturation region Ar2, and the appropriate thermal denaturation region Ar3 correspond to the thermal denaturation information according to the present invention.

[0150] By repeatedly executing steps S2D to S2H described above, the display image DI generated by the image processing unit 406 is changed as follows.

[0151] First, in the state before the heat treatment by the heat treatment device 5, the image processing unit 406 generates Figure 12 The display image DI is shown.

[0152] Specifically, as determined from the position P1 , the display image DI shows that no thermal treatment has been performed, and therefore no insufficient thermal denaturation region Ar1 , excessive thermal denaturation region Ar2 , or appropriate thermal denaturation region Ar3 exists within the treatment target range ArT.

[0153] When the heat treatment device 5 continues to perform heat treatment, the image processing unit 406 generates Figure 13 The display image DI is shown.

[0154] Specifically, the display image DI is judged based on the position P1 because the treatment target area ArT is being moved from the Figure 13 The uppermost portion of the image is subjected to downward heat treatment and is halfway through, so the image is an image in which at least one of the insufficient heat denaturation region Ar1, the excessive heat denaturation region Ar2, and the appropriate heat denaturation region Ar3 exists halfway through the treatment target range ArT. Figure 13 In the example of , only the thermal denaturation suitable region Ar3 exists until the middle.

[0155] Then, in a state where the heat treatment is completed by the heat treatment device 5, the image processing unit 406 generates Figure 14 The display image DI is shown.

[0156] Specifically, the display image DI is determined based on the position P1, and the treatment target area ArT is within the range of Figure 14Since the heat treatment is completed up to the bottom of the image, the entire treatment target range ArT contains at least one of the insufficient heat denaturation region Ar1, the excessive heat denaturation region Ar2, and the appropriate heat denaturation region Ar3. Figure 14 In the example of , there are a region Ar1 with insufficient thermal denaturation, a region Ar2 with excessive thermal denaturation, and a region Ar3 with appropriate thermal denaturation. Figure 14 In the figure, the insufficient heat denaturation region Ar1 is represented by a hollow circle, the excessive heat denaturation region Ar2 is represented by a dotted circle, and the appropriate heat denaturation region Ar3 is represented by a slanted line.

[0157] For example, after the completion of the thermal treatment, the user such as the operator can return the distal end of the insertion portion 21 to the position of the insufficiently thermally denatured region Ar1 while checking the position P1 in the tissue image FA displayed on the display image DI of the display device 3. Figure 15 As shown, the insufficient thermal denaturation region Ar1 is confirmed by the thermal denaturation image FB.

[0158] According to the embodiment described above, the following effects are achieved.

[0159] The control device 4 according to this embodiment acquires the tissue image FA and range information indicating the treatment target area ArT. Furthermore, the control device 4 determines the state of thermal denaturation caused by the thermal treatment based on the fluorescence image. The control device 4 then causes a notification unit to notify the patient of thermal denaturation information indicating the state of thermal denaturation within the treatment target area ArT. In this embodiment, the control device 4 displays a superimposed image on the tissue image FA on the display device 3 serving as the notification unit, in which the thermal denaturation information is superimposed.

[0160] Therefore, the control device 4 according to the present embodiment can allow a user such as a surgeon to determine whether or not thermal treatment has been appropriately performed over the entire wide treatment target area ArT, thereby improving convenience.

[0161] Furthermore, when determining the state of thermal denaturation caused by heat treatment, the control device 4 according to this embodiment extracts the insufficient thermal denaturation region Ar1 , the excessive thermal denaturation region Ar2 , and the appropriate thermal denaturation region Ar3 based on the fluorescence intensity of each pixel in the fluorescence image.

[0162] Therefore, the state of thermal denaturation caused by heat treatment can be easily and accurately determined.

[0163] (Other embodiments)

[0164] Although the embodiments for carrying out the present invention have been described so far, the present invention should not be limited to the above-described embodiments.

[0165] In the above embodiment, the medical device involved in the present invention is mounted on an endoscopic system for transesophageal endoscopic myotomy (POEM), but is not limited to this. It can also be mounted on an endoscopic system for other surgeries, such as ablation of the inferior turbinate mucosa for allergic rhinitis and ablation of hematocysts for endometriosis.

[0166] In the above embodiment, the medical device according to the present invention is mounted on an endoscope system using a flexible endoscope, but the present invention is not limited thereto and may be mounted on an endoscope system using a rigid endoscope or an endoscope system using a medical surgical robot.

[0167] In the above embodiment, the display device 3 is used as the notification unit according to the present invention, but the present invention is not limited thereto. In addition to the configuration of displaying an image, the notification unit according to the present invention may also use a speaker that notifies by sound.

[0168] In the above embodiment, the insufficient thermal denaturation region Ar1, the excessive thermal denaturation region Ar2, and the appropriate thermal denaturation region Ar3 are superimposed on the tissue image FA including the treatment target area ArT to display thermal denaturation information indicating the state of thermal denaturation within the treatment target area ArT. However, the present invention is not limited thereto.

[0169] For example, a configuration may be employed in which only the thermal denaturation image FB is displayed instead of the tissue image FA itself. Furthermore, when thermal treatment of the treatment target area ArT is completed, a notification is provided (via a display of images, characters, or the like, or a sound notification) indicating that thermal treatment of the treatment target area ArT has been appropriately completed, or indicating that an insufficiently thermally denatured region Ar1 or an excessively thermally denatured region Ar2 exists within the treatment target area ArT.

[0170] In the above-described embodiment, as the tissue image according to the present invention, the tissue image FA, which is a tomographic image captured by a tomographic apparatus, is used, but the present invention is not limited thereto.

[0171] For example, an image (hereinafter referred to as a connected image) generated by connecting endoscopic images (observation images (white light images)) captured by the endoscope 2 using SLAM (Simultaneous Localization and Mapping) technology may be used as the tissue image involved in the present invention.

[0172] Furthermore, the tissue image according to the present invention may be a two-dimensional image or a three-dimensional image.

[0173] In the above embodiment, the control device 4 calculates the three-dimensional coordinates of the position of the distal end (distal end portion 24 ) of the insertion portion 21 by using the TOF sensor included in the imaging element 204 , the sensor unit 209 , and the receiving unit 411 , but the present invention is not limited thereto.

[0174] For example, the TOF sensor, the sensor unit 209, and the receiving unit 411 may be omitted, and the control device 4 may calculate the three-dimensional coordinates of the position of the distal end of the insertion portion 21 based on the connected image.

[0175] In the above embodiment, the control device 4 acquires (extracts) the area information indicating the treatment target area ArT in the tissue image FA by image recognition using the learning model recorded in the learning model recording unit 408 b , but the present invention is not limited thereto.

[0176] For example, the control device 4 may be configured to display the tissue image FA on the display device 3 and acquire a range selected from the tissue image FA as the treatment target range ArT by an operation of the input unit 407 by a user such as a surgeon.

[0177] Description of Reference Numerals

[0178] 1: Endoscope system; 2: Endoscope; 3: Display device; 4: Control device; 5: Heat treatment device; 21: Insertion portion; 22: Operation portion; 23: Universal cable; 24: Front end portion; 25: Bending portion; 26: Flexible tube portion; 27: First connector portion; 27a: Cable; 28: Second connector portion; 201: Illumination optical system; 202: Imaging optical system; 203: Cutoff filter; 204: Imaging element; 205: A / D conversion portion; 206: P / S conversion portion; 207: Imaging recording portion; 208: Imaging control portion; 209: Sensor portion; 221: Bending knob; 222: Insertion port; 223: Operation member; 231: Light guide; 232: First signal line; 233: Second signal line; 401: Converging lens; 402: First light source unit; 403: Second light source unit; 404: Light source control unit; 405: S / P conversion unit; 406: Image processing unit; 407: Input unit; 408: Recording unit; 408a: Program recording unit; 408b: Learning model recording unit; 409: Control unit; 410: Communication unit; 411: Receiving unit; Ar1: Insufficient thermal denaturation area; Ar2: Excessive thermal denaturation area; Ar3: Appropriate thermal denaturation area; ArT: Treatment target area; DI: Display image; FA: Tissue image; FB: Thermal denaturation image; G11 to G13, G21 to G23: Graphs; L1: Optical axis; L B 、L G 、L R 、L V: curve; L F : Transmission characteristics of the cutoff filter; L NG : wavelength characteristics of fluorescence; L Y : straight line; O10: biological tissue; P1: position; Th1: first fluorescence intensity; Th2: second fluorescence intensity; W10, WB30, WG30, WR30: reflected light; WF10: fluorescence.

Claims

1. A medical device comprising a processor configured to process a captured image of fluorescence generated from living tissue by irradiating the living tissue with excitation light. The processor performs the following processing: acquiring a tissue image including a target site for heat treatment in the biological tissue; acquiring range information indicating a treatment target range of the thermal treatment in the tissue image; determining a state of thermal denaturation caused by the heat treatment based on the captured image; as well as The notification unit is caused to notify thermal denaturation information indicating the state of the thermal denaturation within the treatment target area.

2. The medical device according to claim 1, wherein The notification unit is a display unit that displays an image. The processor causes the display unit to display a superimposed image in which the thermal denaturation information is superimposed on the tissue image.

3. The medical device according to claim 1, wherein The fluorescence is fluorescence generated from advanced glycation end products produced by subjecting the biological tissue to the heat treatment.

4. The medical device according to claim 1, wherein The processor determines a state of thermal denaturation caused by the heat treatment based on the fluorescence intensity of each pixel in the captured image.

5. The medical device according to claim 4, wherein The processor performs the following processing: determining a region composed of pixels having a fluorescence intensity equal to or less than a first fluorescence intensity among all pixels of the captured image as an insufficiently thermally denatured region in which thermal denaturation due to the heat treatment is insufficient; as well as The notification unit is caused to notify the thermal denaturation information indicating the insufficiently thermally denatured region within the treatment target range.

6. The medical device according to claim 5, wherein The processor performs the following processing: determining a region composed of pixels having a fluorescence intensity greater than or equal to a second fluorescence intensity among all pixels of the captured image as an excessive thermal denaturation region where thermal denaturation is excessive due to the heat treatment, wherein the second fluorescence intensity is higher than the first fluorescence intensity; and The notification unit is caused to notify the thermal denaturation information indicating the excessive thermal denaturation region within the treatment target area.

7. The medical device according to claim 1, wherein The tissue image is a tomographic image captured by a tomographic device or an ultrasonic image generated by an ultrasonic observation device.

8. The medical device according to claim 1, wherein The tissue image is an image generated by connecting endoscopic images captured by an endoscope.

9. The medical device according to claim 1, wherein The captured image is an endoscopic image captured by an endoscope having an insertion portion inserted into a subject. The processor calculates the three-dimensional coordinates of the subject in the camera image based on position information indicating the position of the front end of the insertion portion, direction information indicating the camera field of view of the front end, and the camera image, and determines the correspondence between each pixel of the tissue image and the camera image based on the three-dimensional coordinates.

10. An endoscope system comprising: a light source device for irradiating excitation light; an endoscope that can be inserted into a subject and outputs a captured image of fluorescence generated from living tissue within the subject by irradiating the living tissue with the excitation light; as well as A medical device comprising a processor for processing the captured image. The processor performs the following processing: acquiring a tissue image including a target site for heat treatment in the biological tissue; acquiring range information indicating a treatment target range of the thermal treatment in the tissue image; determining a state of thermal denaturation caused by the heat treatment based on the captured image; and The notification unit is caused to notify thermal denaturation information indicating the state of the thermal denaturation within the treatment target area.

11. A control method executed by a medical device, wherein: acquiring a tissue image including a target site for thermal treatment in a living tissue, acquiring range information indicating a treatment target range of the thermal treatment in the tissue image, determining a state of thermal denaturation caused by the heat treatment based on a captured image of fluorescence generated from the living tissue by irradiating the living tissue with excitation light, The notification unit is caused to notify thermal denaturation information indicating the state of the thermal denaturation within the treatment target area.

12. A control program for causing a medical device to execute a control program, wherein: The control program instructs the medical device to execute the following processing: acquiring a tissue image including a target site for heat treatment in a living tissue; acquiring range information indicating a treatment target range of the thermal treatment in the tissue image; determining a state of thermal denaturation caused by the heat treatment based on a captured image of fluorescence generated from the living tissue by irradiating the living tissue with excitation light; as well as The notification unit is caused to notify thermal denaturation information indicating the state of the thermal denaturation within the treatment target area.

Images