Image processing device, medical system, image processing device operating method, and learning device

Abstract

An image processing apparatus includes a processor that acquires a fluorescence image in which a biological tissue is irradiated with excitation light and fluorescence is captured, extracts a first pixel having a brightness value equal to or greater than a first threshold value from the fluorescence image, specifies a first region on the basis of position information of the first pixel, and transmits the first region to the biological tissue. A second pixel having a luminance value equal to or less than a second threshold value is extracted within the first region of the fluorescence image, a second region is determined on the basis of position information of the second pixel, and information obtained by superimposing the second region on the fluorescence image is output. As a result, provided is an image processing device capable of easily recognizing a region in which hemostasis treatment is insufficient.

Description

Technical Field

[0001] The present invention relates to an image processing device, a medical system, a working method of the image processing device, and a learning device. Background Art

[0002] In recent years, minimally invasive treatments using endoscopes, laparoscopy, etc. have been widely performed in the medical field. For example, endoscopic submucosal dissection (ESD) has been widely performed as a minimally invasive treatment using endoscopes, laparoscopy, etc.

[0003] In ESD, an energy device, such as a high-frequency electrosurgical unit, is used to excise the periphery of a lesion in biological tissue. When the tissue undergoes thermal denaturation due to the heat treatment applied by the energy device, AGEs (advanced glycation end products) are generated. AGEs fluoresce when irradiated with excitation light, allowing the status of thermal treatment to be visualized using fluorescence imaging (e.g., see Patent Document 1). The surgeon then performs hemostatic treatment using thermal coagulation on the excised area while observing the fluorescence image.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: International Publication No. 2020 / 054723 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] However, it is sometimes difficult to identify areas with insufficient hemostatic treatment based on fluorescence images.

[0009] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing device, a medical system, an operating method of the image processing device, and a learning device that can easily identify an area where hemostasis treatment is insufficient.

[0010] Solutions for solving problems

[0011] In order to solve the above-mentioned problems and achieve the purpose, an image processing device involved in one embodiment of the present invention includes a processor, the processor having hardware, wherein the processor obtains a fluorescent image in which fluorescence is captured by irradiating excitation light to biological tissue, the processor extracts a first pixel whose brightness value is greater than a first threshold value from the fluorescent image, the processor determines a first area based on position information of the first pixel, the processor extracts a second pixel whose brightness value is less than a second threshold value within the first area of ​​the fluorescent image, the processor determines a second area based on position information of the second pixel, and the processor outputs information after superimposing the second area on the fluorescent image.

[0012] Furthermore, in the image processing device according to one aspect of the present invention, the first threshold is larger than the second threshold.

[0013] Furthermore, in the image processing device according to one embodiment of the present invention, the fluorescence image is an image in which the fluorescence generated from a thermally denatured region of the biological tissue is captured.

[0014] Furthermore, in the image processing apparatus according to one embodiment of the present invention, the thermally denatured region is formed by subjecting the biological tissue to a thermal treatment using an energy device.

[0015] Furthermore, in the image processing device according to one embodiment of the present invention, the first threshold value and the second threshold value are set based on the brightness value of the fluorescence generated from the advanced glycation end products produced by thermal denaturation of the biological tissue.

[0016] Furthermore, in the image processing device according to one embodiment of the present invention, the processor outputs information in which the first region is superimposed on the fluorescent image.

[0017] Furthermore, in the image processing device according to one aspect of the present invention, the processor outputs information obtained by superimposing the second region on a white-light image obtained by irradiating the biological tissue with white light and capturing return light.

[0018] Furthermore, in the image processing device according to one aspect of the present invention, the processor outputs information obtained by superimposing the first region on a white-light image obtained by irradiating the biological tissue with white light and capturing return light.

[0019] Furthermore, in the image processing device according to one aspect of the present invention, the processor specifies a circular area formed by connecting the first pixels as the first area.

[0020] In addition, a medical system involved in one embodiment of the present invention comprises: a light source device that irradiates excitation light to biological tissue; an endoscope that has an imaging element that outputs an imaging signal that captures fluorescence emitted by the excitation light; and an image processing device that has a processor that generates a fluorescence image based on the imaging signal, wherein the processor extracts a first pixel whose brightness value is greater than a first threshold value from the fluorescence image, the processor determines a first area based on position information of the first pixel, the processor extracts a second pixel whose brightness value is less than a second threshold value within the first area of ​​the fluorescence image, the processor determines a second area based on position information of the second pixel, and the processor outputs information after superimposing the second area on the fluorescence image.

[0021] In addition, an operating method of an image processing device involved in one embodiment of the present invention is an operating method of an image processing device having a processor with hardware, wherein the processor obtains a fluorescent image in which fluorescence is captured by irradiating excitation light to biological tissue, the processor extracts a first pixel having a brightness value greater than a first threshold value from the fluorescent image, the processor determines a first area based on position information of the first pixel, the processor extracts a second pixel having a brightness value less than a second threshold value within the first area of ​​the fluorescent image, the processor determines a second area based on position information of the second pixel, and the processor outputs information after superimposing the second area on the fluorescent image.

[0022] In addition, a learning device involved in one embodiment of the present invention includes a learning unit, which generates a learned model by performing machine learning using teaching data that uses a fluorescent image obtained by irradiating excitation light onto biological tissue and capturing fluorescence as input data, and uses information obtained by superimposing position information of a second area determined based on a second pixel with a brightness value below a second threshold value in a first area in the fluorescent image as output data, wherein the first area is an area determined based on a first pixel with a brightness value above the first threshold value in the fluorescent image.

[0023] Effects of the Invention

[0024] According to the present invention, it is possible to realize an image processing device, a medical system, an operating method of the image processing device, and a learning device that can easily identify an area where hemostasis treatment is insufficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a diagram schematically showing the overall configuration of an endoscope system according to one embodiment.

[0026] Figure 2This is a block diagram showing the functional configuration of a main portion of an endoscope system according to one embodiment.

[0027] Figure 3 This is a flowchart showing an overview of processing executed by the control device.

[0028] Figure 4 This is an example of a fluorescence image.

[0029] Figure 5 3 is a diagram showing a state where the second region is superimposed on the fluorescent image.

[0030] Figure 6 3 is a diagram showing a state where the second region is superimposed on the white light image. DETAILED DESCRIPTION

[0031] Below, as a method for implementing the present disclosure (hereinafter referred to as an "embodiment"), an endoscope system having an endoscope with a soft insertion portion is described, but it is not limited to this. For example, a rigid endoscope and a surgical robot can also be applied. In addition, the present disclosure is not limited to this embodiment. Moreover, in the description of the accompanying drawings, the same parts are marked with the same figure numerals for description. In addition, it should be noted that the accompanying drawings are schematic, and the relationship between the thickness and width of each component, the ratio of each component, etc. are different from the actual ones. In addition, the drawings also include parts with different sizes and ratios.

[0032] [Structure of the endoscope system]

[0033] Figure 1 This is a diagram schematically showing the overall configuration of an endoscope system according to one embodiment. Figure 1 The endoscope system 1 shown in the figure inserts an insertion portion of an endoscope into a body cavity or lumen of a patient or other subject to be examined, thereby imaging the subject's internal organs and displaying images based on the captured imaging signals on a display device. The endoscope system 1 includes an endoscope 2, a light source device 3, a control device 4 as an image processing device, and a display device 5.

[0034] [Structure of an endoscope]

[0035] First, the structure of the endoscope 2 will be described.

[0036] The endoscope 2 generates an imaging signal (RAW data) capturing the interior of the subject's body and outputs the generated imaging signal to the control device 4. Specifically, the endoscope 2 generates a first imaging signal capturing return light by irradiating white light and a second imaging signal capturing fluorescence by irradiating excitation light. The endoscope 2 includes an insertion portion 21, an operating portion 22, and a universal cable 23.

[0037] The insertion section 21 is inserted into the subject. It has a flexible, elongated shape and includes a distal end 24 housing an imaging element (described later), a freely bendable bending section 25 composed of a plurality of bending pieces, and a flexible, elongated flexible tubular section 26 connected to the proximal end of the bending section 25.

[0038] The distal end portion 24 is made of glass fiber, etc. It forms a light guide path for illumination light supplied from the control device 4 via the universal cable 23 and the operation unit 22 , generates an imaging signal that captures return light of the illumination light, and outputs the imaging signal to the control device 4 .

[0039] The operating section 22 includes a bending knob 221 for bending the bending section 25 in the vertical and horizontal directions; a treatment instrument insertion section 222 for inserting treatment instruments; and a plurality of switches 223 serving as an operation input section. These switches 223 input, in addition to operation instruction signals for the control device 4, operation instruction signals for peripheral devices such as the air supply unit, water supply unit, and gas delivery unit, a pre-freeze signal for instructing the endoscope system 1 to perform still image capture, and a switching signal for switching the observation mode of the endoscope system 1. Treatment instruments inserted through the treatment instrument insertion section 222 are exposed from the opening via the treatment instrument channel of the distal end portion 24.

[0040] The universal cable 23 contains at least a light guide and a converging cable that brings together one or more cables. The converging cable is a signal line that transmits and receives signals between the endoscope 2 and the control device 4. It includes a signal line for transmitting and receiving imaging signals (RAW data) and a signal line for transmitting and receiving timing signals (synchronization signals and clock signals) used to drive the imaging element (described later). The universal cable 23 has a connector portion 27 and a connector portion 28. The connector portion 27 is removably connected to the control device 4. A spiral cable 27a extends from the spiral cable 27a. The connector portion 28 is removably connected to the control device 4 at the end of the spiral cable 27a.

[0041] [Structure of light source device]

[0042] Next, the structure of the light source device will be described.

[0043] The light source device 3 irradiates white light and excitation light as illumination light to the biological tissue. The light source device 3 is connected to one end of the light guide of the endoscope 2, and under the control of the control device 4, provides the illumination light to be irradiated into the subject to the one end of the light guide. The light source device 3 is implemented using one or more light sources such as LED (Light Emitting Diode: Light Emitting Diode) light source, xenon lamp and LD (laser Diode: Laser Diode) semiconductor laser elements, a processor and a memory, wherein the processor is a processing device having hardware such as FPGA (Field Programmable Gate Array: Field Programmable Gate Array), CPU (Central Processing Unit: Central Processing Unit), and the memory is a temporary storage area used by the processor. In addition, the light source device 3 and the control device 4 can be as follows: Figure 1 Although the structures shown are configured to perform communication individually, they may be configured as an integrated structure.

[0044] [Structure of control device]

[0045] Next, the configuration of the control device 4 will be described.

[0046] The control device 4 controls the various components of the endoscope system 1. The control device 4 controls the light source device 3 to provide illumination light for the endoscope 2 to irradiate the subject. Furthermore, the control device 4 performs various image processing on the imaging signals input from the endoscope 2 and outputs the processed imaging signals to the display device 5.

[0047] [Structure of display device]

[0048] Next, the structure of the display device 5 will be described.

[0049] The display device 5 displays an image based on a video signal input from the control device 4 under the control of the control device 4. The display device 5 is implemented using a display panel such as an organic EL (Electro Luminescence) or a liquid crystal.

[0050] [Functional structure of the main parts of the endoscope system]

[0051] Next, the functional configuration of the main parts of the endoscope system 1 will be described. Figure 2 1 is a block diagram showing the functional configuration of the main parts of the endoscope system 1 .

[0052] [Structure of an endoscope]

[0053] First, the structure of the endoscope 2 will be described.

[0054] The endoscope 2 includes an illumination optical system 201, an imaging optical system 202, a cutoff filter 203, an imaging element 204, an A / D converter 205, a P / S converter 206, an imaging recording unit 207, and an imaging control unit 208. Furthermore, the illumination optical system 201, the imaging optical system 202, the cutoff filter 203, the imaging element 204, the A / D converter 205, the P / S converter 206, the imaging recording unit 207, and the imaging control unit 208 are each disposed within the distal end portion 24.

[0055] The illumination optical system 201 irradiates, toward a subject (biological tissue), illumination light supplied from a light guide 231 formed of an optical fiber, etc. The illumination optical system 201 is implemented using one or more lenses, etc.

[0056] The imaging optical system 202 collects reflected light from the subject, return light from the subject, fluorescence emitted by the subject, and other light to form a subject image (light rays) on the light receiving surface of the imaging element 204. The imaging optical system 202 is implemented using one or more lenses and the like.

[0057] The cutoff filter 203 is disposed on the optical axis O1 between the imaging optical system 202 and the imaging element 204. The cutoff filter 203 shields light in the wavelength range of reflected light or return light reflected from the subject by the excitation light provided by the light source device 3, and transmits light in the wavelength range on the longer wavelength side than the wavelength range of the excitation light.

[0058] Under the control of the imaging control unit 208, the imaging element 204 receives the subject image (light) formed by the imaging optical system 202 and transmitted through the cutoff filter 203, performs photoelectric conversion on the subject image (light) to generate an imaging signal (RAW data), and outputs the imaging signal (RAW data) to the A / D converter 205. The imaging element 204 is implemented using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor in which each of a plurality of pixels arranged in a two-dimensional matrix is ​​provided with one of the color filters forming a Bayer array (RGGB).

[0059] Under the control of the imaging control unit 208, the A / D converter 205 performs A / D conversion processing on the analog imaging signal input from the imaging element 204 and outputs the processed imaging signal to the P / S converter 206. The A / D converter 205 is implemented using an A / D conversion circuit or the like.

[0060] Under the control of the camera control unit 208, the P / S converter 206 performs parallel / serial conversion on the digital camera signal input from the A / D converter 205 and outputs the parallel / serial converted camera signal to the control device 4 via the first transmission cable 232. The P / S converter 206 is implemented using a P / S conversion circuit, etc. In the first embodiment, an E / O converter that converts the camera signal into an optical signal may be provided in place of the P / S converter 206, and the camera signal may be output to the control device 4 in the form of an optical signal. Alternatively, the camera signal may be transmitted to the control device 4 via wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0061] The image recording unit 207 records various information related to the endoscope 2 (e.g., pixel information of the image sensor 204 and characteristics of the cutoff filter 203). Furthermore, the image recording unit 207 records various setting data and control parameters transmitted from the control device 4 via the second transmission cable 233. The image recording unit 207 is configured using a nonvolatile memory or a volatile memory.

[0062] The imaging control unit 208 controls the operations of the imaging element 204, the A / D converter 205, and the P / S converter 206 based on the setting data received from the control device 4 via the second transmission cable 233. The imaging control unit 208 is implemented using a TG (Timing Generator), a processor, and a memory. The processor is a processing device including hardware such as a CPU, and the memory is a temporary storage area used by the processor.

[0063] [Structure of light source device]

[0064] Next, the structure of the light source device 3 will be described.

[0065] The light source device 3 includes a condenser lens 30 , a first light source unit 31 , a second light source unit 32 , and a light source control unit 33 .

[0066] The condenser lens 30 condenses the light emitted from the first light source unit 31 and the second light source unit 32 and emits the light to the light guide 231. The condenser lens 30 is formed using one or more lenses.

[0067] Under the control of the light source control unit 33, the first light source unit 31 emits white light (normal light) as visible light, thereby providing white light as illumination light to the light guide 231. The first light source unit 31 is configured using, for example, a collimating lens, a white LED lamp, and a driver. Alternatively, the first light source unit 31 can simultaneously emit red, green, and blue LED lamps to provide white light as visible light. Alternatively, the first light source unit 31 can be configured using, for example, a halogen lamp or a xenon lamp.

[0068] Under the control of the light source control unit 33, the second light source unit 32 provides the excitation light as illumination light to the light guide 231 by emitting excitation light having a prescribed wavelength range. Here, the excitation light is a wavelength that excites substances such as advanced glycation end products (AGEs) contained in the thermally denatured region, for example, a wavelength range of 400 nm or more and 430 nm or less (center wavelength is 415 nm). The thermally denatured region is a region where biological tissue is denatured due to heat treatment by energy equipment such as a high-frequency electric knife. The excitation light irradiated by the second light source unit 32 is cut off by the cutoff filter 203, and the fluorescence (wavelength 540 nm) generated from the AGEs passes through the cutoff filter 203, so that a fluorescence image can be captured. The second light source unit 32 is implemented using a semiconductor laser such as a collimating lens, a purple LD (Laser Diode), and a driver.

[0069] The light source control unit 33 is composed of a processor (a processing device comprising hardware such as an FPGA (Field-Programmable Gate Array) or a CPU (Central Processing Unit)) and memory (a temporary storage area used by the processor). Based on control data input from the control unit 405, the light source control unit 33 controls the light emission timing, light intensity, and light emission duration of the first and second light source units 31 and 32, respectively.

[0070] [Structure of control device]

[0071] Next, the configuration of the control device 4 will be described.

[0072] The control device 4 includes an S / P conversion unit 401 , an image processing unit 402 , an input unit 403 , a recording unit 404 , and a control unit 405 .

[0073] Under the control of the control unit 405, the S / P converter 401 performs serial / parallel conversion on the imaging signal received from the endoscope 2 via the first transmission cable 232, and outputs the converted imaging signal to the image processing unit 402. Furthermore, if the endoscope 2 outputs the imaging signal in the form of an optical signal, an O / E converter that converts the optical signal into an electrical signal may be provided in place of the S / P converter 401. Furthermore, if the endoscope 2 transmits the imaging signal via wireless communication, a communication module capable of receiving wireless signals may be provided in place of the S / P converter 401.

[0074] The image processing unit 402 is implemented using a processor comprising hardware such as a CPU, GPU (Graphics Processing Unit), or FPGA, and a memory serving as a temporary storage area for the processor. Under the control of the control unit 405, the image processing unit 402 performs predetermined image processing on the imaging signal input from the S / P conversion unit 401 and outputs the processed imaging signal to the display device 5. The image processing unit 402 generates a white light image based on the first imaging signal and a fluorescence image based on the second imaging signal. The image processing unit 402 includes an image generation unit 402a, an acquisition unit 402b, an extraction unit 402c, a determination unit 402d, and an output unit 402e.

[0075] The image generator 402a generates a white-light image based on a first imaging signal obtained by capturing return light from the first light source unit 31 while irradiating biological tissue with white light. Furthermore, the image generator 402a generates a fluorescence image based on a second imaging signal obtained by capturing fluorescence from the second light source unit 32 while irradiating biological tissue with excitation light.

[0076] The acquisition unit 402 b acquires the white light image and the fluorescence image from the image generation unit 402 a . In addition, the acquisition unit 402 b acquires the first imaging signal and the second imaging signal from the endoscope 2 .

[0077] The extraction unit 402c extracts first pixels having a brightness value greater than a first threshold value from the fluorescence image. Furthermore, the extraction unit 402c extracts second pixels having a brightness value less than a second threshold value from the first region of the fluorescence image. Furthermore, the first threshold value is greater than the second threshold value.

[0078] The specifying unit 402d specifies the first region as a circular region formed by connecting the first pixels based on the position information of the first pixels. Furthermore, the specifying unit 402d specifies the second region based on the position information of the second pixels.

[0079] The output unit 402e outputs information obtained by superimposing the first region and the second region on the fluorescent image. Alternatively, the output unit 402e may output information obtained by superimposing the first region and the second region on the white light image.

[0080] The input unit 403 receives input of various operations related to the endoscope system 1, and outputs the received operations to the control unit 405. The input unit 403 is configured using a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, and the like.

[0081] The recording unit 404 is implemented using a recording medium such as a volatile memory, a nonvolatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory card. The recording unit 404 records data including various parameters required for the operation of the endoscope system 1. For example, the recording unit 404 stores positional information of a first pixel and a second pixel, positional information of a first area and a second area, and the like. The recording unit 404 also includes a program recording unit 404a, which records various programs used to operate the endoscope system 1.

[0082] The control unit 405 is realized by using a processor having hardware such as an FPGA or a CPU and a memory as a temporary storage area used by the processor. The control unit 405 comprehensively controls each unit constituting the endoscope system 1 .

[0083] 〔Control device processing〕

[0084] Next, the processing executed by the control device 4 will be described.

[0085] Figure 3 FIG. 1 is a flowchart showing an overview of the processing performed by the control device. Figure 3 As shown, first, the acquisition unit 402b acquires a second imaging signal obtained by irradiating the biological tissue with excitation light from the second light source unit 32 and capturing fluorescence (step S1).

[0086] Next, the image generation unit 402 a generates a fluorescent image based on the second imaging signal acquired by the acquisition unit 402 b (step S2 ). The fluorescent image generated by the image generation unit 402 a is stored in the storage unit 404 .

[0087] Thereafter, the acquisition unit 402b acquires the fluorescent image from the recording unit 404 (step S3). Alternatively, the acquisition unit 402b may acquire the fluorescent image from an external server via the Internet.

[0088] Figure 4 This is an example of a fluorescence image. Figure 4The fluorescence image FI1 shown captures fluorescence generated by a thermally denatured area of ​​biological tissue undergoing thermal denaturation. This area is formed by heat treatment of biological tissue using an energy device and contains AGEs (advanced glycation end products). In ESD, the periphery of a lesion in biological tissue is excised using an energy device such as a high-frequency electrosurgical unit. AGEs fluoresce when irradiated with excitation light, resulting in a roughly circular whitening of the periphery of the lesion in the fluorescence image FI1.

[0089] Return to Figure 3 The extraction unit 402c extracts the first pixel having a brightness value greater than or equal to the first threshold value from the fluorescence image FI1 (step S4). This allows the high brightness portion removed by the energy device to be extracted from the fluorescence image FI1.

[0090] Based on the positional information of the first pixel, the identification unit 402d identifies the first region as a circular region formed by connecting the first pixels (step S5). By connecting the portions with high brightness values ​​in the fluorescence image FI1, the circular first region indicated by the dotted line L1 can be identified. The first region is the entire area within the dotted line L1.

[0091] Next, the extraction unit 402c extracts second pixels having a luminance value less than or equal to the second threshold value in the first region of the fluorescence image FI1 (step S6). This allows extraction of portions of the fluorescence image FI1 having low luminance values ​​and insufficient heat treatment.

[0092] Then, the identification unit 402d identifies the second region based on the position information of the second pixel (step S7). The second region having a low luminance value indicated by the solid line L2 can be identified in the fluorescence image FI1.

[0093] Then, the output unit 402e outputs information obtained by superimposing the first region and the second region on the fluorescent image FI1 (step S8). As a result, the information is displayed on the display device 5. Figure 4 The fluorescent image FI1 shown is an image in which a dotted line L1 indicating a first region and a solid line L2 indicating a second region are superimposed.

[0094] According to the embodiment described above, since the solid line L2 indicating the second area is superimposed on the fluorescent image FI1 , the operator can easily recognize the area where the hemostatic treatment is insufficient.

[0095] In addition, in ESD, when the operator presses a predetermined button after removing the lesion, the display of the display device 5 switches from the white light image, and the image can be observed. Figure 4The fluorescent image FI1 is superimposed with the solid line L2 representing the second area. The operator then performs hemostatic treatment using the energy device while confirming the state of the second area. As a result, insufficient hemostatic treatment can be prevented during ESD.

[0096] (Variation)

[0097] Figure 5 FIG is a diagram showing a state where the second region is superimposed on the fluorescent image. Figure 5 As shown, the second area A1 may be superimposed on the fluorescent image FI2 by filling the second area A1 with a specific color. This makes it easier to observe the second area.

[0098] Figure 6 : is a diagram showing a situation where the second area is superimposed on the white light image. Figure 6 As shown in FIG, a dotted line L1 indicating the first region and a solid line L2 indicating the second region may be superimposed on the white-light image WI1. This allows the user to identify a region where hemostasis is insufficient while observing the white-light image.

[0099] Alternatively, you can Figure 4 The fluorescence images FI1 and Figure 6 The white light images shown are displayed side by side, enabling simultaneous observation.

[0100] Furthermore, the control unit 405 may also function as a learning unit of the learning device of the control device 4. The control unit 405 may also generate a learned model by performing machine learning using teaching data that uses as input a fluorescence image obtained by irradiating biological tissue with excitation light and capturing fluorescence, and as output information obtained by superimposing on the fluorescence image a second region identified based on the positional information of a second pixel with a brightness value below a second threshold value within a first region. The first region is defined based on first pixels within the fluorescence image with a brightness value above the first threshold value. The learned model is comprised of a neural network with each layer having one or more nodes. The type of machine learning is not particularly limited; for example, teaching data and learning data are prepared that associate fluorescence images of multiple subjects with images obtained by superimposing on the fluorescence images the second region identified based on the multiple fluorescence images, and these teaching data and learning data are input into a computational model based on a multilayer neural network for learning. Machine learning methods include, for example, methods based on multilayer neural networks such as CNNs (Convolutional Neural Networks) and 3D-CNNs (Deep Neural Networks). Alternatively, as a machine learning method, methods based on recurrent neural networks (RNNs) or LSTMs (Long Short-Term Memory units) that are an extension of RNNs may be used. Furthermore, these functions may be performed by a learning unit of a learning device separate from the control device 4.

[0101] Those skilled in the art will readily derive further effects and modifications. Therefore, the broader aspects of the present invention are not limited to the specific details and representative embodiments shown and described above. Therefore, various modifications may be made without departing from the spirit or scope of the comprehensive inventive concept as defined by the appended claims and their equivalents.

[0102] Description of Reference Numerals

[0103] 1: Endoscope system; 2: Endoscope; 3: Display device; 4: Control device; 5: Laser irradiation device; 7: Display device; 21: Insertion portion; 22: Operation portion; 23: Universal cable; 24: Front end portion; 25: Bending portion; 26: Flexible tube portion; 27: Connector portion; 27a: Coiled cable; 28: Connector portion; 30: Converging lens; 31: First light source portion; 32: Second light source portion; 33: Light source control portion; 201: Illumination optical system; 202: Imaging optical system; 203: Cutoff filter; 204: Imaging element; 205: A / D Conversion unit; 206: P / S conversion unit; 207: Video recording unit; 208: Video control unit; 221: Bending knob; 222: Treatment instrument insertion unit; 223: Switch; 231: Light guide; 232: First transmission cable; 233: Second transmission cable; 401: S / P conversion unit; 402: Image processing unit; 402a: Image generation unit; 402b: Acquisition unit; 402c: Extraction unit; 402d: Determination unit; 402e: Output unit; 403: Input unit; 404: Recording unit; 404a: Program recording unit; 405: Control unit.

Claims

1. An image processing device comprising a processor, wherein the processor comprises hardware, The processor acquires a fluorescence image obtained by irradiating excitation light to biological tissue and capturing fluorescence. The processor extracts a first pixel having a brightness value greater than a first threshold value from the fluorescent image, The processor determines a first area based on the position information of the first pixel, The processor extracts a second pixel whose brightness value is below a second threshold value in the first area of ​​the fluorescent image, The processor determines a second area based on the position information of the second pixel, The processor outputs information after superimposing the second region on the fluorescent image.

2. The image processing apparatus according to claim 1, wherein: The first threshold is greater than the second threshold.

3. The image processing apparatus according to claim 1, wherein: The fluorescent image is an image obtained by capturing the fluorescence generated from a thermally denatured region of the biological tissue.

4. The image processing apparatus according to claim 3, wherein: The thermally denatured region is formed by subjecting the biological tissue to thermal treatment using an energy device.

5. The image processing apparatus according to claim 1, wherein: The first threshold and the second threshold are set according to the brightness value of the fluorescence generated from the advanced glycation end products produced by thermal denaturation of the biological tissue. The image processing apparatus according to claim 1 , wherein: The processor outputs information after the first region is superimposed on the fluorescent image.

7. The image processing apparatus according to claim 1, wherein: The processor outputs information obtained by superimposing the second region on a white light image obtained by irradiating white light onto the biological tissue and capturing return light.

8. The image processing apparatus according to claim 1, wherein: The processor outputs information obtained by superimposing the first region on a white light image captured by irradiating white light onto the biological tissue and capturing return light.

9. The image processing apparatus according to claim 1, wherein: The processor determines a circular area formed by connecting the first pixels as the first area.

10. A medical system comprising: a light source device for irradiating excitation light onto biological tissue; an endoscope including an imaging element configured to output an imaging signal capturing fluorescence emitted by the excitation light; and an image processing device having a processor for generating a fluorescence image based on the imaging signal, in, The processor extracts a first pixel having a brightness value greater than a first threshold value from the fluorescent image, The processor determines a first area based on the position information of the first pixel, The processor extracts a second pixel whose brightness value is below a second threshold value in the first area of ​​the fluorescent image, The processor determines a second area based on the position information of the second pixel, The processor outputs information after superimposing the second region on the fluorescent image.

11. An operating method of an image processing device, the image processing device comprising a processor, the processor comprising hardware, wherein: The processor acquires a fluorescence image obtained by irradiating excitation light to biological tissue and capturing fluorescence. The processor extracts a first pixel having a brightness value greater than a first threshold value from the fluorescent image, The processor determines a first area based on the position information of the first pixel, The processor extracts a second pixel whose brightness value is below a second threshold value in the first area of ​​the fluorescent image, The processor determines a second area based on the position information of the second pixel, The processor outputs information after superimposing the second region on the fluorescent image.

12. A learning device, A learning unit is provided for generating a learned model by performing machine learning using teaching data in which a fluorescence image obtained by irradiating biological tissue with excitation light and capturing fluorescence is used as input data, and information obtained by superimposing a second region determined based on positional information of a second pixel having a brightness value below a second threshold value within a first region on the fluorescence image is used as output data, wherein: The first region is a region determined based on first pixels having a brightness value greater than or equal to a first threshold value in the fluorescent image.

Images