Image processing device, medical system, image processing device operating method, and learning device

Abstract

The image processing device is provided with a processor having hardware. The processor performs: a process for acquiring a white light image obtained by irradiating a biological tissue with white light and capturing return light, and a fluorescence image obtained by irradiating the biological tissue with excitation light and capturing fluorescence; an incision line estimation unit that estimates an incision line, which is an auxiliary line for incision of the biological tissue, on the basis of position information of pixels having luminance values equal to or greater than a first threshold value in the fluorescence image; performing position alignment between the white light image and the fluorescence image; and outputting information obtained by superimposing the cut line on the white light image. As a result, provided is an image processing device capable of easily recognizing the cut-out line.

Description

Technical Field

[0001] The present invention relates to an image processing device, a medical system, a working method of the image processing device, and a learning device. Background Art

[0002] In recent years, minimally invasive treatments using endoscopes, laparoscopy, etc. have become widespread in the medical field. For example, endoscopic submucosal dissection (ESD) is widely performed as a minimally invasive treatment using endoscopes, laparoscopy, etc.

[0003] In ESD, multiple dot-shaped marks are formed by cauterizing biological tissue using an energy device or other device to surround the lesion to be removed. The tissue, thermally denatured by the marks, contains AGEs (advanced glycation end products), which fluoresce when irradiated with excitation light. This allows the location of the marks to be confirmed through fluorescence imaging (see, for example, Patent Document 1). The surgeon then uses the marked locations as markers to determine the incision line and makes an incision along the line to remove the lesion.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: International Publication No. 2020 / 054723 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] During incision, the tip of the endoscope is brought close to the biological tissue, so sometimes the endoscopic image only captures a portion of the marking, making it difficult to determine the incision line and direction based on the position of the marking. In such cases, the tip of the endoscope must be moved away from the biological tissue to confirm the entire marking, which is time-consuming.

[0009] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing device, a medical system, an operating method of the image processing device, and a learning device that can easily identify an incision line.

[0010] Solutions for solving problems

[0011] In order to solve the above-mentioned problems and achieve the purpose, an image processing device involved in one embodiment of the present invention includes a processor with hardware, wherein the processor performs the following processing: obtaining a white light image in which white light is irradiated onto biological tissue and return light is captured, and a fluorescence image in which excitation light is irradiated onto the biological tissue and fluorescence is captured; estimating an auxiliary line for incising the biological tissue, i.e., an incision line, based on position information of pixels having brightness values ​​greater than a first threshold in the fluorescence image; performing positional alignment between the white light image and the fluorescence image; and outputting information after the incision line is superimposed on the white light image.

[0012] In the image processing device according to one embodiment of the present invention, the processor performs the following processing: determining the position of a mark formed by cauterizing the living tissue based on position information of pixels having brightness values ​​greater than or equal to the first threshold value in the fluorescent image; and estimating the incision line based on the position of the mark.

[0013] In the image processing device according to one aspect of the present invention, the processor identifies a line that cuts the living tissue, ie, a completed incision line, based on position information of pixels having a luminance value equal to or greater than a second threshold value in the fluorescence image.

[0014] Furthermore, in the image processing device according to one aspect of the present invention, the second threshold is larger than the first threshold.

[0015] Furthermore, in the image processing device according to one aspect of the present invention, the processor acquires output information of an energy device.

[0016] Furthermore, in the image processing apparatus according to one aspect of the present invention, the output information includes a type of energy output by the energy device.

[0017] Furthermore, in the image processing device according to one aspect of the present invention, the processor sets the first threshold value based on the output information.

[0018] In addition, in the image processing device involved in one embodiment of the present invention, the processor performs the following processing: setting a second threshold value based on the output information; and determining a line that cuts the living tissue, i.e., a completed incision line, based on position information of pixels whose brightness values ​​are greater than the second threshold value in the fluorescent image.

[0019] Furthermore, in the image processing device according to one aspect of the present invention, the processor outputs information in which the cutting line and the completed cutting line are superimposed on the white-light image in different ways.

[0020] Furthermore, in the image processing device according to one aspect of the present invention, the processor stores the position information of the incision line and the position information of the completed incision line in a distinguishable manner.

[0021] Furthermore, in the image processing device according to one aspect of the present invention, the processor generates the fluorescence image based on an imaging signal obtained by irradiating the biological tissue with the excitation light and capturing the fluorescence.

[0022] In addition, in an image processing device involved in one embodiment of the present invention, the processor performs the following processing: obtaining a reference light image in which reference light is irradiated onto the biological tissue and the return light is photographed, wherein the reference light has a wavelength included in the wavelength range of the white light and does not include the wavelength range of the fluorescence; and performing positional alignment between the white light image and the reference light image.

[0023] Furthermore, in the image processing device according to one aspect of the present invention, the reference light image is captured simultaneously with the fluorescence image.

[0024] Furthermore, in the image processing device according to one aspect of the present invention, the processor extracts feature points from the white-light image.

[0025] Furthermore, in the image processing device according to one aspect of the present invention, the processor stores the position information of the feature points.

[0026] Furthermore, in the image processing device according to one aspect of the present invention, the processor estimates the incision line by comparing the position information of the feature point with position information of pixels having luminance values ​​equal to or greater than the first threshold value in the fluorescence image.

[0027] Furthermore, in the image processing device according to one embodiment of the present invention, the first threshold value is set to a value capable of extracting the fluorescence generated from advanced glycation end products generated by thermal denaturation of the biological tissue.

[0028] In addition, a medical system involved in one embodiment of the present invention comprises: a light source device that irradiates white light and excitation light to biological tissue; an endoscope having an imaging element that outputs a first imaging signal that captures return light of the white light and a second imaging signal that captures fluorescence based on the excitation light; and an image processing device having a processor that generates a white light image based on the first imaging signal and a fluorescence image based on the second imaging signal, wherein the processor performs the following processing: estimating an auxiliary line for incising the biological tissue, i.e., an incision line, based on position information of pixels having brightness values ​​greater than a first threshold in the fluorescence image; performing positional alignment between the white light image and the fluorescence image; and outputting information after the incision line is superimposed on the white light image.

[0029] In addition, an operating method of an image processing device involved in one embodiment of the present invention is an operating method of an image processing device having a processor with hardware, wherein the processor performs the following processing: obtaining a white light image in which white light is irradiated onto biological tissue and return light is captured, and a fluorescence image in which excitation light is irradiated onto the biological tissue and fluorescence is captured; estimating an auxiliary line for incising the biological tissue, i.e., an incision line, based on position information of pixels having brightness values ​​greater than a first threshold in the fluorescence image; performing positional alignment between the white light image and the fluorescence image; and outputting information after the incision line is superimposed on the white light image.

[0030] In addition, a learning device involved in one embodiment of the present invention includes a learning unit that generates a learned model by performing machine learning using training data, wherein the training data includes a white light image in which white light is irradiated onto biological tissue and return light is captured, and a fluorescence image in which excitation light is irradiated onto the biological tissue and fluorescence is captured, as input data, and information in which an auxiliary line for incising the biological tissue, i.e., an incision line, is superimposed on the white light image as output data.

[0031] Effects of the Invention

[0032] According to the present invention, it is possible to realize an image processing device, a medical system, an operating method of the image processing device, and a learning device that can easily recognize an incision line. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a diagram schematically showing the overall configuration of an endoscope system according to one embodiment.

[0034] Figure 2 This is a block diagram showing the functional configuration of a main portion of an endoscope system according to one embodiment.

[0035] Figure 3 This is a flowchart showing an overview of processing executed by the control device.

[0036] Figure 4 This is a diagram showing an example of a fluorescence image.

[0037] Figure 5 This is a diagram showing an example of a cutting line.

[0038] Figure 6 This is a diagram showing how a cutting line is superimposed on a white light image.

[0039] Figure 7 This is a diagram showing an example of a cut line.

[0040] Figure 8 This is a diagram showing a state where a portion of the cutting line is deleted.

[0041] Figure 9 This is a diagram showing an example of a cut line.

[0042] Figure 10 This is a diagram showing a state where a portion of the cutting line is deleted. DETAILED DESCRIPTION

[0043] Below, as a method for implementing the present disclosure (hereinafter referred to as an "embodiment"), an endoscope system having an insertion portion with a soft endoscope is described, but it is not limited to this. For example, even a rigid endoscope and a surgical robot can be applied. In addition, the present disclosure is not limited to this embodiment. Moreover, in the description of the drawings, the same parts are marked with the same figure marks for description. In addition, the drawings are schematic, and it is necessary to note that the relationship between the thickness and width of each component and the ratio of each component are different from the actual one. In addition, the drawings also include parts with different sizes or ratios.

[0044] [Structure of the endoscope system]

[0045] Figure 1 This is a diagram schematically showing the overall configuration of an endoscope system according to one embodiment. Figure 1 The endoscope system 1 shown in the figure inserts an insertion portion of an endoscope into a body cavity or lumen of a patient or other subject to be examined, thereby imaging the subject's interior and displaying an image based on the captured imaging signal on a display device. The endoscope system 1 includes an endoscope 2, a light source device 3, a control device 4 as an image processing device, and a display device 5.

[0046] [Structure of an endoscope]

[0047] First, the structure of the endoscope 2 will be described.

[0048] The endoscope 2 generates an imaging signal (RAW data) that captures the interior of the subject's body and outputs the generated imaging signal to the control device 4. Specifically, the endoscope 2 generates a first imaging signal that captures the return light by irradiating white light, and a second imaging signal that captures the fluorescence by irradiating excitation light. The endoscope 2 includes an insertion portion 21, an operating portion 22, and a universal cable 23.

[0049] Insertion section 21 is inserted into the subject. Insertion section 21 is flexible and elongated. It includes a distal end 24 housing an imaging element (described later); a freely bendable bending section 25 composed of multiple bendable pieces; and a long, flexible flexible tubular section 26 connected to the proximal end of bending section 25.

[0050] The distal end portion 24 is made of glass fiber, etc. It forms a light guide path for illumination light supplied from the control device 4 via the universal cable 23 and the operation unit 22 , and generates an imaging signal capturing return light of the illumination light and outputs it to the control device 4 .

[0051] The operating section 22 includes a bending knob 221 for bending the bending section 25 in the vertical and horizontal directions; a treatment instrument insertion section 222 for inserting a body treatment instrument; and a plurality of switches 223. These switches 223 serve as an operation input section for inputting operation instruction signals for the control device 4, operation instruction signals for peripheral devices such as the air supply unit, water supply unit, and gas supply unit, a pre-freeze signal for instructing the endoscope system 1 to capture a still image, or a switching signal for switching the observation mode of the endoscope system 1. A treatment instrument inserted through the treatment instrument insertion section 222 is exposed from an opening (not shown) through a treatment instrument channel (not shown) of the distal end portion 24.

[0052] The universal cable 23 contains at least a light guide and a bundled cable that bundles one or more cables. The bundled cable is a signal line that transmits and receives signals between the endoscope 2 and the control device 4. It includes a signal line for transmitting and receiving imaging signals (RAW data) and a signal line for transmitting and receiving timing signals (synchronization signals and clock signals) used to drive the imaging element (described later). The universal cable 23 has a connector 27 that is removably attached to the control device 4; a coiled cable 27a extending from the coiled cable 27a; and a connector 28 that is removably attached to the control device 4 at the extended end of the coil cable 27a.

[0053] [Structure of light source device]

[0054] Next, the structure of the light source device will be described.

[0055] The light source device 3 irradiates white light and excitation light as illumination light to the biological tissue. The light source device 3 is connected to one end of the light guide of the endoscope 2, and supplies the illumination light to be irradiated into the subject to the one end of the light guide under the control of the control device 4. The light source device 3 is implemented using the following components: one or more light sources such as LED (Light Emitting Diode: Light Emitting Diode) light source, xenon lamp and LD (laser Diode: Laser Diode) semiconductor laser element; a processor as a processing device having hardware such as FPGA (Field Programmable Gate Array: Field Programmable Gate Array) or CPU (Central Processing Unit: Central Processing Unit); and a memory as a temporary storage area used by the processor. In addition, the light source device 3 and the control device 4 can be as follows: Figure 1 Although the structure is set as a structure in which communication is performed individually as shown, it can also be an integrated structure.

[0056] [Structure of control device]

[0057] Next, the configuration of the control device 4 will be described.

[0058] The control device 4 controls each unit of the endoscope system 1 . The control device 4 supplies illumination light for the endoscope 2 to irradiate the subject. The control device 4 also performs various image processing on the imaging signal input from the endoscope 2 and outputs the image to the display device 5 .

[0059] [Structure of display device]

[0060] Next, the structure of the display device 5 will be described.

[0061] The display device 5 displays an image based on a video signal input from the control device 4 under the control of the control device 4. The display device 5 is implemented using a display panel such as an organic EL (Electro Luminescence) or a liquid crystal.

[0062] [Functional structure of the main parts of the endoscope system]

[0063] Next, the functional configuration of the main parts of the endoscope system 1 will be described. Figure 2 1 is a block diagram showing the functional configuration of the main parts of the endoscope system 1 .

[0064] [Structure of an endoscope]

[0065] First, the structure of the endoscope 2 will be described.

[0066] The endoscope 2 includes an illumination optical system 201, an imaging optical system 202, a cutoff filter 203, an imaging element 204, an A / D converter 205, a P / S converter 206, an imaging recording unit 207, and an imaging control unit 208. Furthermore, the illumination optical system 201, the imaging optical system 202, the cutoff filter 203, the imaging element 204, the A / D converter 205, the P / S converter 206, the imaging recording unit 207, and the imaging control unit 208 are each disposed within the distal end portion 24.

[0067] The illumination optical system 201 irradiates the subject (biological tissue) with illumination light supplied from a light guide 231 formed of an optical fiber or the like. The illumination optical system 201 is realized using one or a plurality of lenses or the like.

[0068] The imaging optical system 202 focuses reflected light from the subject, return light from the subject, fluorescence emitted by the subject, and other light to form a subject image (light rays) on the light receiving surface of the imaging element 204. The imaging optical system 202 is implemented using one or more lenses.

[0069] The cutoff filter 203 is disposed on the optical axis O1 between the imaging optical system 202 and the imaging element 204. The cutoff filter 203 shields light in the wavelength range of the excitation light supplied from the light source device 3, reflected light from the subject, or return light, and transmits light in the wavelength range on the long-wavelength side that is longer than the wavelength range of the excitation light.

[0070] Under the control of the imaging control unit 208, the imaging element 204 receives the subject image (light) formed by the imaging optical system 202 and transmitted through the cut filter 203, performs photoelectric conversion to generate an imaging signal (RAW data), and outputs it to the A / D converter 205. The imaging element 204 is implemented using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor, in which one of the color filters forming a Bayer array (RGGB) is arranged at each of a plurality of pixels arranged in a two-dimensional matrix.

[0071] Under the control of the imaging control unit 208, the A / D converter 205 performs A / D conversion processing on the analog imaging signal input from the imaging element 204 and outputs the signal to the P / S converter 206. The A / D converter 205 is implemented using an A / D conversion circuit or the like.

[0072] Under the control of the camera control unit 208, the P / S converter 206 performs parallel / serial conversion on the digital camera signal input from the A / D converter 205, and outputs the parallel / serial converted camera signal to the control device 4 via the first transmission cable 232. The P / S converter 206 is implemented using a P / S conversion circuit, etc. In the first embodiment, an E / O converter that converts the camera signal into an optical signal may be provided instead of the P / S converter 206, and the camera signal may be output to the control device 4 via an optical signal. For example, the camera signal may be transmitted to the control device 4 via wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0073] The image recording unit 207 records various information related to the endoscope 2 (e.g., pixel information of the image sensor 204 and characteristics of the cutoff filter 203). Furthermore, the image recording unit 207 records various setting data and control parameters transmitted from the control device 4 via the second transmission cable 233. The image recording unit 207 is configured using nonvolatile memory and volatile memory.

[0074] The imaging control unit 208 controls the operations of the imaging element 204, the A / D converter 205, and the P / S converter 206 based on the setting data received from the control device 4 via the second transmission cable 233. The imaging control unit 208 is implemented using a TG (Timing Generator), a processor as a processing device including hardware such as a CPU, and a memory as a temporary storage area used by the processor.

[0075] [Structure of light source device]

[0076] Next, the structure of the light source device 3 will be described.

[0077] The light source device 3 includes a condenser lens 30 , a first light source unit 31 , a second light source unit 32 , and a light source control unit 33 .

[0078] The condenser lens 30 condenses the light emitted from the first light source unit 31 and the second light source unit 32 and emits the light toward the light guide 231. The condenser lens 30 is formed using one or a plurality of lenses.

[0079] Under the control of the light source control unit 33, the first light source unit 31 supplies white light as illumination light to the light guide 231 by emitting white light (ordinary light) which is visible light. The first light source unit 31 is constructed using a collimating lens, a white LED lamp, a driver for driving, and the like. In addition, the first light source unit 31 can also supply white light of visible light by using a red LED lamp, a green LED lamp, and a blue LED lamp to emit light simultaneously. Of course, the first light source unit 31 can also be constructed using a halogen lamp or a xenon lamp, and the like. In addition, the first light source unit 31 can also irradiate a reference light having a wavelength that is included in the wavelength range of white light and does not include the wavelength range of fluorescence. By irradiating the reference light and the excitation light irradiated by the second light source unit 32 simultaneously and photographing them, it is possible to perform position alignment between the white light image and the reference light image.

[0080] Under the control of the light source control unit 33, the second light source unit 32 supplies excitation light as illumination light to the light guide 231 by emitting excitation light having a predetermined wavelength range. Here, the excitation light has a wavelength that excites substances such as advanced glycation end products (AGEs) contained in the thermally denatured region, for example, a wavelength range of 400 nm to 430 nm (center wavelength of 415 nm). The thermally denatured region is a region where biological tissue is denatured by heat treatment with an energy device such as a high-frequency knife. The excitation light irradiated by the second light source unit 32 is shielded by the cutoff filter 203, and the fluorescence (wavelength 540 nm) generated from the AGEs passes through the cutoff filter 203, so that a fluorescence image can be captured. The second light source unit 32 is implemented using a semiconductor laser such as a collimating lens, a violet LD (Laser Diode), and a driver for driving.

[0081] The light source control unit 33 is configured using a processor as a processing device, such as a hardware device such as an FPGA (Field-Programmable Gate Array) or a CPU (Central Processing Unit), and a memory serving as a temporary storage area used by the processor. Based on control data input from the control unit 405, the light source control unit 33 controls the light emission timing, light emission intensity, and light emission duration of each of the first light source unit 31 and the second light source unit 32.

[0082] [Structure of control device]

[0083] Next, the configuration of the control device 4 will be described.

[0084] The control device 4 includes an S / P conversion unit 401 , an image processing unit 402 , an input unit 403 , a recording unit 404 , and a control unit 405 .

[0085] Under the control of the control unit 405, the S / P converter 401 performs serial / parallel conversion on the imaging signal received from the endoscope 2 via the first transmission cable 232 and outputs the serial / parallel signal to the image processing unit 402. Furthermore, if the endoscope 2 outputs the imaging signal as an optical signal, an O / E converter that converts the optical signal into an electrical signal may be provided in place of the S / P converter 401. Furthermore, if the endoscope 2 transmits the imaging signal via wireless communication, a communication module capable of receiving wireless signals may be provided in place of the S / P converter 401.

[0086] The image processing unit 402 is implemented using the following components: a processor comprising hardware such as a CPU, GPU (Graphics Processing Unit), or FPGA; and a memory serving as a temporary storage area used by the processor. Under the control of the control unit 405, the image processing unit 402 performs predetermined image processing on the imaging signal input from the S / P conversion unit 401 and outputs the image to the display device 5. The image processing unit 402 generates a white light image based on the first imaging signal and a fluorescence image based on the second imaging signal. The image processing unit 402 includes an image generation unit 402a, an acquisition unit 402b, a determination unit 402c, an estimation unit 402d, an adjustment unit 402e, and an output unit 402f.

[0087] The image generating unit 402a generates a white-light image based on a first imaging signal generated by irradiating white light from the first light source unit 31 onto living tissue and capturing the return light. Furthermore, the image generating unit 402a generates a fluorescence image based on a second imaging signal generated by irradiating excitation light from the second light source unit 32 onto living tissue and capturing the fluorescence. Furthermore, the image generating unit 402a may also generate a reference-light image based on a third imaging signal generated by irradiating reference light from the first light source unit 31 onto living tissue and capturing the return light.

[0088] The acquisition unit 402 b acquires the white-light image, the fluorescent image, and the reference-light image from the image generation unit 402 a .

[0089] The determination unit 402c determines the location of a mark formed by cauterizing biological tissue based on the positional information of pixels in the fluorescence image whose brightness values ​​exceed a first threshold. The first threshold is set to a value sufficient to detect fluorescence generated by advanced glycation end products (AGEs) produced by thermal denaturation of biological tissue. Consequently, the location of a mark containing AGEs can be determined. Alternatively, the determination unit 402c can determine the line through which the biological tissue was incised, i.e., the completed incision line, based on the positional information of pixels in the fluorescence image whose brightness values ​​exceed a second threshold. Furthermore, the second threshold is set to a value greater than the first threshold. Since the output of the energy device for incision is greater than that for marking, more AGEs are generated during incision, and the amount of fluorescence increases. Therefore, by setting the second threshold greater than the first threshold, the location of the mark can be distinguished from the completed incision line.

[0090] The estimation unit 402d estimates the incision line, which is an auxiliary line for incising the living tissue, based on the position of the marker. Alternatively, when the distal end of the endoscope is close to the living tissue and only a portion of the marker is captured in the endoscopic image, the estimation unit 402d may estimate the incision line by comparing the positional information of characteristic points (characteristic points of the image, such as the end of a lesion or a bleeding point) in the captured portion of the living tissue in the endoscopic image with the positional information of pixels having a brightness value greater than a first threshold value in the fluorescence image.

[0091] Adjustment unit 402e performs positional alignment between the white-light image and the fluorescence image. It extracts feature points from the white-light image and the fluorescence image and performs alignment so that the positions of the feature points correspond. Adjustment unit 402e can also perform positional alignment between the white-light image and the reference light image. The reference light image is captured simultaneously with the fluorescence image, allowing alignment between the white-light image and the fluorescence image to be performed using the reference light image, thereby improving alignment accuracy.

[0092] The output unit 402f outputs information resulting from superimposing the incision line on the white light image. For example, the output unit 402f outputs a display control signal causing the display device 5 to display an image resulting from superimposing the incision line on the white light image. Alternatively, the output unit 402f may output information resulting from superimposing the completed incision line on the white light image. For example, the output unit 402f outputs a display control signal causing the display device 5 to display an image resulting from superimposing the completed incision line on the white light image. Furthermore, the output unit 402f may output information resulting from superimposing the incision line and the completed incision line on the white light image in different manners.

[0093] The input unit 403 receives input of various operations related to the endoscope system 1 and outputs the received operations to the control unit 405. The input unit 403 is configured using a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, and the like.

[0094] The recording unit 404 is implemented using a recording medium such as a volatile memory, a non-volatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory card. The recording unit 404 records data including various parameters required for the operation of the endoscope system 1. The recording unit 404 stores, for example, position information of feature points of a white light image, a fluorescent image, and a reference light image. Furthermore, the recording unit 404 stores, for example, position information of an incision line and position information of an already incision line in an identifiable manner. Furthermore, the recording unit 404 includes a program recording unit 404a that records various programs for operating the endoscope system 1.

[0095] The control unit 405 is realized by using a processor having hardware such as an FPGA or a CPU, and a memory serving as a temporary storage area used by the processor. The control unit 405 comprehensively controls the various components constituting the endoscope system 1 .

[0096] 〔Control device processing〕

[0097] Next, the processing executed by the control device 4 will be described.

[0098] Figure 3 FIG. 1 is a flowchart showing an outline of the processing performed by the control device. Figure 3 As shown, first, the image generating unit 402 a generates a white-light observation image based on a first imaging signal obtained by irradiating white light from the first light source unit 31 to biological tissue and capturing the white light (step S1 ).

[0099] Then, the image generating unit 402 a generates a fluorescence image based on the second imaging signal obtained by irradiating the biological tissue with excitation light from the second light source unit 32 and capturing fluorescence (step S2 ). Figure 4 FIG is a diagram showing an example of a fluorescence image. Figure 4 As shown, the fluorescent image FI1 includes markers M1 to M6.

[0100] Next, the identification unit 402c identifies the positions of the marks M1-M6 formed by cauterizing the living tissue based on the positional information of pixels in the fluorescence image FI1 whose brightness values ​​exceed the first threshold value (step S3). Because the marks M1-M6 formed by cauterizing the living tissue using the energy device contain fluorescent advanced glycation end products, the marks M1-M6 can be identified as positions where the brightness value of the fluorescence image FI1 exceeds the first threshold value.

[0101] Thereafter, the estimation unit 402d estimates an incision line, which is an auxiliary line for incising the living tissue, based on the positions of the markers M1 to M6 (step S4). Figure 5 : is a diagram showing an example of a cutting line. Figure 5 As shown, the estimating unit 402d estimates, for example, a curved line connecting the markers M1 to M6 as the cutting line L1.

[0102] Next, the adjustment unit 402e performs positional alignment between the white-light image and the fluorescent image (step S5).

[0103] Then, the output unit 402f outputs a display control signal for causing the display device 5 to display an image in which the cutting line is superimposed on the white light image (step S6). Figure 6 is a diagram showing a situation where a cutting line is superimposed on a white light image. Figure 6 As shown, the output unit 402 f outputs a display control signal for causing the display device 5 to display an image in which the positions of the markers M11 to M1N and the cutting line L2 are superimposed on the white light image WL1 .

[0104] The operator then incises the area surrounding the lesion using the incision line as a reference. At this point, the identification unit 402c identifies the line where the biological tissue has been incised, or the completed incision line, based on the positional information of pixels in the fluorescence image whose brightness values ​​exceed the second threshold (step S7). Because the area where the biological tissue was incised by the energy device contains a greater amount of advanced glycation end products than the marked area, this area can be identified as a location where the brightness value of the fluorescence image FI1 exceeds the second threshold, which is greater than the first threshold. Figure 7 : is a diagram showing an example of a cut line. Figure 7 As shown, the identifying unit 402c identifies the incised line L11 in the fluorescent image FI1.

[0105] Then, the output unit 402f outputs a display control signal for causing the display device 5 to display an image in which the incision line is superimposed on the white light image (step S8).

[0106] Alternatively, the output unit 402f may output a display control signal for causing the display device 5 to display an image superimposed on the white light image in such a manner that a portion of the incision line is deleted in accordance with the incision line. Figure 8FIG is a diagram showing a situation where a portion of the cutting line is deleted. Figure 8 As shown, the output unit 402f can also delete the incision line of the already incised area A1 from the fluorescence image FI1. Specifically, the output unit 402f simply outputs a display control signal that causes the display device 5 to display an image in which the incision line L3 and the already incised line L11 are superimposed on the white light image. The incision line L3 is the incision line obtained by deleting the incision line between the markers M1 and M2 through which the already incised line L11 passes. This prevents the incision line and the already incised line from overlapping and becoming difficult to discern.

[0107] Figure 9 : is a diagram showing an example of a cut line. Figure 9 As shown, there is a case where the cut line L12 does not pass through the adjacent mark M2 after passing through the mark M1. Figure 10 FIG is a diagram showing a situation where a portion of the cutting line is deleted. Figure 10 As shown, the output unit 402f can also delete the incision line of the incised area A2 in the fluorescence image FI1. Specifically, if the incision line L12 passes through a predetermined line (for example, such that the midpoint of the line connecting markers M1 and M3 coincides with the line connecting marker M2), the output unit 402f can simply delete the incision line of area A2. In this case, the output unit 402f can also connect the end of the incision line L12 to the end of the incision line L4. Furthermore, the output unit 402f can simply output a display control signal that causes the display device 5 to display an image in which the incision line L4 and the incision line L12 are superimposed on the white light image. As a result, it is possible to prevent the incision line and the incision line from overlapping and becoming difficult to discern.

[0108] According to the embodiment described above, the incision line is superimposed on the white light image, so the operator can easily recognize the incision line.

[0109] In addition, the completed incision line is superimposed on the white light image, so the operator can easily recognize the positional relationship between the completed incision line and the incision line.

[0110] [Variation]

[0111] In the control device 4 of the modified example, the acquisition unit 402b acquires the output information of the energy device used for marking. The output information may also include the type of energy output by the energy device. In addition, the type of energy may be, for example, high-frequency waves, ultrasonic waves, or microwaves.

[0112] The determination unit 402c sets a first threshold value based on the output information acquired by the acquisition unit 402b. The determination unit 402c then determines the position of the mark formed by cauterizing the living tissue based on the position information of pixels in the fluorescence image whose brightness values ​​are greater than the set first threshold value.

[0113] According to the modification described above, the first threshold value is set based on the output information of the energy device used for marking, and therefore the incision line can be determined with higher accuracy.

[0114] Alternatively, the acquisition unit 402b may acquire output information from the energy device used for incision. In this case, the determination unit 402c sets a second threshold based on the output information acquired by the acquisition unit 402b. Furthermore, the determination unit 402c determines the line that incised the living tissue, i.e., the completed incision line, based on the positional information of pixels in the fluorescence image whose brightness values ​​exceed the second threshold. As a result, the incision line and the completed incision line can be distinguished based on the output information from the energy device.

[0115] Alternatively, the control unit 405 may also function as a learning unit of a learning device, which is the control unit 4. The control unit 405 may also generate a learned model by performing machine learning using training data. The training data includes white light images obtained by irradiating biological tissue with white light and capturing the return light, and fluorescence images obtained by irradiating biological tissue with excitation light and capturing the fluorescence. The output data includes information obtained by superimposing the incision line, an auxiliary line for incising the biological tissue, on the white light image. The learned model is constructed from a neural network with each layer having one or more nodes. The type of machine learning is not particularly limited. For example, training data and learning data may be prepared by pairing white light images and fluorescence images of multiple subjects with images obtained by superimposing the incision line estimated from these multiple white light and fluorescence images on the white light image. This training data and learning data may then be input into a computational model based on a multi-layer neural network for learning. Machine learning techniques, for example, include techniques based on multi-layer neural networks such as CNNs (Convolutional Neural Networks) and 3D-CNNs (Deep Neural Networks). Furthermore, as a machine learning technique, techniques based on recurrent neural networks (RNNs) or LSTMs (Long Short-Term Memory units) that are an extension of RNNs may also be used. Furthermore, these functions may be performed by a learning unit of a learning device separate from the control device 4.

[0116] Further effects and modifications can be readily derived by those skilled in the art. Therefore, the broader aspects of the present invention are not limited to the specific details and representative embodiments shown and described above. Therefore, various modifications can be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.

[0117] Description of Reference Numerals

[0118] 1: Endoscope system; 2: Endoscope; 3: Display device; 4: Control device; 5: Laser irradiation device; 7: Display device; 21: Insertion portion; 22: Operation portion; 23: Universal cable; 24: Distal end portion; 25: Bend portion; 26: Flexible tube portion; 27: Connector portion; 27a: Coil cable; 28: Connector portion; 30: Converging lens; 31: First light source portion; 32: Second light source portion; 33: Light source control portion; 201: Illumination optical system; 202: Imaging optical system; 203: Cutoff filter; 204: Imaging element; 205: A / D converter; 206: P / S conversion unit; 207: Video recording unit; 208: Video control unit; 221: Bending knob; 222: Treatment instrument insertion unit; 223: Switch; 231: Light guide; 232: First transmission cable; 233: Second transmission cable; 401: S / P conversion unit; 402: Image processing unit; 402a: Image generation unit; 402b: Acquisition unit; 402c: Determination unit; 402d: Estimation unit; 402e: Adjustment unit; 402f: Output unit; 403: Input unit; 404: Recording unit; 404a: Program recording unit; 405: Control unit.

Claims

1. An image processing device comprising a processor having hardware, wherein: The processor performs the following processing: acquiring a white light image obtained by irradiating white light onto biological tissue and capturing return light, and a fluorescence image obtained by irradiating excitation light onto the biological tissue and capturing fluorescence; estimating an auxiliary line for incising the living tissue, namely, an incision line, based on position information of pixels having a brightness value greater than a first threshold value in the fluorescent image; performing positional alignment between the white light image and the fluorescent image; and Information obtained by superimposing the cutting line on the white light image is output.

2. The image processing apparatus according to claim 1, wherein: The processor performs the following processing: determining a position of a mark formed by cauterizing the living tissue based on position information of pixels in the fluorescent image whose brightness values ​​are greater than or equal to the first threshold; as well as The incision line is estimated based on the position of the marker.

3. The image processing apparatus according to claim 2, wherein: The processor identifies a line that cuts the living tissue, ie, a completed incision line, based on position information of pixels having a brightness value equal to or greater than a second threshold value in the fluorescent image.

4. The image processing apparatus according to claim 3, wherein: The second threshold is greater than the first threshold.

5. The image processing apparatus according to claim 1, wherein: The processor obtains output information of the energy device. The image processing apparatus according to claim 5 , wherein: The output information includes the type of energy output by the energy device.

7. The image processing apparatus according to claim 5, wherein: The processor sets the first threshold according to the output information.

8. The image processing apparatus according to claim 7, wherein: The processor performs the following processing: setting a second threshold according to the output information; and A line through which the living tissue is incised, ie, a completed incision line, is determined based on position information of pixels having a brightness value equal to or greater than the second threshold value in the fluorescent image.

9. The image processing apparatus according to claim 3 or 8, wherein: The processor outputs information obtained by superimposing the incision line and the already incised line on the white light image in different ways.

10. The image processing apparatus according to claim 3 or 8, wherein: The processor stores the position information of the incision line and the position information of the completed incision line in an identifiable manner.

11. The image processing apparatus according to claim 1, wherein: The processor generates the fluorescence image based on an imaging signal obtained by irradiating the biological tissue with the excitation light and capturing the fluorescence.

12. The image processing apparatus according to claim 1, wherein: The processor acquires a reference light image obtained by irradiating the biological tissue with reference light and capturing return light, wherein the reference light has a wavelength included in the wavelength range of the white light and excluding the wavelength range of the fluorescence; and Position alignment is performed between the white light image and the reference light image.

13. The image processing apparatus according to claim 12, wherein: The reference light image is captured simultaneously with the fluorescence image.

14. The image processing apparatus according to claim 1, wherein: The processor extracts feature points from the white light image.

15. The image processing apparatus according to claim 14, wherein: The processor stores position information of the feature points.

16. The image processing apparatus according to claim 15, wherein: The processor estimates the incision line by comparing the position information of the feature point with the position information of pixels having a brightness value equal to or greater than the first threshold value in the fluorescent image.

17. The image processing apparatus according to claim 1, wherein: The first threshold value is set to a value capable of extracting the fluorescence generated from advanced glycation end products generated by thermal denaturation of the biological tissue.

18. A medical system comprising: a light source device for irradiating white light and excitation light toward biological tissue; an endoscope including an imaging element configured to output a first imaging signal capturing an image of return light of the white light and a second imaging signal capturing an image of fluorescence based on the excitation light; and An image processing device including a processor configured to generate a white light image based on the first imaging signal and a fluorescence image based on the second imaging signal. in, The processor performs the following processing: estimating an auxiliary line for incising the living tissue, namely, an incision line, based on position information of pixels having a brightness value greater than a first threshold value in the fluorescent image; performing positional alignment between the white light image and the fluorescent image; and Information obtained by superimposing the cutting line on the white light image is output.

19. An operating method of an image processing device, the image processing device comprising a processor having hardware, wherein: The processor performs the following processing: acquiring a white light image obtained by irradiating white light onto biological tissue and capturing return light, and a fluorescence image obtained by irradiating excitation light onto the biological tissue and capturing fluorescence; estimating an auxiliary line for incising the living tissue, namely, an incision line, based on position information of pixels having a brightness value greater than a first threshold value in the fluorescent image; performing positional alignment between the white light image and the fluorescent image; and Information obtained by superimposing the cutting line on the white light image is output.

20. A learning device, wherein: The learning device includes a learning unit that generates a learned model by performing machine learning using training data, wherein the training data includes a white light image obtained by irradiating white light onto biological tissue and capturing return light, and a fluorescence image obtained by irradiating excitation light onto the biological tissue and capturing fluorescence, as input data, and information obtained by superimposing an auxiliary line for incising the biological tissue, i.e., an incision line, on the white light image as output data.

Images