Medical device, medical system, learning device, method for operating medical device, and program

Abstract

Provided are a medical device, a medical system, a learning device, a method for operating the medical device, and a program with which it is possible to ascertain whether or not a thermal change region is present outside a region of interest. This medical device is provided with a processor that: acquires setting information in which a region of interest is set for biological tissue, and thermal denaturation information relating to a thermal denaturation region in which thermal denaturation has occurred due to thermal treatment of the biological tissue; determining whether or not there is a thermally denatured region outside the region of interest on the basis of the setting information and the thermally denatured information; and when it is determined that the thermally denatured region exists outside the region of interest, outputting support information indicating that the thermally denatured region exists outside the region of interest.

Description

Technical Field

[0001] The present disclosure relates to a medical device, a medical system, a learning device, an operating method of the medical device, and a program. Background Art

[0002] Conventionally, a technique for visualizing the state of cauterization of a subject, such as biological tissue, using an energy device or the like is known in the medical field (see, for example, Patent Document 1). This technique irradiates the subject with excitation light and displays an image and information containing fluorescence image data generated based on an imaging signal acquired by capturing fluorescence generated from a heat-affected region of the subject in response to the excitation light, thereby visualizing the state of cauterization for a user, such as a surgeon.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2020 / 054723 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] In addition, in transurethral resection of bladder tumors (TUR-Bt), a surgical endoscope (resectoscope) is inserted through the urethra of the subject. The surgeon uses the eyepiece of the surgical endoscope to observe the lesion while using resection instruments such as energy devices to perform resection of the area of ​​interest including the lesion or a specified organ.

[0008] However, Patent Document 1 mentioned above does not consider the presence or absence of a thermally denatured region outside the region of interest at all, and a technique capable of grasping the presence or absence of a thermally denatured region outside the region of interest has been desired.

[0009] The present disclosure has been made in view of the above, and an object thereof is to provide a medical device, a medical system, a learning device, an operating method of a medical device, and a program that can detect the presence or absence of a thermally denatured region outside a region of interest.

[0010] Solutions for solving problems

[0011] In order to solve the above-mentioned problems and achieve the purpose, the medical device involved in the present disclosure is a medical device equipped with a processor, wherein the processor performs the following processing: obtaining setting information for setting a region of interest for biological tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to heat treatment of the biological tissue; determining whether the thermally denatured region exists outside the region of interest based on the setting information and the thermal denaturation information; and, if it is determined that the thermally denatured region exists outside the region of interest, outputting auxiliary information indicating that the thermally denatured region exists outside the region of interest.

[0012] In the medical device according to the present disclosure, in the above disclosure, the processor acquires a first image obtained by capturing the living tissue, and the first image includes the setting information.

[0013] In addition, in the medical device involved in the present disclosure, in the above disclosure, the first image is a special light image generated based on a camera signal, the camera signal is generated by capturing return light generated by irradiating the biological tissue with special light, the setting information is a feature quantity contained in the special light image, and the processor sets the area of ​​interest based on the feature quantity contained in the special light image.

[0014] In addition, in the above-mentioned disclosure of the medical device involved in the present disclosure, the first image is a white light image generated based on a camera signal, and the camera signal is generated by capturing the return light generated by irradiating the biological tissue with white light, and the processor performs the following processing: obtaining the position information output by the learning model as the setting information, the learning model is a model that uses a plurality of images and annotation information of the region of interest contained in each of the plurality of images to establish corresponding training data for machine learning, takes the white light image as input data and outputs the position of the region of interest in the white light image as output data; and sets the region of interest based on the setting information.

[0015] In addition, in the medical device involved in the present disclosure, in the above disclosure, the first image is a white light image generated based on a camera signal, and the camera signal is generated by capturing return light generated by irradiating white light on the biological tissue, and the processor performs the following processing: obtaining instruction information for indicating annotations input from the outside for the white light image as the setting information; and setting the area of ​​interest based on the setting information.

[0016] Furthermore, in the medical device according to the present disclosure, in the above disclosure, the processor sets the thermal denaturation region based on instruction information for instructing annotations input from the outside.

[0017] Furthermore, in the medical device according to the present disclosure, in the above disclosure, the processor acquires the thermal denaturation information from a second image obtained by capturing the living tissue.

[0018] In addition, in the medical device involved in the present disclosure, in the above disclosure, the second image is a fluorescence image generated based on a camera signal, and the camera signal is generated by capturing light emitted from the thermal denaturation area due to irradiation of the biological tissue with excitation light, and the thermal denaturation area is an area that emits fluorescence.

[0019] In the medical device according to the present disclosure, it is determined whether the signal value of each pixel constituting the fluorescence image is equal to or greater than a predetermined threshold value, and pixels having the signal value equal to or greater than the predetermined threshold value are identified as the thermally denatured region.

[0020] In the medical device according to the present disclosure, the processor performs the following processing: acquiring a reference image based on an imaging signal obtained by imaging the living tissue before the thermal treatment; and specifying the thermally denatured region based on the reference image and the second image.

[0021] In the medical device according to the present disclosure, in the above disclosure, the processor performs the following processing: generating a display image indicating that the thermally denatured region exists outside the region of interest; and outputting the display image as the auxiliary information.

[0022] In the medical device according to the present disclosure, the processor performs the following processing: generating a display image capable of distinguishing the thermally denatured region generated outside the region of interest from the thermally denatured region within the region of interest; and outputting the display image as the auxiliary information.

[0023] In addition, in the medical device involved in the present disclosure, the processor performs the following processing: acquiring a white light image generated based on a camera signal, wherein the camera signal is a camera signal generated by capturing return light generated by irradiating white light to the biological tissue; generating the display image by superimposing the thermal denaturation area generated outside the area of ​​interest and the thermal denaturation area within the area of ​​interest on the white light image in a manner that can be distinguished; and outputting the display image as the auxiliary information.

[0024] In the medical device according to the present disclosure, in the above disclosure, the processor performs the following processing: generating a display image capable of distinguishing the region of interest from the thermally denatured region; and outputting the display image as the auxiliary information.

[0025] In the medical device according to the present disclosure described above, the processor outputs position information of the thermally denatured region outside the region of interest as the auxiliary information to a projection device capable of projecting information onto the living tissue.

[0026] In addition, the medical system according to the present disclosure is a medical system including a light source device, an imaging device, and a medical device, wherein the light source device includes: a special light source that generates special light for biological tissue; and an excitation light source that generates excitation light that excites advanced glycation end products generated by heat treatment of the biological tissue; the imaging device includes an imaging element that generates an imaging signal by capturing return light or luminescence from the biological tissue irradiated with the special light or the excitation light; and the medical device includes a processor that performs the following processing: acquiring setting information that sets a region of interest for the biological tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to the heat treatment of the biological tissue; determining whether the thermally denatured region exists outside the region of interest based on the setting information and the thermal denaturation information; and, if it is determined that the thermally denatured region exists outside the region of interest, outputting auxiliary information indicating that the thermally denatured region exists outside the region of interest.

[0027] In addition, the operating method of the medical device involved in the present disclosure is an operating method of a medical device including a processor, wherein the processor performs the following processing: obtaining setting information for setting a region of interest for biological tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to heat treatment of the biological tissue; determining whether the thermally denatured region exists outside the region of interest based on the setting information and the thermal denaturation information; and, if it is determined that the thermally denatured region exists outside the region of interest, outputting auxiliary information indicating that the thermally denatured region exists outside the region of interest.

[0028] In addition, the program involved in the present disclosure is a program executed by a medical device including a processor, wherein the program causes the processor to perform the following processing: acquiring setting information for setting a region of interest for living tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to heat treatment of the living tissue; determining whether the thermally denatured region exists outside the region of interest based on the setting information and the thermal denaturation information; and, if it is determined that the thermally denatured region exists outside the region of interest, outputting auxiliary information indicating that the thermally denatured region exists outside the region of interest.

[0029] Effects of the Invention

[0030] According to the present disclosure, it is possible to understand whether or not there is a thermally altered region outside the region of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a diagram showing a schematic configuration of an endoscope system according to the first embodiment.

[0032] Figure 2 This is a block diagram showing the functional configuration of the main parts of the endoscope system according to the first embodiment.

[0033] Figure 3 This is a diagram schematically showing the wavelength characteristics of the excitation light emitted by the light source unit according to the second embodiment of the first embodiment.

[0034] Figure 4 This is a diagram schematically showing the structure of a pixel portion according to Embodiment 1.

[0035] Figure 5 This is a diagram schematically showing the structure of the color filter according to Embodiment 1.

[0036] Figure 6 This is a diagram schematically showing the sensitivity and wavelength range of each filter according to the first embodiment.

[0037] Figure 7A Schematically shows the signal value of the R pixel of the image sensor according to the first embodiment.

[0038] Figure 7B Schematically showing the signal values ​​of the G pixels of the image sensor according to the first embodiment.

[0039] Figure 7C This is a diagram schematically showing the signal value of the B pixel of the image sensor according to the first embodiment.

[0040] Figure 8 This is a diagram schematically showing the structure of the cut filter according to the first embodiment.

[0041] Figure 9 This is a diagram schematically showing the transmission characteristics of the cut filter according to the first embodiment.

[0042] Figure 10 This is a diagram schematically showing the transmission characteristics of the cut filter according to the first embodiment.

[0043] Figure 11 This is a flowchart showing an overview of processing executed by the control device 9 according to the first embodiment.

[0044] Figure 12This is a diagram schematically showing the region of interest set by the setting unit 953 according to the first embodiment for the first image.

[0045] Figure 13 Schematically shows the thermally denatured region identified by the identification unit 954 according to the first embodiment for the second image.

[0046] Figure 14 This is a diagram schematically showing the alignment process performed by the alignment unit 955 according to the first embodiment.

[0047] Figure 15 This is a flowchart showing an overview of processing executed by the control device 9 according to the second embodiment.

[0048] Figure 16 This is a diagram showing a schematic configuration of an endoscope system according to a third embodiment.

[0049] Figure 17 This is a flowchart showing an outline of processing executed by the control device 9 according to the third embodiment.

[0050] Figure 18 This is a diagram showing a schematic configuration of an endoscope system according to a fourth embodiment.

[0051] Figure 19 This is a block diagram showing the functional configuration of the medical device 13 according to the fourth embodiment.

[0052] Figure 20 This is a diagram illustrating the functional configuration of the main parts of an endoscope system 1C according to the fifth embodiment. DETAILED DESCRIPTION

[0053] The following is a method for implementing the present disclosure and the attached Figure 1 Detailed description will be given below. In addition, the present disclosure is not limited to the following embodiments. In addition, the figures referred to in the following description are merely schematic illustrations of shapes, sizes and positional relationships to the extent that the contents of the present disclosure can be understood. That is, the present disclosure is not limited to the shapes, sizes and positional relationships illustrated in the figures. Moreover, in the description of the drawings, the same parts are marked with the same figure numbers for description. In addition, as an example of the endoscope system involved in the present disclosure, an endoscope system including a rigid endoscope and a medical imaging device is described.

[0054] (Implementation Method 1)

[0055] [Structure of the endoscope system]

[0056] Figure 1 This is a diagram showing a schematic configuration of an endoscope system according to the first embodiment. Figure 1The endoscope system 1 shown is a system used in the medical field to observe and treat biological tissues in a subject such as a living body. Figure 1 The rigid endoscope system shown in FIG. 1 is a rigid endoscope system having a rigid endoscope (insertion portion 2), but the present invention is not limited thereto and may also be an endoscope system having a flexible endoscope. Furthermore, the endoscope system 1 may also be applied to a medical microscope or medical surgical robot system that includes a medical imaging device for imaging a subject and performs surgery or treatment while displaying an observation image based on an imaging signal (image data) captured by the medical imaging device on a display device.

[0057] Furthermore, in recent years, minimally invasive treatments using endoscopes and laparoscopy have become increasingly common in the medical field. For example, widely performed minimally invasive treatments using endoscopes and laparoscopy include endoscopic submucosal dissection (ESD), laparoscopic endoscopic cooperative surgery (LECS), non-exposed endoscopic wall-inversion surgery (NEWS), and transurethral resection of the bladder tumor (TUR-bt). In these minimally invasive treatments, when performing treatments on living tissue, for example, doctors or other operators use treatment instruments such as energy devices that emit high-frequency waves, ultrasound waves, microwaves, etc. to excise the area of ​​interest (pathogenic area) containing the lesion by cauterization, or to mark the area of ​​interest (pathogenic area) containing the lesion by thermal treatment, thereby marking the surgical target area as a preliminary treatment. Furthermore, during the actual treatment, the operator also uses energy devices such as those for excision and coagulation of the subject's living tissue.

[0058] therefore, Figure 1 The endoscope system 1 shown is used when performing surgery or treatment on a subject using a treatment instrument (not shown) such as an energy device capable of performing heat treatment. Figure 1 The endoscope system 1 shown is used for transurethral resection of bladder tumor (TUR-Bt), and is used when treating a tumor (bladder cancer) or a diseased area of ​​the bladder.

[0059] Figure 1The illustrated endoscope system 1 includes an insertion portion 2 , a light source device 3 , a light guide 4 , an endoscopic camera head 5 (endoscope imaging device), a first transmission cable 6 , a display device 7 , a second transmission cable 8 , a control device 9 , and a third transmission cable 10 .

[0060] The insertion portion 2 is rigid or at least partially flexible and has an elongated shape. The insertion portion 2 is inserted into a subject such as a patient via a cannula. The insertion portion 2 is internally provided with an optical system such as a lens for forming an observation image.

[0061] The light source device 3 is connected to one end of the light guide 4. Under the control of the control device 9, the light source device 3 supplies illumination light to the one end of the light guide 4 to irradiate the subject. The light source device 3 is implemented using the following components: any one or more light sources such as LED (Light Emitting Diode: Light Emitting Diode) light source, xenon lamp and LD (Laser Diode: Laser Diode) semiconductor laser element; a processor as a processing device having hardware such as FPGA (Field Programmable Gate Array: Field Programmable Gate Array), CPU (Central Processing Unit: Central Processing Unit); and a memory as a temporary storage area used by the processor. In addition, the light source device 3 and the control device 9 can be as follows: Figure 1 Although the structure is set as a structure in which communication is performed individually as shown, it can also be an integrated structure.

[0062] One end of the light guide 4 is detachably connected to the light source device 3 , and the other end is detachably connected to the insertion portion 2 . The light guide 4 guides illumination light supplied from the light source device 3 from one end to the other end, and supplies the illumination light to the insertion portion 2 .

[0063] The endoscope camera head 5 is detachably connected to the eyepiece portion 21 of the insertion portion 2. Under the control of the control device 9, the endoscope camera head 5 receives the observation image formed by the insertion portion 2 and performs photoelectric conversion to generate an imaging signal (RAW data). The imaging signal is output to the control device 9 via the first transmission cable 6.

[0064] One end of the first transmission cable 6 is detachably connected to the control device 9 via a video connector 61, and the other end is detachably connected to the endoscopic camera 5 via a camera connector 62. The first transmission cable 6 transmits the imaging signal output from the endoscopic camera 5 to the control device 9, and transmits setting data and power output from the control device 9 to the endoscopic camera 5. Here, the setting data refers to control signals, synchronization signals, clock signals, and the like for controlling the endoscopic camera 5.

[0065] The display device 7 displays an observation image based on an imaging signal processed by the control device 9 and various information related to the endoscope system 1 under the control of the control device 9. The display device 7 is implemented using a display monitor such as liquid crystal or organic EL (Electro Luminescence).

[0066] One end of the second transmission cable 8 is detachably connected to the display device 7 , and the other end is detachably connected to the control device 9 . The second transmission cable 8 transmits the image signal processed by the control device 9 to the display device 7 .

[0067] The control device 9 is implemented using the following components: a processor (including hardware such as a GPU (Graphics Processing Unit), FPGA, or CPU) as a processing device; and memory as a temporary storage area used by the processor. The control device 9 comprehensively controls the operation of the light source device 3, the endoscopic camera head 5, and the display device 7 via the first transmission cable 6, the second transmission cable 8, and the third transmission cable 10, in accordance with a program stored in the memory. Furthermore, the control device 9 performs various image processing on the imaging signal input via the first transmission cable 6 and outputs it to the second transmission cable 8.

[0068] One end of the third transmission cable 10 is detachably connected to the light source device 3 , and the other end is detachably connected to the control device 9 . The third transmission cable 10 transmits control data from the control device 9 to the light source device 3 .

[0069] [Functional structure of the main parts of the endoscope system]

[0070] Next, the functional configuration of the main parts of the endoscope system 1 will be described. Figure 2 1 is a block diagram showing the functional configuration of the main parts of the endoscope system 1 .

[0071] [Structure of the insertion part]

[0072] First, the structure of the insertion portion 2 will be described. The insertion portion 2 includes an optical system 22 and an illumination optical system 23 .

[0073] The optical system 22 forms an image of the subject by converging reflected light from the subject, return light from the subject, excitation light from the subject, and fluorescent light emitted from thermally denatured areas that have undergone thermal treatment by an energy device, etc. The optical system 22 is implemented using one or more lenses, etc.

[0074] The illumination optical system 23 irradiates the subject with illumination light supplied from the light guide 4. The illumination optical system 23 is implemented using one or more lenses and the like.

[0075] [Structure of light source device]

[0076] Next, a description will be given of the configuration of the light source device 3 . The light source device 3 includes a condenser lens 30 , a first light source unit 31 , a second light source unit 32 , a third light source unit 33 , and a light source control unit 34 .

[0077] The condenser lens 30 condenses the light emitted by each of the first light source unit 31 , the second light source unit 32 , and the third light source unit 33 and emits the light toward the light guide 4 .

[0078] Under the control of the light source control unit 34, the first light source unit 31 emits white light (normal light) as visible light, thereby supplying white light to the light guide 4 as illumination light. The first light source unit 31 is constructed using a collimating lens, a white LED lamp, and a driver. Alternatively, the first light source unit 31 can simultaneously emit red, green, and blue LED lamps to supply visible white light. Of course, the first light source unit 31 can also be constructed using a halogen lamp or a xenon lamp.

[0079] Under the control of the light source control unit 34, the second light source unit 32 emits first narrowband light having a predetermined wavelength range, thereby supplying the first narrowband light as illumination light to the light guide 4. The wavelength of the first narrowband light is 530 nm to 550 nm (with a center wavelength of 540 nm). The second light source unit 32 is constructed using a green LED lamp, a collimating lens, a transmission filter that transmits light in the 530 nm to 550 nm range, and a driver.

[0080] Under the control of the light source control unit 34, the third light source unit 33 emits a second narrowband light having a wavelength range different from that of the first narrowband light, thereby supplying the second narrowband light as illumination light to the light guide 4. The wavelength of the second narrowband light is 400 nm to 430 nm (with a center wavelength of 415 nm). The third light source unit 33 is implemented using a collimating lens, a semiconductor laser such as a violet laser diode (LD), and a driver. Furthermore, in the first embodiment, the second narrowband light functions as excitation light for exciting advanced glycation end products produced by heat treatment of biological tissue.

[0081] The light source control unit 34 is implemented using a processor (e.g., a hardware processor such as an FPGA or CPU) as a processing device and a memory (e.g., a temporary storage area) used by the processor. Based on control data input from the control device 9, the light source control unit 34 controls the light emission timing and duration of each of the first light source unit 31, the second light source unit 32, and the third light source unit 33.

[0082] Here, the wavelength characteristics of the light emitted by each of the second light source unit 32 and the third light source unit 33 will be described. Figure 3 Schematically shows the wavelength characteristics of the light emitted by the second light source unit 32 and the third light source unit 33. Figure 3 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents wavelength characteristics. Figure 3 Middle, broken line L NG The broken line L represents the wavelength characteristic of the first narrowband light emitted by the second light source unit 32. V represents the wavelength characteristics of the second narrowband light (excitation light) emitted by the third light source unit 33. Figure 3 In the middle, curve L B Indicates the wavelength range of blue, curve L G Indicates the wavelength range of green, curve L R Indicates the wavelength range of red.

[0083] like Figure 3 The broken line L NG As shown, the second light source unit 32 emits narrowband light with a central wavelength (peak wavelength) of 540 nm and a wavelength range of 530 nm to 550 nm. In addition, the third light source unit 33 emits excitation light with a central wavelength (peak wavelength) of 415 nm and a wavelength range of 400 nm to 430 nm.

[0084] In this manner, the second light source unit 32 and the third light source unit 33 respectively emit the first narrowband light and the second narrowband light (excitation light) having wavelength ranges different from each other.

[0085] Furthermore, the first narrowband light serves as light for layer discrimination in living tissue. Specifically, the difference in absorbance of the first narrowband light by the mucous membrane layer, the subject, and the muscularis layer, the subject, is large enough to distinguish the two subjects. Therefore, in the second layer discrimination image obtained by irradiating the first narrowband light for layer discrimination, the region capturing the mucous membrane layer has smaller and darker pixel values ​​(brightness values) than the region capturing the muscularis layer. In other words, in Embodiment 1, by using the second layer discrimination image for display image generation, the mucous membrane layer and muscularis layer can be displayed in a manner that makes them easily distinguishable.

[0086] Furthermore, the second narrowband light (excitation light) is different from the first narrowband light and is used for layer discrimination in living tissue. Specifically, the difference in absorbance of the second narrowband light by the muscle layer, the subject, and the fat layer, the subject, is large enough to distinguish the two subjects. Therefore, in the second light layer discrimination image obtained by irradiating the second narrowband light for layer discrimination, the areas where the muscle layer is captured have smaller and darker pixel values ​​(brightness values) than the areas where the fat layer is captured. In other words, by using the second layer discrimination image for display image generation, the muscle layer and fat layer can be easily distinguished.

[0087] The mucosal layer (organism mucosa) and the muscular layer are both subjects that contain a large amount of myoglobin. However, the concentration of myoglobin contained in the mucosal layer is relatively high and relatively low in the muscular layer. The reason for the difference in the light absorption characteristics of the mucosal layer and the muscular layer is the difference in the myoglobin concentration contained in the mucosal layer (organism mucosa) and the muscular layer. Moreover, the difference in absorbance between the mucosal layer and the muscular layer is greatest near the wavelength at which the absorbance of the organism mucosa reaches its maximum. In other words, the first narrowband light used for layer discrimination becomes light that more significantly shows the difference between the mucosal layer and the muscular layer than light with a peak wavelength in other wavelength ranges.

[0088] Furthermore, the absorbance of the second narrowband light for fat layer discrimination is lower than that of the second narrowband light for muscle layer discrimination. Therefore, in the second image captured by irradiation with the second narrowband light for layer discrimination, the pixel values ​​(brightness values) of the region capturing the muscle layer are lower than the pixel values ​​(brightness values) of the region capturing the fat layer. In particular, the second narrowband light for layer discrimination corresponds to a wavelength that maximizes the absorbance of the muscle layer, resulting in a significant difference between the muscle layer and the fat layer. Specifically, the difference in pixel values ​​(brightness values) between the muscle layer region and the fat layer region in the second image for layer discrimination is large enough to allow for differentiation.

[0089] In this manner, the light source device 3 irradiates the living tissue with each of the first and second narrowband lights. Consequently, the endoscopic camera head 5, described later, can capture the return light from the living tissue and thereby obtain an image capable of distinguishing the various layers comprising the living tissue: the mucous membrane, the muscular layer, and the fat layer. Hereinafter, the light resulting from the combination of the first and second narrowband lights will be referred to as special light.

[0090] Furthermore, in Embodiment 1, the second narrowband light (excitation light) excites advanced glycation end products (AGEs) produced by heat treatment of biological tissue using an energy device, etc. Furthermore, when amino acids and reducing sugars are heated, a glycation reaction (Maillard reaction) occurs. The end products produced as a result of this Maillard reaction are collectively referred to as advanced glycation end products (AGEs). AGEs are known to contain substances with fluorescent properties. Specifically, AGEs are generated when biological tissue is heat treated using an energy device, causing amino acids and reducing sugars in the tissue to heat and undergo a Maillard reaction. The AGEs generated by this heating can be visualized through fluorescence observation to reveal the state of heat treatment. Furthermore, AGEs are known to emit stronger fluorescence than autofluorescent substances already present in biological tissue. Specifically, in Embodiment 1, the fluorescent properties of AGEs generated in biological tissue by heat treatment using an energy device, etc., are utilized to visualize areas of thermal denaturation caused by heat treatment. Therefore, in the first embodiment, the second light source unit 32 (excitation light) irradiates the living tissue with blue excitation light having a wavelength of approximately 415 nm, which excites AGEs. This allows the first embodiment to observe a fluorescence image (thermal denaturation image) based on an imaging signal capturing fluorescence (e.g., green light having a wavelength of 490 nm to 625 nm) emitted from thermally denatured regions of AGEs. Therefore, below, when the second narrowband light is used alone, it will be referred to as excitation light.

[0091] [Structure of endoscope camera]

[0092] return Figure 2 , the structure of the endoscope system 1 is further described.

[0093] Next, the configuration of the endoscopic camera head 5 will be described. The endoscopic camera head 5 includes an optical system 51, a drive unit 52, an imaging element 53, a cut filter 54, an A / D converter 55, a P / S converter 56, an imaging and recording unit 57, and an imaging control unit 58.

[0094] The optical system 51 forms an image of the subject, focused by the optical system 22 of the insertion portion 2, onto the light-receiving surface of the imaging element 53. The optical system 51 is capable of changing the focal length and focus position. The optical system 51 is constructed using a plurality of lenses 511. The optical system 51 changes the focal length and focus position by moving each of the plurality of lenses 511 along the optical axis L1 via the drive unit 52.

[0095] The drive unit 52 moves the multiple lenses 511 of the optical system 51 along the optical axis L1 under the control of the imaging control unit 58. The drive unit 52 is configured using a motor such as a stepping motor, a DC motor, or a voice coil motor, and a transmission mechanism such as gears that transmits the motor's rotation to the optical system 51.

[0096] The imaging element 53 is implemented using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor having a plurality of pixels arranged in a two-dimensional matrix. Under the control of the imaging control unit 58, the imaging element 53 receives the subject image (light) formed by the optical system 51 and passed through the cut filter 54, performs photoelectric conversion on the subject image, generates an imaging signal (RAW data), and outputs it to the A / D converter 55. The imaging element 53 includes a pixel unit 531 and a color filter 532.

[0097] Figure 4 Schematic diagram showing the structure of the pixel portion 531. Figure 4 As shown, the pixel portion 531 is composed of a plurality of pixels P such as photodiodes that store charges according to the amount of light. nm (n=integer greater than or equal to 1, m=integer greater than or equal to 1) are arranged in a two-dimensional matrix. The pixel unit 531 is controlled by the imaging control unit 58 to obtain a plurality of pixels P nm The pixel P in the reading area arbitrarily set as the reading object nm The image signal is read as image data and output to the A / D conversion unit 55 .

[0098] Figure 5 Schematically shows the structure of the color filter 532. Figure 5 As shown, the color filter 532 is formed of a Bayer array with 2×2 units. The color filter 532 is composed of a filter R that transmits light in the red wavelength range, two filters G that transmit light in the green wavelength range, and a filter B that transmits light in the blue wavelength range.

[0099] Figure 6 is a diagram schematically showing the sensitivity and wavelength range of each filter. Figure 6 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents transmission characteristics (sensitivity characteristics). Figure 6 In the middle, curve L B Indicates the transmission characteristics of filter B, curve L G Curve L represents the transmission characteristics of filter G. R Indicates the transmission characteristics of filter R.

[0100] like Figure 6 The curve L B As shown in FIG, filter B transmits light in the blue wavelength range. Figure 6 The curve L G As shown in FIG, the filter G transmits light in the green wavelength range. Figure 6 The curve L R As shown, the filter R transmits light in the red wavelength range. In the following, the filter R is arranged on the light receiving surface to form a pixel P. nm The pixel P is formed by placing the filter G on the light receiving surface. nm The pixel P is formed by placing the filter B on the light receiving surface. nm This will be described as a B pixel.

[0101] According to the imaging element 53 configured in this manner, when receiving the subject image formed by the optical system 51, Figures 7A to 7C As shown, color signals (R signal, G signal, and B signal) are generated for each of the R pixel, the G pixel, and the B pixel.

[0102] return Figure 2 , the structure of the endoscope system 1 is further described.

[0103] The cutoff filter 54 is disposed on the optical axis L1 between the optical system 51 and the imaging element 53. The cutoff filter 54 is provided on the light-receiving surface side (incident surface side) of at least the G pixel of the color filter 532, which is provided with the filter G that transmits the green wavelength range. The cutoff filter 54 blocks light in the short wavelength range including the wavelength range of the excitation light, and transmits light in the long wavelength range that is longer than the wavelength range of the excitation light.

[0104] Figure 8 Schematically shows the structure of the cut filter 54. Figure 8 As shown, the filter F constituting the cutoff filter 54 11 Configured on filter G 11 (Refer to Figure 5 ) is configured at a position where the filter G 11 The light-receiving side directly above the .

[0105] Figure 9 and Figure 10 is a diagram schematically showing the transmission characteristics of the cut filter 54. Figure 9 and Figure 10 In the figure, the horizontal axis represents wavelength (nm) and the vertical axis represents transmittance. Figure 9 Middle, broken line L FThe transmission characteristics of the cut filter 54 are shown in FIG. NG The broken line L represents the wavelength characteristic of the first narrowband light. V Indicates the wavelength characteristics of the second narrowband light (excitation light). Figure 10 The transmission characteristics when receiving fluorescence emitted from the thermally denatured region when the thermally denatured region is irradiated with excitation light are shown.

[0106] like Figure 9 As shown, the cutoff filter 54 shields the wavelength range of the second narrowband light (excitation light) and transmits the wavelength range on the long wavelength side that is longer than the wavelength range of the second narrowband light (excitation light). Specifically, the cutoff filter 54 shields the light in the wavelength range on the short wavelength side that is shorter than the wavelength range of 400nm to 430nm that includes the second narrowband light (excitation light) and transmits the light in the wavelength range on the long wavelength side that is longer than the wavelength range of 400nm to 430nm that includes the second narrowband light (excitation light). And, as Figure 10 As shown, the cut filter 54 transmits fluorescence in a wavelength range emitted from the thermally denatured region when the thermally denatured region is irradiated with excitation light.

[0107] return Figure 2 , continue to explain the structure of the endoscope camera 5.

[0108] Under the control of the imaging control unit 58, the A / D converter 55 performs A / D conversion processing on the analog imaging signal input from the imaging element 53 and outputs the resulting signal to the P / S converter 56. The A / D converter 55 is implemented using an A / D conversion circuit or the like.

[0109] Under the control of the camera control unit 58, the P / S converter 56 performs parallel / serial conversion on the digital camera signal input from the A / D converter 55, and outputs the parallel / serial converted camera signal to the control device 9 via the first transmission cable 6. The P / S converter 56 is implemented using a P / S conversion circuit, etc. In the first embodiment, an E / O converter that converts the camera signal into an optical signal may be provided in place of the P / S converter 56, and the camera signal may be output to the control device 9 via an optical signal. Alternatively, the camera signal may be transmitted to the control device 9 via wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0110] The image recording unit 57 records various information related to the endoscopic camera head 5 (e.g., pixel information of the image sensor 53 and characteristics of the cutoff filter 54). Furthermore, the image recording unit 57 records various setting data and control parameters transmitted from the control device 9 via the first transmission cable 6. The image recording unit 57 is configured using a nonvolatile memory or a volatile memory.

[0111] The imaging control unit 58 controls the operation of the drive unit 52, the imaging element 53, the A / D converter 55, and the P / S converter 56 based on the setting data received from the control device 9 via the first transmission cable 6. The imaging control unit 58 is implemented using the following components: a TG (Timing Generator); a processor including hardware such as an ASIC (Application Specific Integrated Circuit) or a CPU; and a memory serving as a temporary storage area used by the processor.

[0112] [Structure of control device]

[0113] Next, the configuration of the control device 9 will be described.

[0114] The control device 9 includes an S / P conversion unit 91 , an image processing unit 92 , an input unit 93 , a recording unit 94 , and a control unit 95 .

[0115] Under the control of the control unit 95, the S / P converter 91 performs serial / parallel conversion on the image data received from the endoscopic camera 5 via the first transmission cable 6 and outputs the data to the image processing unit 92. Furthermore, if the endoscopic camera 5 outputs the imaging signal via an optical signal, an O / E converter that converts the optical signal into an electrical signal may be provided in place of the S / P converter 91. Furthermore, if the endoscopic camera 5 transmits the imaging signal via wireless communication, a communication module capable of receiving wireless signals may be provided in place of the S / P converter 91.

[0116] Under the control of the control unit 95, the image processing unit 92 performs predetermined image processing on the parallel data imaging signal input from the S / P conversion unit 91 and outputs the image to the display device 7. The predetermined image processing herein includes demosaicing, white balance, gain adjustment, gamma correction, and format conversion. The image processing unit 92 is implemented using a processor as a processing device, including hardware such as a GPU or FPGA, and a memory as a temporary storage area used by the processor.

[0117] In addition, the image processing unit 92 generates a pseudo-color image (narrow-band image) by performing image processing on the signal values ​​of the G pixels and the B pixels contained in the imaging signal input from the endoscopic camera 5 via the S / P conversion unit 91 when the light source device 3 irradiates special light. In this case, the signal value of the G pixel contains the deep layer information of the mucosa of the subject. In addition, the signal value of the B pixel contains the surface layer information of the mucosa of the subject. Therefore, the image processing unit 92 performs image processing such as gain control processing, pixel interpolation processing, and mucosa emphasis processing on the signal values ​​of the G pixels and the B pixels contained in the imaging signal to generate a pseudo-color image, and outputs the pseudo-color image to the display device 7. Here, the pseudo-color image refers to an image generated using only the signal values ​​of the G pixels and the signal values ​​of the B pixels. In addition, although the image processing unit 92 obtains the signal value of the R pixel, it does not use it for the generation of the pseudo-color image, but deletes it.

[0118] Furthermore, the image processing unit 92 generates a fluorescence image (pseudo-color image) by performing image processing on the signal values ​​of the G and B pixels contained in the imaging signal input from the endoscopic camera head 5 via the S / P conversion unit 91 when the light source device 3 emits excitation light. In this case, the signal values ​​of the G pixels include information about fluorescence emitted from the thermally treated area. Furthermore, the B pixels include background information about the surrounding biological tissue of the thermally treated area. Therefore, the image processing unit 92 performs image processing such as gain control, pixel interpolation, and mucosal enhancement on the signal values ​​of the G and B pixels contained in the image data to generate a fluorescence image (pseudo-color image) and outputs this fluorescence image (pseudo-color image) to the display device 7. In this case, the image processing unit 92 performs gain control processing to increase the gain of the signal values ​​for the G pixels compared to the gain of the signal values ​​for the G pixels during normal light observation, while decreasing the gain of the signal values ​​for the B pixels compared to the gain of the signal values ​​for the B pixels during normal light observation. Furthermore, the image processing unit 92 performs gain control processing so that the signal values ​​for the G pixels and the B pixels are equal (1:1).

[0119] The input unit 93 receives input of various operations related to the endoscope system 1 and outputs the received operations to the control unit 95. The input unit 93 is configured using a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, and the like.

[0120] The recording unit 94 is implemented using a recording medium such as a volatile memory, a nonvolatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory card. The recording unit 94 records data including various parameters required for the operation of the endoscope system 1. The recording unit 94 also includes a program recording unit 941 for recording various programs used to operate the endoscope system 1.

[0121] The control unit 95 is implemented using a processor with hardware such as an FPGA or CPU, and a memory serving as a temporary storage area for the processor. The control unit 95 comprehensively controls the various components that make up the endoscope system 1. Specifically, the control unit 95 reads the program recorded in the program recording unit 941 into the working area of ​​the memory and executes it. The processor executes the program to control the various components, thereby enabling the hardware and software to collaborate and implement functional modules that meet the specified objectives. Specifically, the control unit 95 includes an acquisition unit 951, a generation unit 952, a setting unit 953, a determination unit 954, a positioning unit 955, a determination unit 956, an output control unit 957, and a learning unit 958.

[0122] The acquisition unit 951 acquires, via the S / P conversion unit 91 and the image processing unit 92, the white light imaging signal generated by the endoscopic camera 5 when the light source device 3 irradiates white light toward the living tissue. Furthermore, the acquisition unit 951 acquires, via the S / P conversion unit 91 and the image processing unit 92, setting information defining a region of interest for the living tissue and thermal denaturation information regarding thermally denatured regions that have undergone thermal denaturation due to thermal treatment of the living tissue. Specifically, the acquisition unit 951 acquires a first image as the setting information and a second image as the thermal denaturation information. Here, the first image is a special light image, and the second image is a fluorescence image.

[0123] The generation unit 952 generates a first image based on the first imaging signal acquired by the acquisition unit 951. Here, the first image is a special light image. Furthermore, the first image represents the setting information in Embodiment 1. Furthermore, the generation unit 952 generates a second image based on the second imaging signal acquired by the acquisition unit 951. Here, the second image is a fluorescence image. Furthermore, the generation unit 952 generates a white light image based on the white light image signal acquired by the acquisition unit 951.

[0124] The setting unit 953 sets the target region based on the feature amount included in the first image generated by the generating unit 952. Specifically, the setting unit 953 determines whether the luminance value of each pixel constituting the first image generated by the generating unit 952 is greater than or equal to a predetermined value, and sets the region formed by a plurality of pixels having luminance values ​​greater than or equal to the predetermined value as the target region.

[0125] The identification unit 954 identifies the thermal denaturation region based on the second image generated by the generation unit 952. Specifically, the identification unit 954 determines whether the brightness value of each pixel constituting the second image is greater than or equal to a predetermined value, and identifies a region formed by a plurality of pixels having a brightness value greater than or equal to the predetermined value as the thermal denaturation region.

[0126] The alignment unit 955 performs alignment processing on the first and second images. For example, the alignment unit 955 performs alignment processing on the first and second images based on the position at which the feature values ​​of each pixel constituting the first image and the feature values ​​of each pixel constituting the second image match. Examples of feature values ​​include pixel values, brightness values, edges, and contrast.

[0127] The determination unit 956 determines whether a thermally denatured region exists outside the region of interest based on the setting information and the thermal denaturation information. Specifically, the determination unit 956 determines whether a thermally denatured region exists outside the region of interest based on the setting information of the first image and the thermal denaturation information of the second image after the alignment process by the alignment unit 955.

[0128] When the determination unit 956 determines that a thermal denaturation region exists outside the region of interest, the output control unit 957 outputs auxiliary information indicating the presence of a thermal denaturation region outside the region of interest to the display device 7. Specifically, the output control unit 957 displays the information by outputting one or more of a message, a graphic, and a sound indicating the presence of a thermal denaturation region outside the region of interest to the display device 7 as auxiliary information. Alternatively, the output control unit 957 may generate a display image by superimposing the thermal denaturation region outside the region of interest and the thermal denaturation region within the region of interest, as determined by the determination unit 954, on the white light image generated by the generation unit 952 so that they can be distinguished, and output this display image to the display device 7 as auxiliary information. Alternatively, the output control unit 957 may generate a display image that distinguishes between the thermal denaturation region outside the region of interest and the thermal denaturation region within the region of interest, as determined by the determination unit 954, and output this display image to the display device 7 as auxiliary information.

[0129] The learning unit 958 generates a learned model by performing machine learning using training data. The training data includes as input a fluorescence image generated by capturing an imaging signal generated by capturing light emitted from a thermally denatured region due to irradiation of biological tissue with excitation light, and a white light image generated by capturing an imaging signal generated by capturing return light due to irradiation of biological tissue with white light. The output data includes as auxiliary information indicating that a thermally denatured region included in the fluorescence image exists outside the region of interest included in the white light image. Specifically, the learning unit 958 may also generate a learned model by performing machine learning using training data. The training data includes as input a fluorescence image captured by irradiating biological tissue with excitation light, and a special light image (narrow-band light observation image) or a white light image captured by irradiating biological tissue with narrow-band light of a wavelength determined by the absorption rate of hemoglobin. The output data includes as auxiliary information indicating that a thermally denatured region included in the fluorescence image exists outside the region of interest included in the white light image. The learned model is composed of a neural network with each layer having one or more nodes.

[0130] In addition, the type of machine learning is not particularly limited. For example, corresponding training data and usage data are prepared by preparing fluorescence images and special light images (narrow-band light observation images) or white light images and annotation information, wherein the annotation information specifies the position or area of ​​the thermal denaturation area outside the area of ​​interest based on the fluorescence image and special light image (narrow-band light observation image) or white light image, and the training data and learning data can be input into a computational model based on a multi-layer neural network for learning.

[0131] Furthermore, as a machine learning technique, for example, a DNN (Deep Neural Network) technique based on a multi-layer neural network such as CNN (Convolutional Neural Network) or 3D-CNN can be used. Furthermore, as a machine learning technique, a technique based on a recurrent neural network (RNN) or an LSTM (Long Short-Term Memory units) that is an extension of RNN can also be used. In addition, a control unit of a learning device that is different from the control device 4 can also perform these functions to generate a learned model. Of course, the function of the learning unit 958 can also be set in the image processing unit 92.

[0132] 〔Control device processing〕

[0133] Next, the processing executed by the control device 9 will be described. Figure 11 It is a flowchart showing an outline of the processing executed by the control device 9 .

[0134] like Figure 11 As shown, first, the control unit 95 controls the light source control unit 34 of the light source device 3 to cause the second light source unit 32 and the third light source unit 33 to emit light to supply special light to the insertion portion 2 , thereby irradiating the living tissue with the special light (step S101 ).

[0135] Next, the control unit 95 controls the imaging control unit 58 to cause the imaging element 53 to capture the return light of the special light from the living tissue (step S102 ).

[0136] Thereafter, the acquisition unit 951 acquires a first imaging signal generated by the imaging element 53 of the endoscopic camera 5 through imaging (step S103 ).

[0137] Next, the control unit 95 controls the light source control unit 34 of the light source device 3 to cause the third light source unit 33 to emit light and irradiate the excitation light (step S104 ).

[0138] Thereafter, the control unit 95 controls the imaging control unit 58 to cause the imaging element 53 to capture fluorescence from the thermally denatured region of the living tissue (step S105 ).

[0139] Next, the acquisition unit 951 acquires a second imaging signal generated by the imaging element 53 of the endoscopic camera 5 (step S106 ).

[0140] Thereafter, the generating unit 952 generates a first image based on the first imaging signal acquired by the acquiring unit 951 (step S107 ). Here, the first image is a special light image. Furthermore, the first image is the setting information in the first embodiment.

[0141] Next, the generating unit 952 generates a second image based on the second imaging signal acquired by the acquiring unit 951 (step S108 ). Here, the second image is a fluorescence image.

[0142] Thereafter, the setting unit 953 sets a region of interest based on the feature amount included in the first image (step S109 ).

[0143] Figure 12 Schematically shows the region of interest set by the setting unit 953 for the first image. Figure 12 As shown, the setting unit 953 sets the target region based on the feature quantity included in the first image. Specifically, the setting unit 953 determines whether the brightness value of each pixel constituting the first image P1 is greater than or equal to a predetermined value, and sets the region formed by the plurality of pixels having brightness values ​​greater than or equal to the predetermined value as the target region D1.

[0144] After step S109 , the determination section 954 determines the thermally denatured region based on the second image (step S110 ).

[0145] Figure 13 Schematic diagram showing the thermal denaturation region determined by the determination unit 954 for the second image. Figure 13 As shown, the determination unit 954 determines the thermally denatured regions based on the second image. Specifically, the determination unit 954 determines whether the brightness value of each pixel constituting the second image P2 is greater than a predetermined value, and determines the regions formed by the plurality of pixels having brightness values ​​greater than the predetermined value as thermally denatured regions R1 and R2. Alternatively, the determination unit 954 may determine the thermally denatured regions (fluorescent regions) resulting from thermal treatment by comparing a reference image with a fluorescence image serving as the second image, where the reference image is based on an image signal generated by the imaging element 53 of the endoscopic camera 5 prior to thermal treatment using a resection treatment instrument or the like. In this case, the reference image may be pre-recorded in the recording unit 94.

[0146] After step S110 , the position alignment unit 955 performs a position alignment process of the first image and the second image (step S111 ).

[0147] Figure 14 Schematically shows the position alignment process of the position alignment unit 955. Figure 14 As shown, the position alignment unit 955 uses a known technique to perform position alignment processing so that the positions of the feature quantities included in the first image P1 and the second image P2 are aligned. For example, the position alignment unit 955 performs position alignment processing on the first and second images P1 and P2 based on the position where the feature quantities of each pixel constituting the first image P1 and the feature quantities of each pixel constituting the second image P2 are aligned. Here, the feature quantities include, for example, pixel values, brightness values, edges, and contrast.

[0148] After step S111, the determination unit 956 determines whether there is a thermal denaturation area outside the region of interest based on the setting information of the first image and the thermal denaturation information of the second image after the position alignment process by the position alignment unit 955 (step S112). Figure 13 In the case shown, the determination unit 956 determines that the thermal denaturation region R2 exists outside the attention region D1 based on the attention region D1 of the first image P1 and the thermal denaturation regions R1 and R2 of the second image P2 after the position alignment process by the position alignment unit 955. Figure 13In the case shown, the determination unit 956 determines that the thermally denatured region R2 exists outside the region of interest D1. If the determination unit 956 determines that the thermally denatured region exists outside the region of interest (step S112: "Yes"), the control device 9 proceeds to step S113, described later. On the other hand, if the determination unit 956 determines that the thermally denatured region does not exist outside the region of interest (step S112: "No"), the control device 9 proceeds to step S114, described later.

[0149] In step S113, the output control unit 957 outputs auxiliary information indicating that there is a thermal denaturation region outside the region of interest to the display device 7. Specifically, the output control unit 957 generates a display image in which information indicating that there is a thermal denaturation region R2 outside the region of interest D1 is superimposed on the first image P1, and outputs the display image as auxiliary information to the display device 7 for display. In this case, the output control unit 957 may output the following as auxiliary information to the display device 7 for display: a display image in which the thermal denaturation region R1 within the region of interest D1 and the thermal denaturation region R2 outside the region of interest D1 are superimposed on the first image P1 in a manner that allows them to be distinguished (for example, see FIG. 1 ). Figure 13 ); and at least one of a message, graphic, and symbol indicating the presence of a thermally denatured region outside the region of interest. Of course, the output control unit 957 may generate a display image in which one or more of a message or graphic indicating the presence of a thermally denatured region outside the region of interest is superimposed on the first image, and output this display image to the display device 7 for display. This allows the user to understand the presence of a thermally denatured region outside the region of interest. Alternatively, the output control unit 957 may output a message or graphic indicating the presence of a thermally denatured region outside the region of interest to the display device 7 for display.

[0150] Next, the control unit 95 determines whether a termination signal for terminating observation of the subject by the endoscope system 1 has been input from the input unit 93 (step S114). If the control unit 95 determines that a termination signal for terminating observation of the subject by the endoscope system 1 has been input from the input unit 93 (step S114: "Yes"), the control device 9 terminates this processing. On the other hand, if the control unit 95 determines that a termination signal for terminating observation of the subject by the endoscope system 1 has not been input from the input unit 93 (step S114: "No"), the control device 9 returns to step S101.

[0151] According to the first embodiment described above, when the output control unit 957 determines through the judgment unit 956 that there is a thermal denaturation area outside the area of ​​interest, the meaning indicating that there is a thermal denaturation area outside the area of ​​interest is output to the display device 7, so that it is possible to grasp whether there is a thermal denaturation area outside the area of ​​interest.

[0152] Furthermore, according to the first embodiment, the setting unit 953 sets the region of interest based on the feature amount included in the first image, and thus can assist the user in performing surgery.

[0153] Furthermore, according to the first embodiment, the identification unit 954 identifies the thermally denatured region based on the second image, and thus the thermally denatured region can be easily identified.

[0154] In addition, in embodiment 1, the learning unit 958 is provided in the control device 4, but it is not limited to this. The learning unit 958 that generates the learned model can also be provided in a device different from the control device 4, such as a learning device or a server that can be connected via a network.

[0155] Furthermore, in the first embodiment, the output control unit 957 may generate a display image by superimposing the thermally denatured regions outside the region of interest and the thermally denatured regions within the region of interest, which are determined by the determination unit 954, on the white light image generated by the generation unit 952 in a distinguishable manner, and output the display image as auxiliary information to the display device 7. Thus, the user can grasp the thermally denatured regions outside the region of interest on the white light image.

[0156] Furthermore, in the first embodiment, the output control unit 957 may generate a display image capable of distinguishing between the thermally denatured region occurring outside the region of interest and the thermally denatured region within the region of special interest, as determined by the determination unit 954, and output the display image as auxiliary information to the display device 7. This allows the user to easily distinguish between the thermally denatured region occurring outside the region of interest and the thermally denatured region within the region of special interest.

[0157] (Implementation Method 2)

[0158] Next, Embodiment 2 will be described. The endoscope system according to Embodiment 2 has the same configuration as the endoscope system 1 according to Embodiment 1 described above, but the processing performed by the control device 9 is different. Specifically, in Embodiment 1, the region of interest is set based on the feature value of the special light image, which is the first image. However, in Embodiment 2, the first image is a white light image, and the region of interest is set based on an instruction signal input from the input unit 93. Therefore, the processing performed by the control device 9 included in the endoscope system 1 according to Embodiment 2 will be described below.

[0159] 〔Control device processing〕

[0160] Figure 15 : is a flowchart showing an outline of the processing executed by the control device 9 according to the second embodiment. Figure 15 In the embodiment, the control device 9 executes step S101A, step S102A, step S107A and step S109A instead of the above step S101A, step S102A, step S107A and step S109A. Figure 11 Steps S101, S102, S107, and S109 described in the preceding paragraph are performed in addition to the above steps. Figure 11 The same treatment is applied to . Therefore, Figure 15 , step S101A, step S102A, step S107A, and step S109A are described.

[0161] like Figure 15 As shown, first, the control unit 95 controls the light source control unit 34 of the light source device 3 to cause the first light source unit 31 to emit light to supply white light to the insertion portion 2 , thereby irradiating the living tissue with white light (step S101A).

[0162] Next, the control unit 95 controls the imaging control unit 58 to cause the imaging element 53 to capture the return light of the white light from the living tissue (step S102A).

[0163] Next, the acquisition unit 951 acquires the first imaging signal generated by the imaging element 53 of the endoscopic camera 5 (step S103A). In this case, the acquisition unit 951 acquires, as setting information, instruction information for specifying annotations input by the user via the input unit 93 for the white light image displayed on the display device 7. After step S103A, the control device 9 proceeds to step S104.

[0164] In step S107A, the generator 952 generates a first image based on the first imaging signal acquired by the acquirer 951. Here, the first image is a white light image. After step S107A, the control device 9 proceeds to step S108.

[0165] In step S109A, the setting unit 953 sets a region of interest for the white-light image displayed on the display device 7 based on the instruction signal obtained by the acquisition unit 951, which indicates an annotation (annotation) indicated by the user via the input unit 93. For example, the setting unit 953 sets the instruction signal corresponding to the region indicated by the user through the input unit 93 as the region of interest to be used as the annotation. After step S109A, the control device 9 proceeds to step S110.

[0166] According to the second embodiment described above, the same effect as in the first embodiment can be achieved, that is, whether or not there is a thermally altered region outside the region of interest can be determined.

[0167] (Implementation 3)

[0168] Next, Embodiment 3 will be described. In Embodiment 1, the setting unit 953 sets the region of interest based on the feature values ​​contained in the first image. However, in Embodiment 3, a medical device separate from the control device detects the region of interest, and the region of interest is set based on the detection results. Therefore, the endoscope system according to Embodiment 3 will be described below. Components identical to those of the endoscope system 1 according to Embodiment 1 are denoted by the same reference numerals, and detailed descriptions will be omitted.

[0169] [Structure of the endoscope system]

[0170] Figure 16 This is a diagram showing a schematic configuration of an endoscope system according to a third embodiment. Figure 16 The illustrated endoscope system 1A includes a medical device 11 and a fourth transmission cable 12 in addition to the configuration of the endoscope system 1 according to the first embodiment described above.

[0171] The medical device 11 is implemented using the following components: a processor as a processing device having hardware such as a GPU, FPGA, or CPU; and a memory as a temporary storage area used by the processor. The medical device 11 obtains various information from the control device 9 via the fourth transmission cable 12 and outputs the obtained various information to the control device 9. In addition, the medical device 11 outputs the position information for setting the area of ​​interest to the control device 9 using a learning model. The learning model is a model that performs machine learning using training data that establishes a correspondence between multiple images and annotation information of the area of ​​interest included in each of the multiple images, uses a white light image as input data, and outputs the position of the area of ​​interest in the white light image as output data. Here, machine learning is deep learning, etc.

[0172] One end of the fourth transmission cable 12 is detachably connected to the control device 9, and the other end is detachably connected to the medical device 11. The fourth transmission cable 12 transmits various information from the control device 9 to the medical device 11, and transmits various information from the medical device 11 to the control device 9.

[0173] 〔Control device processing〕

[0174] Next, the processing executed by the control device 9 will be described.

[0175] Figure 17: is a flowchart showing an outline of the processing executed by the control device 9. Figure 17 In the embodiment, the control device 9 executes step S109B to replace the above Figure 15 Step S109A described in the above is performed except that Figure 15 The same treatment is applied to . Therefore, Figure 17 , step S109B is described.

[0176] In step S109B, the setting unit 953 sets the ROI in the white-light image based on the positional information for setting the ROI input from the medical device 11 and acquired by the acquisition unit 951. For example, the setting unit 953 uses a learning model included in the medical device 11 to set the ROI in the white-light image based on the positional information for setting a region, such as a tumor, included in the white-light image as the ROI. After step S109B, the control device 9 proceeds to step S110.

[0177] According to the third embodiment described above, the same effect as in the first embodiment can be achieved, that is, whether or not there is a thermally altered region outside the region of interest can be ascertained.

[0178] (Implementation 4)

[0179] Next, Embodiment 4 will be described. In Embodiment 1, the control device 9 determines whether a thermally denatured region exists outside the region of interest. However, in Embodiment 4, a separate medical device is provided that determines whether a thermally denatured region exists outside the region of interest and outputs the determination result. The configuration of the endoscope system according to Embodiment 4 will be described below. Components identical to those of the endoscope system 1 according to Embodiment 1 are denoted by the same reference numerals, and detailed descriptions will be omitted.

[0180] [Structure of the endoscope system]

[0181] Figure 18 This is a diagram showing a schematic configuration of an endoscope system according to a fourth embodiment. Figure 18 The illustrated endoscope system 1B includes a control device 9B instead of the control device 9 according to the aforementioned embodiment 1. In addition to the configuration of the endoscope system 1 according to the aforementioned embodiment 1, the endoscope system 1B further includes a medical device 13 and a fifth transmission cable 14 .

[0182] The control device 9B is implemented using the following components: a processor serving as a processing device, including hardware such as a GPU, FPGA, or CPU; and a memory serving as a temporary storage area used by the processor. The control device 9B comprehensively controls the operation of the light source device 3, the endoscopic camera head 5, the display device 7, and the medical device 13 via each of the first transmission cable 6, the second transmission cable 8, the third transmission cable 10, and the fifth transmission cable 14, in accordance with a program stored in the memory. The control device 9B omits the functions of the acquisition unit 951, the generation unit 952, the setting unit 953, the determination unit 954, the positioning unit 955, the determination unit 956, the output control unit 957, and the learning unit 958 from the control unit 95 of the first embodiment described above.

[0183] The medical device 13 is implemented using the following components: a processor (e.g., a GPU, FPGA, or CPU) serving as a processing device; and a memory serving as a temporary storage area used by the processor. The medical device 13 receives various information from the control device 9B via a fifth transmission cable 14 and outputs the received information to the control device 9B. The detailed functional structure of the medical device 13 will be described later.

[0184] One end of the fifth transmission cable 14 is detachably connected to the control device 9B, and the other end is detachably connected to the medical device 13. The fifth transmission cable 14 transmits various information from the control device 9B to the medical device 13, and transmits various information from the medical device 13 to the control device 9B.

[0185] [Functional structure of medical devices]

[0186] Next, the functional configuration of the medical device 13 will be described. Figure 19 1 is a block diagram showing the functional structure of the medical device 13. Figure 19 As shown, the medical device 13 includes a communication I / F 131 , an input unit 132 , a recording unit 133 , and a control unit 134 .

[0187] The communication I / F 131 is an interface for communicating with the control device 9B via the fifth transmission cable 14. The communication I / F 131 receives various information from the control device 9A according to a predetermined communication standard and outputs the received information to the control unit 134.

[0188] The input unit 132 receives input of various operations related to the endoscope system 1B and outputs the received operations to the control unit 134. The input unit 132 is configured using a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, and the like.

[0189] The recording unit 133 is implemented using a recording medium such as a volatile memory, a nonvolatile memory, an SSD, a HDD, or a memory card. The recording unit 133 records data including various parameters required for the operation of the medical device 13. The recording unit 133 also includes a program recording unit 133a that records various programs for operating the medical device 13.

[0190] The control unit 134 is implemented using a processor comprising hardware such as an FPGA or CPU, and a memory serving as a temporary storage area for the processor. The control unit 134 comprehensively controls the various components comprising the medical device 13. The control unit 134 has the same functions as the control unit 95 described in the first embodiment. Specifically, the control unit 134 includes an acquisition unit 951, a generation unit 952, a setting unit 953, a determination unit 954, a positioning unit 955, a determination unit 956, an output control unit 957, and a learning unit 958.

[0191] The medical device 13 thus configured performs the same processing as the control device 9 according to the first embodiment described above, and outputs the processing results to the control device 9B. In this case, the control device 9B outputs information corresponding to the presence or absence of a thermally denatured region outside the region of interest as the processing result of the medical device 13 to the display device 7.

[0192] According to the fourth embodiment described above, the same effect as in the first embodiment can be achieved, that is, whether or not there is a thermally altered region outside the region of interest can be determined.

[0193] (Implementation 5)

[0194] Next, Embodiment 5 will be described. In Embodiment 1 described above, the output control unit 957 outputs the determination result of the presence or absence of a thermally denatured region outside the region of interest, as determined by the determination unit 956, to the display device 7. In Embodiment 5, the light source device irradiates the thermally denatured region outside the region of interest toward the living tissue. Therefore, the functional structure of the main components of the endoscope system according to Embodiment 5 will be described. Components identical to those of the endoscope system 1 according to Embodiment 1 will be denoted by the same reference numerals, and detailed descriptions will be omitted.

[0195] (Functional Structure of Main Parts of Endoscope System)

[0196] Figure 20 This is a diagram illustrating the functional configuration of the main parts of an endoscope system 1C according to the fourth embodiment. Figure 20 The endoscope system 1C shown includes a light source device 3C instead of the light source device 3 of the endoscope system 1 according to the first embodiment described above.

[0197] The light source device 3C includes, in addition to the configuration of the light source device 3 according to the first embodiment, a projection unit 35 that projects information on a region of interest and a thermally denatured region outside the region of interest toward a living tissue.

[0198] Under the control of the light source control unit 34, the projection unit 35 projects auxiliary information input from the control device 9, including information regarding the region of interest and information regarding thermally denatured regions outside the region of interest, onto the living tissue via the condenser lens 30 and the light guide 4. Specifically, the projection unit 35 projects information indicating the region of interest and information indicating the region of thermally denatured regions outside the region of interest as auxiliary information, in a manner that allows for distinction between the region of living tissue corresponding to the region of interest and the region of living tissue corresponding to the thermally denatured regions outside the region of interest. For example, the projection unit 35 projects the region of living tissue corresponding to the region of interest in blue and the region of living tissue corresponding to the thermally denatured regions outside the region of interest in green. Furthermore, in Embodiment 5, the projection unit 35 functions as a projection device.

[0199] According to the fifth embodiment described above, the same effect as in the first embodiment can be achieved, that is, whether or not there is a thermally altered region outside the region of interest can be determined.

[0200] (Other embodiments)

[0201] By appropriately combining the multiple components disclosed in the endoscope systems according to the first to fifth embodiments of the present disclosure, various inventions can be formed. For example, some components may be deleted from all the components described in the endoscope systems according to the embodiments of the present disclosure. Furthermore, the components described in the endoscope systems according to the embodiments of the present disclosure may be appropriately combined.

[0202] Furthermore, in the endoscope systems according to Embodiments 1 to 5 of the present disclosure, the systems are connected to each other by wires, but they may be connected wirelessly via a network.

[0203] Furthermore, in Embodiments 1 to 5 of the present disclosure, the functions of the control unit, the acquisition unit 951, the generation unit 952, the setting unit 953, the determination unit 954, the positioning unit 955, the determination unit 956, and the functional modules of the output control unit 957 included in the endoscope system may be provided on a server or the like that can be connected via a network. Of course, a server may be provided for each functional module.

[0204] Furthermore, in Embodiments 1 to 5 of the present disclosure, an example of application to transurethral bladder tumor resection has been described, but the present invention is not limited thereto and can be applied to various surgeries for resecting lesions using energy devices or the like, for example.

[0205] In the endoscope systems according to Embodiments 1 to 5 of the present disclosure, the aforementioned “units” can be replaced by “units” or “circuits”, etc. For example, the control unit can be replaced by a control unit or a control circuit.

[0206] Furthermore, although expressions such as "first," "afterwards," and "next" are used in the descriptions of the flowcharts in this specification to clarify the sequential relationship between the processes in the steps, the order of the processes required to implement the present invention is not solely defined by these expressions. In other words, the order of the processes in the flowcharts described in this specification can be changed within a range consistent with the order in which they are performed.

[0207] While several embodiments of the present application have been described in detail above based on the drawings, these are merely examples and the present invention can be implemented in other ways that are variously modified and improved based on the knowledge of those skilled in the art, represented by the methods described in the columns of this disclosure.

[0208] Description of Reference Numerals

[0209] 1, 1A, 1B, 1C: Endoscope system; 2: Insertion unit; 3, 3C: Light source device; 4: Light guide; 5: Endoscope camera head; 6: First transmission cable; 7: Display device; 8: Second transmission cable; 9, 9A, 9B: Control device; 10: Third transmission cable; 11, 13: Medical device; 12: Fourth transmission cable; 13: Medical device; 14: Fifth transmission cable; 31: First light source unit; 32: Second light source unit; 33: Third light source unit; 34: Light source control unit; 35: Projection unit ; 51: optical system; 92: image processing unit; 93, 132: input unit; 94, 133: recording unit; 95, 134: control unit; 131: communication I / F; 133a, 941: program recording unit; 951: acquisition unit; 952: generation unit; 953: setting unit; 954: determination unit; 955: positioning unit; 956: judgment unit; 957: output control unit; 958: learning unit; D1: region of interest; R1, R2: thermal denaturation region; P1: first image; P2: second image.

Claims

1. A medical device comprising a processor, wherein: The processor performs the following processing: acquiring setting information for setting a region of interest for a living tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to thermal treatment of the living tissue; determining whether the thermal denaturation region exists outside the region of interest based on the setting information and the thermal denaturation information; as well as When it is determined that the thermally denatured region exists outside the region of interest, auxiliary information indicating that the thermally denatured region exists outside the region of interest is output.

2. The medical device according to claim 1, wherein The processor acquires a first image of the biological tissue. The first image includes the setting information.

3. The medical device according to claim 2, wherein: The first image is a special light image generated based on an imaging signal generated by capturing return light generated by irradiating the living tissue with special light. The setting information is a feature quantity included in the special light image. The processor sets the region of interest based on a feature amount included in the special light image.

4. The medical device according to claim 2, wherein: The first image is a white light image generated based on an imaging signal generated by capturing return light generated by irradiating the living tissue with white light. The processor performs the following processing: acquiring, as the setting information, position information output by a learning model, the model performing machine learning using training data in which a plurality of images are associated with annotation information of the region of interest included in each of the plurality of images, the model taking the white light image as input data and outputting the position of the region of interest in the white light image as output data; and The region of interest is set based on the setting information.

5. The medical device according to claim 2, wherein The first image is a white light image generated based on an imaging signal generated by capturing return light generated by irradiating the living tissue with white light. The processor performs the following processing: acquiring, as the setting information, instruction information for instructing an annotation input from the outside with respect to the white light image; and The region of interest is set based on the setting information.

6. The medical device according to claim 1, wherein The processor sets the thermal denaturation region according to instruction information for instructing annotation input from the outside.

7. The medical device according to claim 1, wherein The processor acquires the thermal denaturation information from a second image obtained by capturing the biological tissue.

8. The medical device according to claim 7, wherein: The second image is a fluorescent image generated based on an imaging signal generated by capturing light emitted from the thermally denatured region due to irradiation of the living tissue with excitation light. The thermally denatured region is a region that emits fluorescence.

9. The medical device according to claim 8, wherein The processor performs the following processing: determining, for each pixel constituting the fluorescent image, whether a signal value is equal to or greater than a predetermined threshold; and Pixels whose signal values ​​are equal to or greater than a predetermined threshold value are determined as the thermally denatured region.

10. The medical device according to claim 8, wherein The processor performs the following processing: acquiring a reference image based on an imaging signal obtained by imaging the living tissue before the thermal treatment; and The thermally denatured region is determined based on the reference image and the second image.

11. The medical device according to claim 1, wherein The processor performs the following processing: generating a display image indicating that the thermally denatured region exists outside the region of interest; and The display image is output as the auxiliary information.

12. The medical device according to claim 1, wherein The processor performs the following processing: generating a display image capable of distinguishing the thermally denatured region generated outside the region of interest from the thermally denatured region within the region of interest; as well as The display image is output as the auxiliary information.

13. The medical device according to claim 12, wherein: The processor performs the following processing: acquiring a white light image generated based on an imaging signal generated by capturing return light generated by irradiating the living tissue with white light; generating the display image by superimposing the thermally denatured region generated outside the region of interest and the thermally denatured region within the region of interest on the white light image in a distinguishable manner; as well as The display image is output as the auxiliary information.

14. The medical device according to claim 1, wherein The processor performs the following processing: generating a display image capable of distinguishing the region of interest from the thermally denatured region; and The display image is output as the auxiliary information.

15. The medical device according to claim 1, wherein The processor outputs position information of the thermally denatured region outside the region of interest as the auxiliary information to a projection device capable of projecting information onto the living tissue.

16. A medical system comprising a light source device, an imaging device, and a medical device, wherein: The light source device comprises: a special light source that generates special light directed at biological tissue; and an excitation light source that generates excitation light for exciting advanced glycation end products produced by heat treatment of the biological tissue, The imaging device includes an imaging element configured to generate an imaging signal by capturing return light or luminescence from the living tissue irradiated with the special light or the excitation light. The medical device includes a processor. The processor performs the following processing: acquiring setting information for setting a region of interest for a living tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to thermal treatment of the living tissue; determining whether the thermal denaturation region exists outside the region of interest based on the setting information and the thermal denaturation information; as well as When it is determined that the thermally denatured region exists outside the region of interest, auxiliary information indicating that the thermally denatured region exists outside the region of interest is output.

17. A learning device comprising a processor, wherein: A learned model is generated by performing machine learning using training data, wherein the training data includes as input a plurality of fluorescence images generated based on an imaging signal generated by capturing light emitted from a thermally denatured region by irradiating biological tissue with excitation light, and a plurality of white light images generated based on an imaging signal generated by capturing return light generated by irradiating the biological tissue with white light, and outputs auxiliary information indicating that the thermally denatured region included in the fluorescence image exists outside the region of interest included in each of the plurality of white light images.

18. A method for operating a medical device, the medical device comprising a processor, wherein: The processor performs the following processing: acquiring setting information for setting a region of interest for a living tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to thermal treatment of the living tissue; determining whether the thermal denaturation region exists outside the region of interest based on the setting information and the thermal denaturation information; as well as When it is determined that the thermally denatured region exists outside the region of interest, auxiliary information indicating that the thermally denatured region exists outside the region of interest is output.

19. A program executed by a medical device having a processor, wherein: The program causes the processor to execute the following processing: acquiring setting information for setting a region of interest for a living tissue and thermal denaturation information related to a thermally denatured region that has undergone thermal denaturation due to thermal treatment of the living tissue; determining whether the thermal denaturation region exists outside the region of interest based on the setting information and the thermal denaturation information; as well as When it is determined that the thermally denatured region exists outside the region of interest, auxiliary information indicating that the thermally denatured region exists outside the region of interest is output.

Images