system

Abstract

A system (10) comprises a storage unit (120) that stores a trained model (121), and a control unit (110). The control unit acquires a captured image in which at least one energy device (310) and at least one biological tissue are captured. The control unit infers a thermal diffusion region and a specific tissue region from the captured image, by processing on the basis of the trained model that is stored in the storage unit. The control unit determines, on the basis of the inferred thermal diffusion region and the inferred specific tissue region, the risk of thermal damage to a specific tissue due to an energy output from the energy device.

Description

[Technical field]

[0001] The present invention relates to a system Mu et al. Regarding. [Background technology]

[0002] Patent Document 1 discloses a surgical system that determines the type of tissue grasped by an energy device based on energy output data of the energy device, tissue position, patient condition, or optical tissue sensor information. For example, whether the tissue is vascular or non-vascular, or the presence or absence of nerves, etc., are recognized as the type of tissue. If the treatment content for the recognized type is inappropriate, this surgical system stops the energy output and warns the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2017 / 0252095 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the degree of thermal diffusion to the tissue to be treated or its surroundings or prevention of thermal diffusion is not taken into consideration, so there is a problem that the energy output cannot be appropriately adjusted according to the state of the tissue or device during treatment with the energy device. Specifically, when thermal diffusion occurs widely or quickly due to the treated tissue, the heat may propagate to important tissue before thermally denaturing the treated tissue, causing thermal damage to the important tissue. In addition, providing various sensors such as an optical tissue sensor is not preferable from the viewpoint of sterility. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to a system including: a memory unit that stores a trained model that has been trained to estimate a thermal diffusion region from an energy device to a biological tissue and a specific tissue region in the biological tissue, which are generated by the energy output of the energy device, from a learning device tissue image, which is an image of at least one energy device and at least one biological tissue during energy output that receives energy supply and outputs energy, or from a learning tissue image, which is an image of the at least one biological tissue, and a control unit; wherein the control unit acquires an image, which is an image of at least one energy device and at least one biological tissue during energy output, estimates the thermal diffusion region and the specific tissue region from the image by processing based on the trained model stored in the memory unit, and determines a risk of thermal damage to the specific tissue due to the energy output of the energy device from the estimated thermal diffusion region and specific tissue region.

[0006] Another aspect of the present disclosure relates to a program for causing a computer to execute the following steps: acquiring in real time an image of at least one energy device outputting energy and at least one biological tissue that receives energy supply and outputs energy; estimating image recognition information from a learning device tissue image that is an image of the at least one energy device outputting energy and the at least one biological tissue, or a learning tissue image that is an image of the at least one biological tissue, by processing based on a trained model that has been trained to output image recognition information that is at least one of information regarding a thermal diffusion region from the energy device to the biological tissue and information regarding a specific tissue region in the biological tissue generated by the energy output of the energy device; and determining a risk of thermal damage to the specific tissue due to the energy output of the energy device from the estimated image recognition information.

[0007] Another aspect of the present disclosure relates to a method for assessing a risk of thermal damage by acquiring in real time an image of at least one energy device outputting energy in response to a supply of energy and at least one biological tissue, estimating a learning device tissue image of the at least one energy device outputting energy and the at least one biological tissue from the learning tissue image of the at least one biological tissue by processing based on a trained model that has been trained to output image recognition information, which is at least one of information regarding a thermal diffusion region from the energy device to the biological tissue and information regarding a specific tissue region in the biological tissue generated by the energy output of the energy device, and determining a risk of thermal damage to the specific tissue due to the energy output of the energy device from the estimated image recognition information. [Brief description of the drawings]

[0008] [Figure 1] An example of system configuration. [Diagram 2] An example of controller configuration. [Diagram 3] 4 is a flowchart illustrating a process performed by the system. [Figure 4] An example of a monopolar device configuration. [Diagram 5] An example of a bipolar device configuration. [Figure 6] An example of an ultrasonic device configuration. [Figure 7] 3 shows a processing example of the first embodiment. [Figure 8] FIG. 13 is an explanatory diagram when the control unit detects a specific tissue. [Figure 9] FIG. 13 is an explanatory diagram of a case where a control unit detects a thermal diffusion region. [Figure 10] FIG. 13 is an explanatory diagram of a case where the control unit determines a risk of thermal damage. [Figure 11] 13 is an example of output adjustment when the control unit adjusts the energy output. [Figure 12] An example of the configuration of a learning device. [Figure 13] An illustration of the learning phase for estimating important tissues. [Figure 14] Example images used in the training phase for estimating the heat diffusion area. [Figure 15] FIG. [Figure 16] 5A to 5C are explanatory diagrams of input information used in the processing example of the embodiment. [Figure 17] FIG. 13 is a graph showing the dependence of the distance between a specific tissue region and a thermal diffusion region on the energy output time. [Figure 18] FIG. 13 is an explanatory diagram of input information according to the third embodiment. [Figure 19] FIG. 13 is a diagram illustrating the learning stage for estimating a thermal diffusion region. [Figure 20] An example of an endoscopic image using special light. [Figure 21] FIG. 13 is an explanatory diagram of a fourth embodiment. [Figure 22] FIG. 13 is a diagram illustrating the learning stage for estimating a thermal diffusion region. [Figure 23] 13 is a processing example of the fifth embodiment. [Figure 24] 13 is a processing example of the sixth embodiment. [Diagram 25] 13 is a processing example of the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the contents described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0010] 1. System Fig. 1 is a configuration example of a system 10 in this embodiment. Fig. 1 shows an example of a system configuration when an operative field is photographed by an endoscope. The system 10 shown in Fig. 1 includes a controller 100, an endoscopic system 200, a generator 300, and an energy device 310. The system 10 is a surgical system for performing surgery using at least one energy device under an endoscope. Here, an example is shown in which the system 10 includes one energy device 310, but the system 10 may include multiple energy devices.

[0011] The endoscope system 200 is a system that performs imaging using an endoscope, image processing of the endoscopic image, and monitor display of the endoscopic image. The endoscope system 200 includes an endoscope 210, a main body device 220, and a display 230. Here, a rigid endoscope for surgical use will be described as an example.

[0012] The endoscope 210 includes an insertion section inserted into a body cavity, an operation section connected to a base end of the insertion section, a universal cord connected to the base end of the operation section, and a connector section connected to the base end of the universal cord. The insertion section includes a rigid tube, an objective optical system, an image sensor, an illumination optical system, a transmission cable, and a light guide. The objective optical system and the image sensor for photographing the inside of the body cavity, and the illumination optical system for illuminating the inside of the body cavity are provided at the tip of a long and thin cylindrical rigid tube. The tip of the rigid tube may be configured to be bendable. A transmission cable for transmitting an image signal acquired by the image sensor, and a light guide for guiding illumination light to the illumination optical system are provided inside the rigid tube. The operation section is held by a user and accepts operations from the user. The operation section is provided with buttons to which various functions are assigned. When the tip of the insertion section is bendable, the operation section is provided with an angle operation lever. The connector portion includes a video connector that detachably connects the transmission cable to the main device 220 , and a light guide connector that detachably connects the light guide to the main device 220 .

[0013] The main unit 220 includes a processing device that controls the endoscope, processes the endoscopic image, and displays the endoscopic image, and a light source device that generates illumination light and controls the illumination light. The main unit 220 is also called a video system center. The processing device is composed of a processor such as a CPU, processes the image signal transmitted from the endoscope 210 to generate an endoscopic image, and outputs the endoscopic image to the display 230 and the controller 100. The illumination light emitted by the light source device is guided to the illumination optical system by a light guide, and is emitted from the illumination optical system into the body cavity.

[0014] The energy device 310 is a device that outputs energy from its tip in the form of high-frequency power or ultrasound, etc., to perform treatment such as coagulation, sealing, hemostasis, incision, resection, or peeling on tissue that its tip contacts. The energy device 310 is also called an energy treatment tool. The energy device 310 is a monopolar device that passes high-frequency power between an electrode at the tip of the device and an electrode outside the body, a bipolar device that passes high-frequency power between two jaws, an ultrasound device that is provided with a probe and jaws and emits ultrasound from the probe, or a combination device that passes high-frequency power between the probe and jaws and emits ultrasound from the probe.

[0015] The generator 300 supplies energy to the energy device 310, controls the energy supply, and acquires electrical information from the energy device 310. When the energy device 310 outputs high-frequency energy, the generator 300 supplies high-frequency power, and the energy device 310 outputs the high-frequency power from an electrode or a jaw. When the energy device 310 outputs ultrasonic energy, the generator 300 supplies power, and the probe of the energy device 310 converts the power into ultrasonic waves and outputs them.

[0016] The electrical information is electrical information of the tissue in contact with the electrodes, probes, or jaws of the energy device 310, specifically, information obtained as a response to the energy device 310 outputting high-frequency power to the tissue. The electrical information is, for example, impedance information of the tissue to be treated by the energy device 310. However, as described later, the electrical information is not limited to impedance information.

[0017] The generator 300 performs control to change the energy output from the energy device 310 over time according to the output sequence. The generator 300 may change the energy output in response to a change in impedance information over time. In this case, the output sequence may specify how to change the energy output in response to a change in impedance information. The generator 300 may also automatically turn off the energy output in response to a change in impedance information over time. For example, when the impedance rises above a certain level, the generator 300 may determine that the treatment is complete and turn off the energy output.

[0018] The controller 100 recognizes the biological tissue and the energy device 310 from the endoscopic image by image recognition processing using machine learning or the like, and outputs an energy output adjustment instruction based on the recognized information to the generator 300. Information related to the biological tissue recognized from the endoscopic image or information related to the energy device 310, or both, are called image recognition information. This information is information related to factors that affect heat diffusion by the energy device 310.

[0019] The information on the biological tissue includes not only specific organs but also parts associated with the organs, such as tissues connecting the organs. In the following description, the biological tissue will be referred to as specific tissue or important tissue as appropriate. Also, a region including a specific tissue will be referred to as a specific tissue region. The energy device 310 is an energy device such as a monopolar device 320, a bipolar device 330, an ultrasonic device 340, etc., as described later with reference to Figs. 4 to 6, but may be other than these.

[0020] The generator 300 adjusts the energy output of the energy device 310 according to the energy output adjustment instruction. That is, the system 10 of this embodiment is a system that automatically adjusts the energy output from the energy device 310 based on an endoscopic image. The generator 300 supplies energy to the energy device 310 at an energy supply amount instructed by the energy output adjustment instruction, and the energy device 310 receives the energy supply and outputs energy, thereby adjusting the energy output according to the energy output adjustment instruction.

[0021] The energy output adjustment instruction is an instruction to increase or decrease the output of the entire waveform of the output sequence, or an instruction to set one of a plurality of selectable output sequences. For example, when the energy output from the energy device 310 is adjustable by stepwise magnification, the energy output adjustment instruction is an instruction indicating the stepwise magnification of the energy output. The generator 300 increases or decreases the high frequency output or ultrasonic output according to the indicated magnification. The magnification may be continuously adjustable. Alternatively, when a plurality of output sequences are provided, the energy output adjustment instruction is an instruction indicating one of the plurality of output sequences. The generator 300 performs energy output from the energy device 310 according to the indicated output sequence. The energy output adjustment instruction may include both an instruction to increase or decrease the energy output and a instruction to change the output sequence.

[0022] 2. Controller Fig. 2 shows a configuration example of the controller 100. The controller 100 includes a control unit 110, a storage unit 120, an I / O device 180, and an I / O device 190. Figs. 1 and 2 show an example in which the controller 100 is configured as a device separate from the generator 300. In this case, the controller 100 is configured as an information processing device such as a PC or a server device. Alternatively, the controller 100 may be realized as a cloud or the like in which processing is executed by one or more information processing devices connected via a network.

[0023] The I / O device 180 receives image data of an endoscopic image from the main unit 220 of the endoscope system 200. The I / O device 180 is a connector to which an image transmission cable is connected, or an interface circuit that is connected to the connector and communicates with the main unit 220.

[0024] The control unit 110 estimates a specific tissue and a thermal diffusion region from an endoscopic image by image recognition processing using the trained model 121, and outputs an energy output adjustment instruction based on the estimation result. The control unit 110 includes one or more processors as hardware. The processor is a general-purpose processor such as a central processing unit (CPU), a graphical processing unit (GPU), or a digital signal processor (DSP). Alternatively, the processor may be a dedicated processor such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0025] The storage unit 120 stores a trained model 121 used in the image recognition process. When the image recognition process is performed by a general-purpose processor, the storage unit 120 stores a program in which an inference algorithm is described and parameters used in the inference algorithm as the trained model 121. The trained model 121 includes a first trained model 122 and a second trained model 123. The first trained model 122 is a trained model 121 related to the estimation of a specific tissue region described in FIG. 12 below, and the second trained model 123 is a trained model 121 related to the estimation of a thermal diffusion region. When the image recognition process is performed by a dedicated processor in which an inference algorithm is implemented as hardware, the storage unit 120 stores parameters used in the inference algorithm as the trained model 121.

[0026] The storage unit 120 is a storage device such as a semiconductor memory, a hard disk drive, an optical disk drive, etc. The semiconductor memory is a RAM, a ROM, a non-volatile memory, etc.

[0027] As an inference algorithm for image recognition processing, for example, a neural network can be adopted. The weight coefficients and biases of the node connections in the neural network are parameters. The neural network includes an input layer to which image data is input, an intermediate layer that performs arithmetic processing on the data input through the input layer, and an output layer that outputs a recognition result based on the arithmetic result output from the intermediate layer. As a neural network used for image recognition processing, for example, a CNN (Convolutional Neural Network) can be adopted.

[0028] The control unit 110 also includes a thermal diffusion detection unit 111, an important tissue detection unit 112, a thermal damage risk judgment unit 114, and an output setting unit 113. The storage unit 120 stores a program in which the functions of the thermal diffusion detection unit 111, the important tissue detection unit 112, the thermal damage risk judgment unit 114, and the output setting unit 113 are described. One or more processors of the control unit 110 read the program from the storage unit 120 and execute the program to realize the functions of the thermal diffusion detection unit 111, the important tissue detection unit 112, the thermal damage risk judgment unit 114, and the output setting unit 113. The program in which the functions of the units are described can be stored in a non-transitory information storage medium that is a computer-readable medium. The information storage medium can be realized by, for example, an optical disk, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM or a non-volatile memory.

[0029] The I / O device 190 transmits a signal instructing adjustment of energy output to the generator 300. The I / O device 190 is a connector to which a cable for signal transmission is connected, or an interface circuit connected to the connector and communicating with the generator 300.

[0030] FIG. 3 is a flowchart for explaining the processing performed by the controller 100 and the system 10. In step S1, the control unit 110 acquires an endoscopic image from the main unit 220 of the endoscope system 200 via the I / O device 180. Next, the control unit 110 executes steps S2A and S2B. In step S2A, the important tissue detection unit 112 performs image recognition processing on the endoscopic image using the first trained model 122 to estimate and detect important tissue from the image. In step S2B, the thermal diffusion detection unit 111 performs image recognition processing on the endoscopic image using the second trained model 123 to estimate and detect a thermal diffusion region around the jaw of the energy device 310. The thermal diffusion region is a region in which heat is diffused by the energy output of the energy device 310. For example, when the temperature of a certain region of biological tissue is equal to or higher than a predetermined temperature, the part can be regarded as a thermal diffusion region. Next, in step S3, the thermal damage risk determination unit 114 determines the risk that heat will diffuse to and cause thermal damage to important tissues based on the estimation results in steps S2A and S2B, and sends a signal to the output setting unit 113. Thermal damage includes, for example, protein denaturation and inactivation of intracellular enzymes. Hereinafter, this risk will be referred to as thermal damage risk. Then, in step 4, the output setting unit 113 adjusts the energy output of the energy device 310 based on the signal.

[0031] 3. Energy Devices As examples of the energy device 310, a monopolar device 320, a bipolar device 330, an ultrasonic device 340, and a combination device will be described below.

[0032] 4 shows a configuration example of monopolar device 320. Monopolar device 320 includes an elongated cylindrical insertion section 322, an electrode 321 provided at the tip of insertion section 322, an operation section 323 connected to the base end of insertion section 322, and a cable 325 connecting operation section 323 to a connector (not shown). The connector is detachably connected to generator 300.

[0033] The high-frequency power output by the generator 300 is transmitted by a cable 325 and output from the electrode 321. A counter electrode plate is provided outside the patient's body, and electricity flows between the electrode 321 and the counter electrode plate. As a result, high-frequency energy is applied to the tissue in contact with the electrode 321, and Joule heat is generated in the tissue. The electrode 321 may have various shapes depending on the treatment. The monopolar device 320 can adjust the degree of coagulation and incision by changing the current pattern. In general, the treatment target of the monopolar device 320 is the tissue in contact with the electrode 321, but the heat diffused around the tissue in contact with the electrode 321 may affect the surrounding tissue.

[0034] FIG. 5 shows a configuration example of the bipolar device 330. The bipolar device 330 includes an elongated cylindrical insertion section 332, two jaws 337 and 338 provided at a tip 331 of the insertion section 332, an operation section 333 connected to a base end of the insertion section 332, and a cable 335 connecting the operation section 333 to a connector (not shown). The connector may be detachably connected to the generator 300. The jaws 337 and 338 are gripping sections for gripping tissue and applying energy to the gripped tissue, and are configured to be openable and closable around an axis provided at the base end 336. The operation section 333 is provided with gripping sections for operating the opening and closing of the jaws 337 and 338. When a doctor grips the gripping sections, the jaws 337 and 338 close and grip the tissue.

[0035] The high-frequency power output by the generator 300 is transmitted by the cable 335, and when the jaws 337, 338 grasp the tissue, electricity is applied between the two jaws 337, 338. As a result, high-frequency energy is applied to the tissue sandwiched between the two jaws 337, 338, Joule heat is generated in the tissue, and the tissue is coagulated. The generator 300 may measure impedance information of the tissue grasped by the jaws 337, 338, detect the completion of the treatment based on the impedance information, and automatically stop the energy output. The generator 300 may also automatically adjust the energy applied to the tissue based on the impedance information. The device temperature of the bipolar device 330 only rises to about 100 degrees Celsius, for example, but a sneak current is generated around the part grasped by the jaws 337, 338, and the sneak current may cause thermal diffusion.

[0036] A derivative of the bipolar device is the vessel sealing device, which is a device with a cutter attached to the jaws of the bipolar device, and cuts the tissue by running the cutter through it after coagulating it with electricity.

[0037] FIG. 6 is a configuration example of the ultrasonic device 340. The ultrasonic device 340 includes an elongated cylindrical insertion section 342, a jaw 347 and a probe 348 provided at a tip end 341 of the insertion section 342, an operation section 343 connected to a base end of the insertion section 342, and a cable 345 connecting the operation section 343 and a connector (not shown). The connector is detachably connected to the generator 300. The jaw 347 is movable around an axis provided at the base end 346, and is configured to be openable and closable relative to the immovable probe 348. The operation section 343 is provided with a gripping section for operating the opening and closing of the jaw 347. When the doctor grips the gripping section, the jaw 347 closes, and the jaw 347 and the probe 348 grip the tissue. The operation section 343 is provided with an operation button 344a to which a first output mode is assigned and an operation button 344b to which a second output mode is assigned, as shown in FIG. 6, for example. The output mode is selected according to the treatment contents, and when the operation button of each output mode is pressed, ultrasonic energy is output in the output sequence of that mode. The number of operation buttons provided on the operation unit 343 is not limited to the configuration in FIG.

[0038] The power output by the generator 300 is transmitted by a cable 335, and when the operation button 344a or 344b is pressed, the probe 348 converts the power into ultrasonic waves and outputs them. As a result, frictional heat is generated in the tissue sandwiched between the jaw 347 and the probe 348, and the tissue is coagulated or incised.

[0039] The combination device using high-frequency power and ultrasound is configured similarly to the ultrasound device in FIG. 6. However, the combination device can generate Joule heat in the tissue held by the jaws and the probe by applying high-frequency power between the jaws and the probe, and can coagulate the tissue. The combination device can also incise the tissue held by the jaws and the probe by outputting ultrasound from the probe, similar to the ultrasound device. A high-frequency mode is assigned to one of the two operation buttons provided on the operation unit, and a seal & cut mode is assigned to the other. The high-frequency mode is a mode in which treatment such as coagulation is performed only by outputting high-frequency energy. The seal & cut mode is a mode in which high-frequency energy and ultrasound energy are used in combination, and is a mode in which tissue is coagulated and cut by outputting high-frequency energy. Regarding the thermal diffusion of the combination device, for example, there is a possibility that thermal diffusion similar to that of a bipolar device, thermal diffusion similar to that of an ultrasound device, or both may occur.

[0040] In the following embodiment, a case where a bipolar device 330 is mainly used as the energy device 310 will be described as an example. However, this embodiment can be applied to the case where the above-mentioned various energy devices in which thermal diffusion may occur are used.

[0041] 4. First embodiment 7 shows a processing example of the first embodiment. First, as shown in S21, an endoscopic image is input to the control unit 110. Specifically, each frame image of a moving image captured by an endoscope is input sequentially to the control unit 110. The endoscopic image input to the control unit 110 shows one or more energy devices outputting energy and one or more biological tissues.

[0042] Next, as shown in S22A, the control unit 110 detects important tissues from the endoscopic image by executing an estimation program adjusted by machine learning. Specifically, the control unit 110 detects important tissues by inputting an endoscopic image during surgery to a network having an estimation program that has been annotated and learned with important tissues in the endoscopic image. Here, the important tissues are, for example, a large blood vessel, a pancreas, a duodenum, etc., but are not limited to these. FIG. 8 is a diagram explaining the detection of important tissues shown in S22A of FIG. 7. As shown in FIG. 8, the control unit 110 estimates an area of ​​important tissue from a subject captured in the endoscopic image based on the estimation program, and labels the area of ​​the important tissue by adding a color. Then, the control unit 110 outputs the labeled endoscopic image to the thermal damage risk determination unit 114.

[0043] As shown in S22B of FIG. 7, the control unit 110 detects the thermal diffusion region from the endoscopic image by executing the estimation program described above. As in S22A, the control unit 110 detects the thermal diffusion region by inputting the endoscopic image during surgery to a network having an estimation program that has been annotated and learned with a temperature range above a threshold value on the endoscopic image. FIG. 9 is a diagram for explaining the detection of the thermal diffusion region in S22B of FIG. 7. As shown in FIG. 9, the control unit 110 labels the range of thermal diffusion by coloring the region of the organ captured in the endoscopic image that is estimated to be at a predetermined temperature or higher based on the estimation program. Then, the control unit 110 outputs the labeled endoscopic image to the thermal damage risk determination unit 114. Note that the labeling of the above-mentioned important tissues and thermal diffusion regions may be performed by a method other than color.

[0044] Next, as shown in S23 of FIG. 7, the thermal damage risk determination unit 114 of the control unit 110 receives information on the estimation results of the important tissue and the thermal diffusion area, and determines the thermal damage risk of the important tissue based on the estimation results. FIG. 10 is a diagram explaining a method for determining the thermal damage risk in S23 of FIG. 7. The thermal damage risk determination unit 114 of the control unit 110 generates an image in which the labeling of the important tissue and the thermal diffusion area is superimposed as shown in FIG. 10 based on the endoscopic images output from the important tissue detection unit 112 and the thermal diffusion detection unit 111, respectively. Then, the distance between the area estimated to be the important tissue and the area estimated to be at a predetermined temperature or higher is calculated. In FIG. 10, for example, when the stomach is estimated as the important tissue, the distance between the stomach and the area estimated to be at a predetermined temperature or higher is shown. Here, the distance is the actual distance between the thermal diffusion area shown in gray around the jaw and the closest part of the stomach, which is the important tissue, when the energy device 310 is a bipolar device. The distance can be obtained, for example, by analyzing the image displayed on the display 230. The control unit 110 uses the distance as a margin M to determine the risk of thermal damage. The margin M is data that the control unit 110 can use as a judgment material when judging the risk of thermal damage, and for example, the above-mentioned distance becomes the margin M. If the margin M is equal to or less than a preset threshold, it is judged that there is a risk of thermal damage to important tissue. If the margin M is greater than the threshold, it is judged that there is no risk of thermal damage to important tissue. The thermal damage risk judgment unit 114 outputs a signal regarding the judgment result regarding the presence or absence of the above-mentioned risk of thermal damage to the output setting unit 113.

[0045] Then, as shown in S24 of FIG. 7, the output setting unit 113 issues an output change instruction based on the signal output by the thermal damage risk determination unit 114. Specifically, the storage unit 120 stores table data on energy output adjustment instructions corresponding to the presence or absence of a thermal damage risk, and the control unit 110 outputs an energy output adjustment instruction corresponding to the presence or absence of a thermal damage risk to the generator 300 via the I / O device 190 by referring to the table data. Then, the generator 300 adjusts the output sequence of the bipolar device according to the energy output adjustment instruction output by the control unit 110. Note that the algorithm for outputting the energy output adjustment instruction according to the presence or absence of a thermal damage risk is not limited to the above. FIG. 11 is an example of output adjustment when the output setting unit 113 of the control unit 110 issues an energy output adjustment instruction based on the judgment result of the thermal damage risk. When the output setting unit 113 judges that there is a risk of thermal damage to important tissue, it adjusts the output so as to weaken the energy output. Specifically, the output setting unit 113 suppresses or stops the voltage, power, or the like. If it is determined that there is no risk of thermal damage to important tissue, the output is adjusted so as to maintain or increase the energy output. The output setting unit 113 outputs a signal regarding these output adjustment instructions to the generator 300.

[0046] In this way, the control unit 110 recognizes important tissue and the thermal diffusion area due to the energy output from the endoscopic image, and if it is determined that there is a risk of thermal damage to the important tissue, the energy output of the generator 300 is automatically adjusted.

[0047] In the judgment of the risk of thermal damage shown in S23 of FIG. 7, the threshold value of the margin M for judging the presence or absence of the risk of thermal damage may be different depending on the tissue type of the specific tissue detected in S22A. Alternatively, the threshold value may be variable. In this way, it is possible to avoid the occurrence of thermal damage in organs in which the occurrence of thermal damage is particularly problematic. For example, in the case of biological tissues such as the pancreas and duodenum, the margin M is made large, and in the case of biological tissues such as the stomach and liver, the margin M is made small, thereby avoiding the occurrence of thermal damage in important tissues in which the degree of severity is particularly high when thermal damage occurs.

[0048] The energy output adjustment instruction is an instruction to increase, decrease, or maintain the energy output based on the reference energy output. The generator 300 has an operation unit that accepts an operation to set the energy output, and the energy output can be set to any one of five levels of intensity 1 to 5 by the operation unit. Intensity 1 indicates the lowest energy output, and intensity 5 indicates the highest energy output. At this time, the reference energy output is, for example, a predetermined energy output such as "intensity 3". In that case, an instruction to increase the energy output from the reference energy output is an instruction to set it to "intensity 4" or "intensity 5", and an instruction to decrease the energy output from the reference energy output is an instruction to set it to "intensity 2" or "intensity 1". Alternatively, the reference energy output may be the current energy output set by the operation unit of the generator 300. In that case, an instruction to increase the energy output from the reference energy output is an instruction to set the energy output to a higher energy output than the currently set energy output, and an instruction to decrease the energy output from the reference energy output is an instruction to set the energy output to a lower energy output than the currently set energy output. Alternatively, the reference energy output may be within the output range of intensity 1 to 5 that can be set in the generator 300. In this case, an instruction to increase the energy output above the reference energy output is an instruction to set the energy output higher than "intensity 5", and an instruction to decrease the energy output below the reference energy output is an instruction to set the energy output lower than "intensity 1".

[0049] Now, one of the points to note when performing energy treatment in a surgical operation is to suppress thermal diffusion from the energy device 310 to avoid thermal damage to surrounding organs. Here, since the tissue to be treated is not uniform, the time required for treatment such as resection varies depending on the type of tissue, the state of the tissue, or the individual differences of the patient, and the degree of thermal diffusion also varies. To deal with these and suppress thermal diffusion, doctors adjust the output energy of the energy device, but this is an operation that requires experience, and it may be difficult for non-experts to make appropriate adjustments.

[0050] Thus, in general, in treatments using energy devices, thermal diffusion to the surroundings is often a problem, so doctors perform treatments while estimating the degree of diffusion. In the surgical system disclosed in the above-mentioned Patent Document 1, the presence or absence of thermal denaturation of the treatment target tissue is determined based on impedance values, etc., and if thermal denaturation has already occurred, the energy output mode is switched to a mode that outputs low energy. However, in this surgical system, only the presence or absence of thermal denaturation of the treatment target tissue by the energy device is determined, and thermal diffusion to the periphery of the treatment tissue is not taken into consideration. For this reason, there is a problem that the energy output cannot be appropriately adjusted according to the state of the treatment tissue and the energy device. For example, when using an energy treatment tool for surgery, especially inexperienced doctors may be less aware of thermal diffusion to the surrounding tissue. In such cases, in tissues where thermal diffusion occurs widely or quickly, heat may propagate to important tissues before the treatment target tissue is thermally denatured, causing thermal damage to the important tissues.

[0051] In this regard, according to the present embodiment, the system 10 acquires an image in which the energy device 310 and the biological tissue are captured, estimates the thermal diffusion region and the specific tissue region by processing based on the trained model, and determines the risk of thermal damage to the specific tissue by the energy device. Based on this risk of thermal damage, the system 10 can adjust the energy applied to the tissue to be treated to an appropriate amount, thereby reducing the risk of thermal damage to important tissue without noticing thermal diffusion. In this way, the system 10 adjusts the output energy of the energy device, which was previously performed by a doctor, thereby reducing the burden on the doctor. In addition, the system 10 autonomously adjusts the output, making it possible for a non-expert to perform stable treatment. As a result, the stability of surgery is improved, or the procedure is standardized regardless of the doctor's experience.

[0052] 12 shows an example of the configuration of a learning device 500 that performs machine learning of an estimation process of an important tissue and a thermal diffusion region. The learning device 500 is realized by an information processing device such as a PC or a server device, and includes a processing unit 510 and a storage unit 520.

[0053] The processing unit 510 is a processor such as a CPU, and the storage unit 520 is a storage device such as a semiconductor memory or a hard disk drive. The storage unit 520 stores a learning model 522 and teacher data 521. Here, the teacher data 521 includes a first teacher data 521A and a second teacher data 521B. The learning model 522 includes a first learning model 522A and a second learning model 522B. The processing unit 510 trains the learning model 522 using the teacher data 521 to generate a trained model 121. The teacher data 521 includes image data of a plurality of learning images and answer data attached to each learning image. Here, the first teacher data 521A is data about an important tissue, and the second teacher data 521B is data about a thermal diffusion region. Also, the first learning model 522A is a learning model about an important tissue, and the second learning model 522B is a learning model about a thermal diffusion region. The multiple learning images include an endoscopic image in which one or more biological tissues and one or more energy devices 310 are shown. This endoscopic image is also called a learning device tissue image. The multiple learning images may also include an endoscopic image in which one or more biological tissues are shown and the energy device 310 is not shown. This endoscopic image is also called a learning tissue image. The correct answer data is an annotation in segmentation (area detection), an annotation in detection (position detection), a correct answer label in classification, or a correct answer label in regression (regression analysis). The processing unit 510 inputs the learning image to an inference process by the learning model 522, and provides feedback to the learning model 522 based on the error between the result of the inference process and the correct answer data, and repeats this process with a large number of teacher data to generate a trained model 121. The generated trained model 121 is transferred to the storage unit 120 of the controller 100.

[0054] FIG. 13 is a diagram illustrating a learning stage for estimating important tissues. As shown in FIG. 13, in the learning device 500 (not shown), a learning device tissue image or a learning tissue image is fed back to the first learning model 522A as the first teacher data 521A. The learning device tissue image is an image of at least one energy outputting energy device 310 that receives energy supply and outputs energy, and at least one living tissue. The learning tissue image is an image of at least one living tissue. The learning device tissue image or the learning tissue image has an area of ​​important tissue labeled on the image. The labeling is performed by, for example, a doctor. In the learning device 500, such a large number of learning device tissue images or learning tissue images are fed back to the first learning model 522A as the first teacher data 521A, thereby improving the accuracy when the control unit 110 estimates important tissues.

[0055] The annotation of the estimation program for the thermal diffusion region described above is performed by creating the second teacher data 521B based on an image captured by, for example, a thermo camera. FIG. 14 shows an example of an endoscopic image in which the thermal diffusion range is colored in various biological tissues using temperature information captured by a thermo camera. In the example shown in FIG. 14, the thermal diffusion state is displayed in three temperature ranges: 59 degrees Celsius or less, 60 degrees Celsius to 79 degrees Celsius, and a threshold temperature or more. The threshold temperature is a temperature above which there is a risk of thermal damage, and in the case of FIG. 14, it is 80 degrees Celsius. In this case, the image shown in FIG. 14 becomes a learning image for the second teacher data 521B, and the information on the thermal diffusion region shown in color on the image becomes the correct answer data. Here, the evaluation of the risk of thermal damage due to thermal diffusion is based on, for example, protein denaturation and intracellular enzyme inactivation. As alternative indicators, risk evaluation based on a simple temperature range, risk evaluation based on the history of the product of temperature and time, risk evaluation based on the amount of heat, or risk evaluation based on activation energy according to the Arrhenius equation is known. When using a simple temperature range, the standard for risk assessment is, for example, 60 degrees Celsius or 50 degrees Celsius. When using the history of the product of temperature and time, the standard for risk assessment is, for example, 0.1 seconds at 60 degrees Celsius or 0.1 seconds at 50 degrees Celsius. Risk assessment based on the amount of heat is used in cases such as liver cauterization, while risk assessment based on activation energy using the Arrhenius equation is mainly used for burns.

[0056] 5. Second embodiment In the second embodiment, history information is used in estimating the thermal diffusion region. Hereinafter, differences from the first embodiment will be mainly described. FIG. 15 is a diagram illustrating a learning stage for estimating the thermal diffusion region. As shown in FIG. 15, in a learning device 500 (not shown), a history image captured using, for example, the above-mentioned thermo camera is fed back to a second learning model 522B as second teacher data 521B. The history image is, for example, an endoscopic image in which the thermal diffusion region is indicated by color or the like based on the temperature information from the above-mentioned thermo camera. In this way, the most recent history image can be used as the second teacher data 521B for learning the network used in the thermal diffusion detection function.

[0057] In addition, in the learning stage for estimating the thermal diffusion region, for example, historical information of the output energy of the energy device 310 can be used as the second teacher data 521B together with the above-mentioned historical image. Specifically, as shown in FIG. 15, information on the time dependency of the output of the output energy of the energy device 310 can be used. The above-mentioned historical image and historical information of the output energy include a large number of data such as scene 1, scene 2, ..., scene n, and are used as the second teacher data 521B. This second teacher data 521B is fed back to the second learning model 522B, and the second trained model 123 is generated. In this way, in addition to the most recent endoscopic image, the historical information of the output energy can also be fed back to the second learning model 522B, and the control unit 110 can estimate the thermal diffusion region with higher accuracy.

[0058] Fig. 16 shows an example of input information used in S21 of the processing example shown in Fig. 7. As shown in Fig. 16, the control unit 110 can use multiple endoscopic images captured at different timings as input information in S21. The endoscopic images captured at different timings are, for example, frame images of a video. The control unit 110 can also use information on the energy output history output from the energy device 310 as input information in addition to the multiple endoscopic images. Then, the detection of the thermal diffusion region in step S22B of Fig. 7 is performed based on the multiple endoscopic images and the energy output history information.

[0059] In this way, when estimating the thermal diffusion region, not only images captured in real time but also the most recent historical images and energy output history are used as input information, making it possible to estimate the thermal diffusion region including dynamic information such as tissue contraction, etc. Therefore, the control unit 110 can estimate the thermal diffusion region with high accuracy.

[0060] FIG. 17 is a diagram showing the dependency of the distance between the important tissue region and the thermal diffusion region, that is, the margin M, on the energy output time T. In FIG. 17, t on the horizontal axis is the current time, and t0 is the time when thermal diffusion is estimated to reach the important tissue. Therefore, for example, time t0-n, a specified time n before time t0, is the timing at which it is determined that there is a risk of thermal damage to the important tissue. As described in FIG. 16, the behavior of the margin M from time 0 to time t in the graph of FIG. 17 can be estimated by the control unit 110 from multiple endoscopic images with different imaging timings, which are the most recent history images. As shown in FIG. 17, the margin M decreases with the passage of time from time 0 to time t. Then, the estimated time t0 at which thermal diffusion reaches the important tissue can be predicted from the behavior of the margin M from time 0 to time t with respect to the energy output time T. For example, the control unit 110 can predict time t0 by extrapolating the dependency of margin M on energy output time T from the behavior of margin M from time 0 to time t and obtaining the intersection with the horizontal axis indicating energy output time T. The time t0-n, which is a specified time n before t0 obtained in this way, is the timing at which it is determined that there is a risk of thermal damage to important tissue.

[0061] In this way, the decrease in the margin M, which is the distance between the important tissue and the thermal diffusion region, is acquired in a time series, and it is possible to determine that there is a risk of thermal damage a specified time n before the time when the thermal diffusion is predicted to arrive. That is, the control unit 110 can output the time when the thermal diffusion will arrive at the specific tissue region as a prediction result, determine the risk of thermal damage, and output the determination result. In this way, the risk of thermal damage occurring in the important tissue can be reduced, and surgery can be performed safely.

[0062] 6. Third embodiment The input information of S21 in the first embodiment shown in FIG. 7 may include history information of the multi-wavelength image. For example, as shown in FIG. 18, the input information may include a real-time image and history image of the special light in addition to the energy output history and an image of the normal light of the endoscope. The image captured by the special light is, for example, a multi-wavelength image by illumination light using narrowband light narrower than the wavelength band of visible light or multiple narrowband lights. Examples of such a multi-wavelength image include AFI (Auto Fluorescence Imaging) and NBI (Narrow Band Imaging). Note that normal light is also called white light. In this way, by using a multi-wavelength image when detecting the thermal diffusion region, information focusing on a specific tissue, such as the connective tissue around the blood vessel, can also be input. In this way, dynamic information such as the aggregation of the connective tissue can be easily obtained, and by utilizing this, the accuracy of estimation of important tissues and estimation of the thermal diffusion region can be improved. The learning stage for estimating the thermal diffusion region has been described in FIG. 14 and FIG. 15, but FIG. 19 is an explanatory diagram of a case where the history information of the multi-wavelength image is included in the second teacher data 521B. 15, the second teacher data 521B includes a real-time multi-wavelength image and its historical image. In the learning device 500 (not shown), the second teacher data 521B including the multi-wavelength image is fed back to the second learning model 522B, and the second learned model 123 is generated.

[0063] Fig. 20 is an example of an image obtained by observing an excised blood vessel using AFI. As shown in Fig. 20, by using AFI, the control unit 110 can clearly distinguish between blood vessels and connective tissues and observe the excised blood vessels. In this way, by using a multi-wavelength image of AFI, for example, tissues that are prone to thermal contraction can be separately observed, and the accuracy of image recognition can be improved.

[0064] 7. Fourth embodiment In the first embodiment shown in FIG. 7, an image including whitening caused by thermal denaturation can be used as the captured image acquired by the control unit 110. Whitening refers to a state in which thermal damage occurs in biological tissue due to heat, and the tissue becomes whiter than other areas that are not thermally damaged, and looks different from other areas in appearance. In this way, the area at risk of thermal damage can be estimated from the whitening range. FIG. 21 is a diagram explaining a case in which an image including whitening caused by thermal denaturation is used as the captured image acquired by the control unit 110. In S31, the control unit 110 inputs an endoscopic image during surgery to the network of this function and estimates the whitening range by image processing. Specifically, a network that has been annotated with the whitening range on the endoscopic image and trained is used. In addition, there are cases in which thermal damage occurs even if no abnormality is observed in appearance. Therefore, in S32, the control unit 110 sets the range obtained by adding a specified range to the extracted whitening range as the thermal diffusion area, and uses it as input information for determining the risk of thermal damage.

[0065] In this way, information on the white-burned area can be obtained from the endoscopic image, eliminating the need to use equipment such as a thermo camera when creating the teacher data 521. In addition, the number of networks used by the control unit 110 can be reduced, and the processing speed of the control unit 110 can be increased.

[0066] 22 is a diagram illustrating a learning stage when the input information includes an image labeled with a white-washed region. The learning device 500 (not shown) feeds back the learning device tissue image or the learning tissue image, which shows a biological tissue labeled with a white-washed region, as the second teacher data 521B to the second learning model 522B to create the second trained model 123.

[0067] 8. Fifth embodiment The fifth embodiment shown in Fig. 23 is different from the first embodiment shown in Fig. 7 in the estimation process of S22A for detecting important tissues and S22B for detecting thermal diffusion regions. Specifically, in the estimation process of the fifth embodiment, TransmissionS42C is added to recognize the heat capacity and heat resistance of the heat path. For example, even if the distance between the important tissue and the thermal diffusion area is estimated, if the tissue heat transfer characteristics of the biological tissue between the important tissue and the thermal diffusion area are different, the time it takes for heat to diffuse to the important tissue will change. For this reason, in the first embodiment shown in FIG. Transmission The effects of thermal properties are not taken into account, and the accuracy of the thermal diffusion prediction results remains at a certain level.

[0068] In this regard, in the fifth embodiment, the trained model 121 is trained to estimate tissue heat transfer characteristics from the training energy output information of the energy device 310, the training device tissue image, or the training tissue image. Then, the control unit 110 estimates tissue heat transfer characteristics from the energy output information or captured image of the energy device 310 by processing based on the trained model 121 stored in the storage unit 120. Therefore, the control unit 110 can predict the distance between the thermal diffusion region and the important tissue, that is, the reduction in the margin M, from the characteristics of the heat transfer path. Therefore, the risk of thermal damage to the important tissue is reduced, and surgery can be performed safely. Note that the change in the reduction rate of the margin M can be predicted by utilizing, for example, machine learning, etc.

[0069] 9. Sixth embodiment The sixth embodiment shown in FIG. 24 is different from the first embodiment in FIG. 7 in the process after S23 in which the risk of thermal damage is determined. In the sixth embodiment, after the control unit 110 determines the risk of thermal damage, it determines in S54 whether or not an alert needs to be presented. Specifically, the control unit 110 determines that an alert needs to be presented if it is determined in the determination in S53 that there is a risk of thermal damage, and determines that an alert does not need to be presented if it is determined that there is no risk of thermal damage. Then, if it is determined that there is a risk of thermal damage, the control unit 110 presents an alert in S55, and if it is determined that there is no risk of thermal damage, it does not present an alert in S56. The alert is presented, for example, by presenting a recommendation to reduce the energy output from the current energy output or to stop the energy output on a monitor screen seen by a doctor performing surgery because there is a risk of thermal damage. In this way, the doctor can reduce or stop the energy output at his / her discretion after notifying the doctor of the risk of thermal damage.

[0070] 10. Seventh embodiment The seventh embodiment shown in Fig. 25 is different from the first embodiment shown in Fig. 7 in S22A regarding important tissue detection. In the seventh embodiment, important tissue is detected not from an endoscopic image but from an , jutsu The area is estimated by 3D matching with the previous image. Important tissues are detected by 3D matching with CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). In this way, it is possible to detect important tissues that cannot be seen with an endoscope.

[0071] The system of the present embodiment can also be realized as a program. That is, the program of the present embodiment can cause a computer to execute the following: acquiring in real time an image of at least one energy device outputting energy and at least one living tissue that receives energy supply and outputs energy; estimating image recognition information from the captured image by processing based on a trained model that has been trained to output image recognition information that is at least one of information on a thermal diffusion region from the energy device to the living tissue and a specific tissue region in the living tissue generated by the energy output of the energy device from a learning device tissue image that is an image of at least one energy device outputting energy and at least one living tissue, or a learning tissue image that is an image of at least one living tissue; and determining a risk of thermal damage to a specific tissue due to the energy output of the energy device from the estimated image recognition information. Here, a network terminal such as a personal computer is assumed as the computer. However, the computer may be a wearable terminal such as a smartphone, a tablet terminal, or a smart watch. In this way, the same effect as above can be obtained.

[0072] The system 10 of the present embodiment described above includes a storage unit 120 that stores a trained model 121, and a control unit 110. The trained model 121 is trained to estimate a thermal diffusion region from the energy device 310 to the biological tissue and a specific tissue region in the biological tissue, which are generated by the energy output of the energy device 310 from a training device tissue image or a training tissue image. The training device tissue image is an image of at least one energy device 310 that receives energy supply and performs energy output, and at least one biological tissue. The training tissue image is an image of at least one biological tissue. The control unit 110 acquires a captured image that is an image during energy output in which at least one energy device 310 and at least one biological tissue are captured. The control unit 110 estimates a thermal diffusion region and a specific tissue region from the captured image by processing based on the trained model 121 stored in the storage unit 120. The control unit 110 determines the risk of thermal damage to the specific tissue due to the energy output of the energy device 310 from the estimated thermal diffusion region and specific tissue region.

[0073] According to this embodiment, the system 10 acquires an image of the energy device 310 and the biological tissue, estimates the thermal diffusion region and the specific tissue region by processing based on the trained model, and determines the risk of thermal damage to the specific tissue by the energy device. Based on this risk of thermal damage, the system 10 can adjust the energy applied to the tissue to be treated to an appropriate amount, thereby reducing the risk of thermal damage to important tissue without noticing thermal diffusion. In this way, the system 10 adjusts the output energy of the energy device, which was previously performed by a doctor, and the burden on the doctor can be reduced. In addition, the system 10 autonomously adjusts the output, making it possible for a non-expert to perform stable treatment. As a result, the stability of the surgery is improved, or the technique is standardized regardless of the doctor's experience. The learning device tissue image, learning tissue image, specific tissue, and important tissue are described in "1. System".

[0074] Furthermore, in this embodiment, control unit 110 may determine that there is a risk of thermal damage when the distance between the specific tissue region and the thermal diffusion region is equal to or less than a preset threshold value.

[0075] According to this embodiment, the distance between the specific tissue region and the thermal diffusion region can be evaluated based on the result estimated from the image acquired by the controller 100, and the distance can be used as a criterion for determining the presence or absence of a risk of thermal damage. The method of determining the risk of thermal damage using a threshold value is described in "4. First embodiment".

[0076] In this embodiment, the threshold value may differ depending on the tissue type of the specific tissue.

[0077] According to this embodiment, the risk of thermal damage occurring in specific tissues where thermal damage is particularly problematic can be avoided. Details are described in "4. First embodiment."

[0078] In the present embodiment, the captured images may be a plurality of endoscopic images captured at different timings.

[0079] According to this embodiment, a thermal diffusion region is created. Therefore, the thermal diffusion region can be estimated taking into account dynamic information, and the estimation accuracy of the thermal diffusion region can be improved. Note that multiple endoscopic images captured at different timings are described in FIG. 16 in "5. Second embodiment".

[0080] In addition, in this embodiment, the captured images are multiple endoscopic images captured at different timings, and the control unit 110 may estimate a thermal diffusion region and a specific tissue region from each image, and output a prediction result of the time for thermal diffusion to reach the specific tissue region based on the multiple thermal diffusion regions and specific tissue regions estimated from each image.

[0081] According to this embodiment, the controller 100 acquires the decrease in the distance between the thermal diffusion region and the specific tissue in a time series, and can predict the time when the thermal diffusion will reach the specific tissue region. This allows the surgeon to adjust the output of the energy device in advance based on the prediction result. This reduces the risk of thermal damage to the specific tissue region, and allows the surgery to be performed safely. The prediction of the time when the thermal diffusion will reach the specific tissue region is described in FIG. 17 of "5. Second embodiment".

[0082] In this embodiment, the control unit 110 may determine the risk of thermal damage before the time that is the predicted result, and output the determination result.

[0083] According to this embodiment, the result of the determination of the risk of thermal damage is output before the thermal diffusion reaches the specific tissue region. Therefore, it becomes possible to adjust the energy output before the specific time when the thermal damage occurs in the specific tissue, and the risk of the thermal damage occurring in the specific tissue region can be reduced, and the operation can be performed safely. The prediction of the time when the thermal diffusion reaches the specific tissue region is described in FIG. 17 of "5. Second embodiment".

[0084] Furthermore, in this embodiment, the control unit 110 may output an energy output adjustment instruction, which is an instruction to reduce the energy output from the current energy output or an instruction to stop the energy output, based on the result of the judgment, to the generator 300, which controls the amount of energy supply to the energy device 310 based on the energy output adjustment instruction.

[0085] According to this embodiment, the system 10 can autonomously adjust the energy output according to the risk of thermal damage occurring in the specific tissue region. Therefore, even a non-expert can avoid the risk of thermal damage occurring in the specific tissue region. The energy output adjustment instruction is described in FIG. 11 in "4. First embodiment".

[0086] Furthermore, in this embodiment, the control unit 110 may present a recommendation to reduce the energy output from the current energy output or a recommendation to stop the energy output, based on the result of the determination.

[0087] According to this embodiment, a recommendation to reduce or stop the energy output can be presented. Therefore, based on this, the person performing the surgery can adjust the energy output. Details are described in FIG. 24 in "9. Sixth embodiment."

[0088] In addition, in this embodiment, the trained model 121 is a trained model that has been trained to estimate the thermal diffusion region and specific tissue region from the energy device 310 to the tissue from the training energy output information of the energy device 310 and the training device tissue image, or the training tissue image, and the control unit 110 may estimate the thermal diffusion region and specific tissue region from the energy output information of the energy device 310 and the captured image by processing based on the trained model 121 stored in the memory unit 120.

[0089] According to this embodiment, the history information of the output energy can also be fed back to the learning model 522 in the learning stage. Therefore, the control unit 110 can estimate the thermal diffusion region with higher accuracy. The learning energy output information is described in FIG. 15 in "5. Second embodiment".

[0090] In this embodiment, the energy device 310 may be a device having two jaws 337, 338 capable of grasping tissue, and may be a device that receives energy supply from the generator 300 and outputs energy from the two jaws 337, 338.

[0091] That is, the energy device 310 may be a bipolar device 330. The bipolar device is described in, for example, FIG. 5 in “3. Energy Device”.

[0092] In this embodiment, the energy device 310 may be an ultrasonic device 340 .

[0093] Here, the ultrasonic device 340 may be a combined device of the ultrasonic device 340 and the bipolar device 330. The ultrasonic device 340 and the combined device are described in, for example, FIG. 6 in “3. Energy Device”.

[0094] In this embodiment, the captured image may include white fading due to thermal denaturation.

[0095] According to this embodiment, information on the white-washed area can be obtained from the endoscopic image, so there is no need to use other devices when creating the teacher data 521. In addition, the number of networks used by the control unit 110 can be reduced, and the processing speed of the control unit 110 can be increased. Note that white-washed areas are described in "7. Fourth embodiment."

[0096] In this embodiment, the captured image may include an endoscopic image captured using special light that is different from normal light.

[0097] According to this embodiment, by using multiple wavelength images as time series data when learning the network used in the thermal diffusion detection function and estimating the thermal diffusion region, it is possible to additionally input information focusing on specific tissues, such as connective tissues around blood vessels. In this way, it becomes easier to obtain dynamic information, such as the aggregation of connective tissues, and by utilizing this information, it is possible to improve the accuracy of estimation of specific tissue regions, thermal diffusion regions, etc. Note that normal light and special light are described in "6. Third embodiment".

[0098] In addition, in this embodiment, the trained model 121 is a trained model trained to estimate tissue heat transfer characteristics from the training energy output information of the energy device 310, a training device tissue image, or a training tissue image, and the control unit 110 may estimate the tissue heat transfer characteristics from the energy output information or the captured image of the energy device 310 by processing based on the trained model 121 stored in the memory unit 120.

[0099] According to this embodiment, the control unit 110 can predict the reduction in the distance between the thermal diffusion region and the specific tissue region from the characteristics of the heat transfer path. Therefore, the risk of thermal damage to the specific tissue region is reduced, and surgery can be performed safely. The tissue heat transfer characteristics are described in "8. Fifth embodiment."

[0100] The above processing may be written as a program. That is, the program of the present embodiment causes the controller 100 to execute the following operations: acquiring a captured image, estimating image recognition information, which is at least one of information on a thermal diffusion region from the energy device to the biological tissue and information on a specific tissue region in the biological tissue, from the captured image by processing based on the trained model 121, and determining a risk of thermal damage to the specific tissue due to the energy output of the energy device 310 from the estimated image recognition information.

[0101] The above processing may be described as a method. That is, the method of the present embodiment includes acquiring in real time an image of at least one energy device outputting energy and at least one living tissue, estimating image recognition information from the image by processing based on a trained model that has been trained to output image recognition information, which is at least one of information on a thermal diffusion region from the energy device to the living tissue and information on a specific tissue region in the living tissue, generated by the energy output of the energy device, from a learning device tissue image, which is an image of at least one energy device outputting energy and at least one living tissue, or a learning tissue image, which is an image of at least one living tissue, and judging a risk of thermal damage to a specific tissue due to the energy output of the energy device from the estimated image recognition information.

[0102] Although the present embodiment and its modified examples have been described above, the present disclosure is not limited to each embodiment or its modified example as it is, and in the implementation stage, the components can be modified and embodied within the scope of the gist. In addition, a plurality of components disclosed in each of the above-mentioned embodiments and modified examples can be appropriately combined. For example, some components may be deleted from all the components described in each embodiment or modified example. Furthermore, the components described in different embodiments or modified examples may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the present disclosure. In addition, a term described at least once in the specification or drawings together with a different term having a broader meaning or the same meaning can be replaced with the different term in any part of the specification or drawings. [Explanation of symbols]

[0103] 10 system, 100 controller, 110 control unit, 111 thermal diffusion detection unit, 112 vital tissue detection unit, 113 output setting unit, 114 thermal damage risk judgment unit, 120 memory unit, 121 trained model, 122 first trained model, 123 second trained model, 180 I / O device, 190 I / O device, 200 endoscope system, 210 endoscope, 220 main unit, 230 display, 300 generator, 310 energy device, 320 monopolar device, 321 electrode, 322 insertion unit, 323 operation unit, 325 cable, 330 bipolar device, 331 tip, 332 insertion unit, 333 operation unit, 335 cable, 336 base end, 337 jaw, 338 jaw, 340 ultrasonic device, 341 Distal end portion, 342 insertion portion, 343 operation portion, 344a operation button, 344b operation button, 345 cable, 346 base end, 347 jaw, 348 probe, 500 learning device, 510 processing unit, 520 memory unit, 521 teacher data, 521A first teacher data, 521B second teacher data, 522 learning model, 522A first learning model, 522B second learning model, M margin, S1 step, S21 step, S22B step, S24 step, S2A step, S2B step, S3 step, T energy output time, n specified time, t0 estimated time

Claims

1. a memory unit that stores a trained model that has been trained to estimate a thermal diffusion region from the energy device to the biological tissue and a specific tissue region in the biological tissue, which are generated by the energy output of the energy device, from a training device tissue image that is an image of at least one energy device outputting energy in response to energy supply and at least one biological tissue, or from a training tissue image that is an image of the training device tissue image and the at least one biological tissue; A control unit; Including, The control unit is Obtaining an image of at least one of the energy devices and at least one of the biological tissues during energy output; Estimating the thermal diffusion region and the specific tissue region from the captured image by processing based on the trained model stored in the storage unit; A system that determines a risk of thermal damage to the specific tissue region due to the energy output of the energy device from the estimated thermal diffusion region and the specific tissue region.

2. 2. The system of claim 1, The control unit is A system characterized by determining that there is a risk of thermal damage when the distance between the specific tissue region and the thermal diffusion region is below a predetermined threshold.

3. 3. The system according to claim 2, The system, wherein the threshold varies depending on the tissue type of the specific tissue region.

4. 2. The system of claim 1, The system is characterized in that the captured images are a plurality of endoscopic images, each of which is captured at a different timing.

5. 5. The system according to claim 4, The control unit is Estimating the thermal diffusion region and the specific tissue region from each of the images; A system characterized by outputting a prediction result of the time it will take for thermal diffusion to reach the specific tissue region based on the multiple thermal diffusion regions and the specific tissue region estimated from each of the images.

6. 6. The system according to claim 5, The control unit is A system that determines a risk of thermal damage before the time that is the predicted result and outputs the determination result.

7. 2. The system of claim 1, The control unit is A system characterized by outputting an energy output adjustment instruction, which is an instruction to reduce the energy output from a current energy output or an instruction to stop the energy output, based on the result of the judgment, to a generator that controls the amount of energy supply to the energy device based on the energy output adjustment instruction.

8. 2. The system of claim 1, The control unit is A system characterized by presenting a recommendation to reduce energy output from a current energy output or to stop energy output based on the result of the judgment.

9. 2. The system of claim 1, The energy device comprises: A system comprising a device having two jaws capable of grasping tissue, the device receiving the energy supply from a generator and outputting energy from the two jaws.

10. 2. The system of claim 1, The system, wherein the energy device is an ultrasonic device.

11. 2. The system of claim 1, The system is characterized in that the captured image includes a white tinge due to thermal denaturation.

12. 2. The system of claim 1, The system is characterized in that the captured image includes an endoscopic image captured using special light that is different from normal light.

13. 2. The system of claim 1, The trained model is a trained model that is trained to estimate tissue heat transfer characteristics from the training energy output information of the energy device, the training device tissue image, or the training tissue image; The control unit is A system characterized by estimating the tissue heat transfer characteristics from energy output information of the energy device or the captured image by processing based on the trained model stored in the memory unit.

Images