System, program and information processing method

Abstract

This system (10) includes a control unit (110) and a storage unit (120) for storing a trained model (121) . The control unit acquires at least one treatment image including images in which at least one energy device (310) and at least one living tissue are captured. The control unit acquires information relating to an energy supply quantity, which is information relating to the energy output setting of the energy device. The control unit estimates estimated residual heat information, which is information relating to the amount of heat or temperature of the energy device, on the basis of the information relating to the energy supply quantity and the trained model. The control unit causes a notification unit (230) to carry out notification based on the estimated residual heat information.

Description

[Technical field]

[0001] The present invention relates to a system, a program, an information processing method, and the like. [Background technology]

[0002] The medical device disclosed in Patent Document 1 displays the presence or absence of a high-temperature device and a warning image in the endoscopic image when a high-temperature energy treatment tool is present in the endoscopic image after use of the energy treatment tool, and displays an image showing the direction of the treatment tool when a high-temperature energy treatment tool is not present in the endoscopic image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2018 / 0160910 Summary of the Invention [Problem to be solved by the invention]

[0004] In the medical device disclosed in Patent Document 1, the specific residual heat temperature is not displayed on the endoscopic image, so the doctor has to judge whether the temperature of the treatment tool has dropped or not. However, this judgment depends on the type of treatment tool, the output energy, or the tissue to be treated, so it depends on the doctor's experience. Therefore, it is difficult to realize an efficient operation regardless of the doctor's experience. In addition, the medical device disclosed in Patent Document 1 uses a thermo camera or the like to measure temperature, but using a thermo camera or the like to measure temperature has the problem of increasing the number of ports in laparoscopic surgery and increasing the thickness of the sensor-equipped device. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to a system including: a memory unit that stores a trained model trained to output residual heat information, which is information regarding the amount of heat or temperature remaining in an energy device after energy is supplied to the energy device, from a training device tissue image, which is an image of at least one energy device and at least one biological tissue that receive energy supply and output energy, or a training tissue image, which is an image of at least one biological tissue, and training information regarding an energy supply amount, which is information regarding an energy output setting of the energy device; and a control unit, wherein the control unit acquires at least one treatment image including an image of the at least one energy device and the at least one biological tissue, acquires information regarding the energy supply amount, which is information regarding an energy output setting of the energy device, estimates estimated residual heat information, which is information regarding the amount of heat or temperature of the energy device, based on the treatment image, the information regarding the energy supply amount, and the trained model, and causes an alarm unit to issue an alarm based on the estimated residual heat information.

[0006] Another aspect of the present disclosure relates to a program that causes a computer to execute the following steps: acquire at least one treatment image including an image of at least one energy device and at least one biological tissue; acquire information regarding the amount of energy supply, which is information regarding the energy output setting of the energy device; estimate estimated residual heat information, which is information regarding the amount of heat or temperature of the energy device, based on the treatment image, the information regarding the amount of energy supply, and the trained model, from a learning device tissue image, which is an image of at least one energy device and at least one biological tissue that receives energy supply and outputs energy, or a learning tissue image, which is an image of at least one biological tissue, by processing based on a trained model that has been trained to output residual heat information, which is information regarding the amount of heat or temperature remaining in the energy device after energy is supplied to the energy device; and display the estimated residual heat information on a monitor.

[0007] Another aspect of the present disclosure relates to an information processing method including: acquiring at least one treatment image including an image of at least one energy device and at least one biological tissue; acquiring information regarding an energy supply amount, which is information regarding an energy output setting of the energy device; estimating estimated residual heat information, which is information regarding the heat amount or temperature of the energy device, based on the treatment image, the information regarding the energy supply amount, and the trained model, from a learning device tissue image, which is an image of at least one energy device and at least one biological tissue that receive energy supply and output energy, or a learning tissue image, which is an image of at least one biological tissue, by processing based on a trained model that has been trained to output residual heat information, which is information regarding the heat amount or temperature remaining in the energy device after energy is supplied to the energy device; and displaying the estimated residual heat information on a monitor. [Brief description of the drawings]

[0008] [Figure 1] An example of system configuration. [Diagram 2] An example of controller configuration. [Diagram 3] 4 is a flowchart illustrating a process performed by the system. [Figure 4] An example of a monopolar device configuration. [Diagram 5] An example of a bipolar device configuration. [Figure 6] An example of an ultrasonic device configuration. [Figure 7] 3 shows a processing example of the first embodiment. [Figure 8] Example of treatment image. [Figure 9] An example of residual heat information display. [Figure 10] An example of residual heat information display. [Figure 11] An example of residual heat information display. [Figure 12] An example of residual heat information display. [Figure 13] An example of the configuration of a learning device. [Figure 14] FIG. 13 is an explanatory diagram of the learning stage for estimating an energy device. [Figure 15] FIG. 1 is an illustration of the learning stages for tissue type estimation. [Figure 16] FIG. 11 is an explanatory diagram of a learning stage for estimating residual heat information. [Figure 17] FIG. 4 is a graph showing the time dependence of the temperature of an energy device. [Figure 18] 13 is an example of an alert displayed by the notification unit. [Figure 19] 13 is an example of an alert displayed by the notification unit. [Figure 20] FIG. 11 is an explanatory diagram of a process for issuing an alert. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the contents described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0010] 1. System Fig. 1 shows an example of the configuration of a system 10 in this embodiment. Fig. 1 shows an example of the system configuration when an operative field is photographed by an endoscope. The system 10 shown in Fig. 1 includes a controller 100, an endoscopic system 200, a generator 300, and an energy device 310. The system 10 is a surgical system for performing surgery using at least one energy device under an endoscope. Here, an example is shown in which the system 10 includes one energy device 310, but the system 10 may include multiple energy devices.

[0011] The endoscope system 200 is a system that performs imaging using an endoscope, image processing of the endoscopic image, and monitor display of the endoscopic image. The endoscope system 200 includes an endoscope 210, a main unit 220, and a notification unit 230. Here, a rigid endoscope for surgical operations will be described as an example.

[0012] The endoscope 210 includes an insertion section inserted into a body cavity, an operation section connected to a base end of the insertion section, a universal cord connected to the base end of the operation section, and a connector section connected to the base end of the universal cord. The insertion section includes a rigid tube, an objective optical system, an image sensor, an illumination optical system, a transmission cable, and a light guide. The objective optical system and the image sensor for photographing the inside of the body cavity, and the illumination optical system for illuminating the inside of the body cavity are provided at the tip of a long and thin cylindrical rigid tube. The tip of the rigid tube may be configured to be bendable. A transmission cable for transmitting an image signal acquired by the image sensor, and a light guide for guiding illumination light to the illumination optical system are provided inside the rigid tube. The operation section is held by a user and accepts operations from the user. The operation section is provided with buttons to which various functions are assigned. When the tip of the insertion section is bendable, the operation section is provided with an angle operation lever. The connector portion includes a video connector that detachably connects the transmission cable to the main device 220 , and a light guide connector that detachably connects the light guide to the main device 220 .

[0013] The main unit 220 includes a processing device that controls the endoscope, processes the endoscopic image, and displays the endoscopic image, and a light source device that generates illumination light and controls the illumination light. The main unit 220 is also called a video system center. The processing device is composed of a processor such as a CPU, processes the image signal transmitted from the endoscope 210 to generate an endoscopic image, and outputs the endoscopic image to the notification unit 230 and the controller 100. The illumination light emitted by the light source device is guided to the illumination optical system by a light guide, and is emitted from the illumination optical system into the body cavity.

[0014] The energy device 310 is a device that outputs energy from its tip by high-frequency power or ultrasound, etc., to perform treatment such as coagulation, sealing, hemostasis, incision, resection, or peeling on tissue that its tip contacts. The energy device 310 is also called an energy treatment tool. The energy device 310 is a monopolar device that passes high-frequency power between an electrode at the tip of the device and an electrode outside the body, a bipolar device that passes high-frequency power between two jaws, an ultrasonic device that is provided with a probe and jaws and emits ultrasound from the probe, or a combination device that passes high-frequency power between the probe and jaws and emits ultrasound from the probe.

[0015] The generator 300 supplies energy to the energy device 310, controls the energy supply, and acquires electrical information from the energy device 310. When the energy device 310 outputs high-frequency energy, the generator 300 supplies high-frequency power, and the energy device 310 outputs the high-frequency power from an electrode or a jaw. When the energy device 310 outputs ultrasonic energy, the generator 300 supplies power, and the probe of the energy device 310 converts the power into ultrasonic waves and outputs them.

[0016] The electrical information is electrical information of the tissue in contact with the electrodes or jaws of the energy device 310, specifically, information obtained as a response to the energy device 310 outputting high-frequency power to the tissue. The electrical information is, for example, impedance information of the tissue to be treated by the energy device 310. However, as described below, the electrical information is not limited to impedance information.

[0017] The generator 300 performs control to change the energy output from the energy device 310 over time according to the output sequence. The generator 300 may change the energy output in response to a change in impedance information over time. In this case, the output sequence may specify how to change the energy output in response to a change in impedance information. The generator 300 may also automatically turn off the energy output in response to a change in impedance information over time. For example, when the impedance rises above a certain level, the generator 300 may determine that the treatment is complete and turn off the energy output.

[0018] The controller 100 acquires images and energy information from the endoscope system 200 and the generator 300. Then, it performs residual heat estimation and outputs the estimation result to a monitor to inform the surgeon.

[0019] The information on the biological tissue includes not only a specific organ but also parts associated with the organ, such as tissues connecting the organs. The energy device 310 is an energy device such as a monopolar device 320, a bipolar device 330, an ultrasonic device 340, etc., as described later with reference to Figs. 4 to 6, but may be other energy devices.

[0020] The generator 300 adjusts the energy output of the energy device 310 according to the energy output adjustment instruction. That is, the system 10 of this embodiment is a system that automatically adjusts the energy output from the energy device 310 based on an endoscopic image. The generator 300 supplies energy to the energy device 310 at an energy supply amount instructed by the energy output adjustment instruction, and the energy device 310 receives the energy supply and outputs energy, thereby adjusting the energy output according to the energy output adjustment instruction.

[0021] The energy output adjustment instruction is an instruction to increase or decrease the output of the entire waveform of the output sequence, or an instruction to set one of a plurality of selectable output sequences. For example, when the energy output from the energy device 310 is adjustable by stepwise magnification, the energy output adjustment instruction is an instruction indicating the stepwise magnification of the energy output. The generator 300 increases or decreases the high frequency output or ultrasonic output according to the indicated magnification. The magnification may be continuously adjustable. Alternatively, when a plurality of output sequences are provided, the energy output adjustment instruction is an instruction indicating one of the plurality of output sequences. The generator 300 performs energy output from the energy device 310 according to the indicated output sequence. The energy output adjustment instruction may include both an instruction to increase or decrease the energy output and a instruction to change the output sequence.

[0022] 2. Controller Fig. 2 shows an example of the configuration of the controller 100. The controller 100 controls the entire system 10. The controller 100 includes a control unit 110, a storage unit 120, and I / O devices 170, 180, and 190. Figs. 1 and 2 show an example in which the controller 100 is configured as a device separate from the generator 300. In this case, the controller 100 is configured as an information processing device such as a PC or a server device. Alternatively, the controller 100 may be realized as a cloud system in which processing is executed by one or more information processing devices connected via a network.

[0023] The I / O device 180 receives image data of an endoscopic image from the main unit 220 of the endoscope system 200. The I / O device 180 is a connector to which an image transmission cable is connected, or an interface circuit that is connected to the connector and communicates with the main unit 220.

[0024] The I / O device 190 is an interface circuit used for output to the notification unit 230. The I / O device 190 is connected to a connector to which a cable for signal transmission is connected, or to the connector, and communicates with the notification unit 230. The I / O device 170 is an interface circuit used for power supply.

[0025] The control unit 110 estimates estimated residual heat information, which is information related to the heat quantity or temperature of the energy device 310, from the endoscopic image by image recognition processing using the trained model 121, and causes the notification unit 230 to perform notification based on the estimated residual heat information. The control unit 110 includes one or more processors as hardware. The processor is a general-purpose processor such as a central processing unit (CPU), a graphical processing unit (GPU), or a digital signal processor (DSP). Alternatively, the processor may be a dedicated processor such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0026] The storage unit 120 stores a trained model 121 used in the image recognition process. When the image recognition process is performed by a general-purpose processor, the storage unit 120 stores a program in which an inference algorithm is described and parameters used in the inference algorithm as the trained model 121. The trained model 121 includes a first trained model 122, a second trained model 123, and a third trained model 124. The first trained model 122 is a trained model 121 related to the estimation of biological tissue, which will be described in FIG. 14 below. The second trained model 123 is a trained model 121 related to the estimation of the energy device 310, and the third trained model 124 is a trained model 121 related to the estimation of residual heat. When the image recognition process is performed by a dedicated processor in which the inference algorithm is implemented as hardware, the storage unit 120 stores parameters used in the inference algorithm as the trained model 121.

[0027] The storage unit 120 is a storage device such as a semiconductor memory, a hard disk drive, an optical disk drive, etc. The semiconductor memory is a RAM, a ROM, a non-volatile memory, etc.

[0028] As an inference algorithm for image recognition processing, for example, a neural network can be adopted. The weight coefficients and biases of the node connections in the neural network are parameters. The neural network includes an input layer to which image data is input, an intermediate layer that performs arithmetic processing on the data input through the input layer, and an output layer that outputs a recognition result based on the arithmetic result output from the intermediate layer. As a neural network used for image recognition processing, for example, a CNN (Convolutional Neural Network) can be adopted.

[0029] The control unit 110 also includes a first recognition unit 111, a second recognition unit 112, an estimation unit 113, an output information determination unit 114, and a presentation unit 115. The storage unit 120 stores a program in which the functions of the first recognition unit 111, the second recognition unit 112, the estimation unit 113, the output information determination unit 114, and the presentation unit 115 are described. One or more processors of the control unit 110 read the program from the storage unit 120 and execute the program to realize the functions of the first recognition unit 111, the second recognition unit 112, the estimation unit 113, the output information determination unit 114, and the presentation unit 115. The program in which the functions of the units are described can be stored in a non-transitory information storage medium that is a computer-readable medium. The information storage medium can be realized by, for example, an optical disk, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM or a non-volatile memory.

[0030] 3 is a flowchart showing the process when the controller 100 analyzes the residual heat information of the energy device 310 and presents the information to a doctor. First, in step S1, the control unit 110 acquires an endoscopic image and energy information from the main body device 220 of the endoscope system 200 via the I / O device 180. The endoscopic image is, for example, an image of the energy device 310 or biological tissue or both, and is also called a processed image. The energy information is, for example, information such as energy output settings and output history.

[0031] Next, the control unit 110 executes steps S2Aa, S2Ab, and S2B. In step S2Aa, the first recognition unit 111 performs image recognition processing on the endoscopic image using the first trained model 122 to estimate the energy device 310 from the image. In step S2Ab, the second recognition unit 112 performs image recognition processing on the endoscopic image using the second trained model 123 to estimate biological tissue from the image. Meanwhile, in step S2B, the estimation unit 113 performs image recognition processing on the endoscopic image using the third trained model 124 to estimate the residual heat temperature of the tip portion of the energy device 310. The tip portion refers to the vicinity of the part where energy is output from the energy device 310. For example, in the case where the energy device 310 is a bipolar device 330 shown in FIG. 5 described later, the tip portion refers to the blade, pad, and shaft shown in FIG. 12 described later. The information about the energy device 310 estimated in step S2Aa and the information about the living tissue estimated in step S2Ab are information that affects the residual heat information of the energy device 310 and the thermal diffusion to the living tissue.

[0032] Next, in step S3, the output information determination unit 114 constructs information to be presented to the doctor based on the estimation results in steps S2Aa and S2B. For example, the output information determination unit 114 constructs information about the display of the energy device 310, biological tissue, temperature, etc. on the endoscopic image and the display location thereof.

[0033] Next, in step S4, the presentation unit 115 creates an image to be displayed on the notification unit based on information about the contents of the display location, etc., constructed by the output information determination unit 114. For example, a single image is created by superimposing a display of a temperature range, etc., to be displayed on an endoscopic image. Then, in step 5, the notification unit 230 displays the image created by the presentation unit 115 in response to an instruction from the control unit 110.

[0034] 3. Energy Devices As examples of the energy device 310, a monopolar device 320, a bipolar device 330, an ultrasonic device 340, and a combination device will be described below.

[0035] 4 shows a configuration example of monopolar device 320. Monopolar device 320 includes an elongated cylindrical insertion section 322, an electrode 321 provided at the tip of insertion section 322, an operation section 323 connected to the base end of insertion section 322, and a cable 325 connecting operation section 323 to a connector (not shown). The connector is detachably connected to generator 300.

[0036] The high frequency power output by the generator 300 is transmitted through a cable 325 and output from the electrode 321. A return electrode is provided outside the patient's body, and electricity flows between the electrode 321 and the return electrode. As a result, high frequency energy is applied to the tissue in contact with the electrode 321, and Joule heat is generated in the tissue. The electrode 321 may have various shapes depending on the type of treatment. The monopolar device 320 can adjust the degree of coagulation and incision by changing the current pattern.

[0037] FIG. 5 shows a configuration example of the bipolar device 330. The bipolar device 330 includes an elongated cylindrical insertion section 332, two jaws 337 and 338 provided at a tip section 331 of the insertion section 332, an operation section 333 connected to a base end of the insertion section 332, and a cable 335 connecting the operation section 333 to a connector (not shown). The connector is detachably connected to the generator 300. The jaws 337 and 338 are movable sections for grasping tissue and applying energy to the grasped tissue, and are configured to be openable and closable around an axis provided at the base end 336. The operation section 333 is provided with a gripping section for operating the opening and closing of the jaws 337 and 338. When a doctor grips the gripping section, the jaws 337 and 338 close and grasp the tissue.

[0038] The high-frequency power output by the generator 300 is transmitted by the cable 335, and when the jaws 337, 338 grasp tissue, electricity is applied between the two jaws 337, 338. As a result, high-frequency energy is applied to the tissue sandwiched between the two jaws 337, 338, Joule heat is generated in the tissue, and the tissue is coagulated. The generator 300 may measure impedance information of the tissue grasped by the jaws 337, 338, detect the completion of the treatment based on the impedance information, and automatically stop the energy output. The generator 300 may also automatically adjust the energy applied to the tissue based on the impedance information. For example, the device temperature of the bipolar device 330 rises to about 100 degrees Celsius.

[0039] A derivative of the bipolar device is the vessel sealing device, which is a device with a cutter attached to the jaws of the bipolar device, and cuts the tissue by running the cutter through it after coagulating it with electricity.

[0040] FIG. 6 is a configuration example of an ultrasonic device 340. The ultrasonic device 340 includes an elongated cylindrical insertion section 342, a jaw 347 and a probe 348 provided at a tip end 341 of the insertion section 342, an operation section 343 connected to a base end of the insertion section 342, and a cable 345 connecting the operation section 343 and a connector (not shown). The connector is detachably connected to the generator 300. The jaw 347 is movable around an axis provided at a base end 346, and is configured to be openable and closable relative to the immovable probe 348. The operation section 343 is provided with a gripping section for operating the opening and closing of the jaw 347. When a doctor grips the gripping section, the jaw 347 closes, and the jaw 347 and the probe 348 grip the tissue. The operation section 343 is provided with an operation button 344a to which a first output mode is assigned, and an operation button 344b to which a second output mode is assigned. The output mode is selected according to the treatment contents, and when the operation button for each output mode is pressed, ultrasonic energy is output in the output sequence of that mode.

[0041] The power output by the generator 300 is transmitted by a cable 335, and when an operation button 344a or 344b is pressed, the probe 348 converts the power into ultrasonic waves and outputs the ultrasonic waves. As a result, frictional heat is generated in the tissue sandwiched between the jaw 347 and the probe 348, and the tissue is coagulated or incised. Regarding the thermal diffusion of the ultrasonic device, the thermal diffusion of the ultrasonic device 340 is smaller than that of a high-frequency device, for example, but the device temperature of the ultrasonic device 340 rises to nearly 200 degrees Celsius.

[0042] The combination device using high-frequency power and ultrasound is configured similarly to the ultrasound device in FIG. 6. However, the combination device can generate Joule heat in the tissue held by the jaws and the probe by passing high-frequency power between the jaws and the probe, and can coagulate the tissue. Similarly to the ultrasound device, the combination device can incise the tissue held by the jaws and the probe by outputting ultrasound from the probe. One of the two operation buttons provided on the operation unit is assigned to the high-frequency mode, and the other is assigned to the seal & cut mode. The high-frequency mode is a mode in which treatment such as coagulation is performed only by high-frequency energy output. The seal & cut mode is a mode in which high-frequency energy and ultrasound energy are used in combination, and the tissue is coagulated and cut by high-frequency energy output.

[0043] In the following embodiment, a case where a bipolar device 330 is mainly used as the energy device 310 will be described as an example. However, this embodiment is applicable to the case where the above-mentioned various energy devices in which thermal diffusion may occur are used.

[0044] 4. First embodiment FIG. 7 is a processing example of the first embodiment. First, as shown in S21A, an endoscopic image is input to the control unit 110 by the processing of the control unit 110. Specifically, each frame image of a moving image captured by the endoscope is sequentially input to the control unit 110. In the endoscopic image, one or more energy devices 310 and one or more living tissues are captured. In the following description, the endoscopic image is appropriately referred to as a treatment image. FIG. 8 is an example of a treatment image that is input information in S21A of FIG. 7. In this way, the treatment image input to the first recognition unit 111 and the second recognition unit 112 includes the energy device 310 and the living tissue.

[0045] Next, in S22A, the first recognition unit 111 of the control unit 110 detects the energy device 310 from the treatment image by executing an estimation program adjusted by machine learning described later in FIG. 14. The estimation program is a program that executes a trained model 121 that learns the relationship between the amount of heat stored according to the energy output information, the amount of heat released according to the type of tissue, and the tissue gripping time by the treatment tool, as described later in FIG. 14 to FIG. 16. In S22A, the first recognition unit 111 detects the energy device 310 by the control unit 110 inputting the treatment image into a network having the estimation program. Specifically, the first recognition unit 111 extracts device information from the treatment image. Here, the device information is the type, the range of existence, or the state of the tip part of the energy device 310. The state of the tip part is, for example, the open / closed state of the jaws 337 and 338. The type of device may be input manually by a doctor. In addition, the treatment image and the type of device obtained from the generator 300 or the like are input into the estimation program of this function to detect the energy device 310 in the treatment image.

[0046] Then, the control unit 110 estimates an area indicating the energy device 310 from among the subjects captured in the treatment image based on the estimation program, and labels the area by applying a color. In this manner, the first recognition unit 111 of the control unit 110 extracts device information, which is information related to the energy device 310, from the treatment image. Also, in S22A, the time during which the energy device 310 gripped the biological tissue is detected from a plurality of endoscopic images. Then, the control unit 110 outputs the labeled treatment image and the gripping time to the second recognition unit 112.

[0047] Then, as shown in S22B of FIG. 7, the second recognition unit 112 of the control unit 110 executes the estimation program described above to detect the type of biological tissue from the treatment image. Here, the type of biological tissue to be detected refers to the biological tissue that is the treatment target of the surgery, regardless of whether the energy device 310 grasps the biological tissue. In addition to the biological tissue, the grasping amount or grasping time can also be detected. In this case, the grasping time is the elapsed time from grasping, and can be recognized from the treatment image. The type of biological tissue is, for example, a large blood vessel, pancreas, duodenum, etc., but is not limited to these. The control unit 110 detects the type of biological tissue shown in the treatment image by inputting the above-mentioned labeled treatment image in the first recognition unit 111 into a network having an estimation program that has been annotated and learned with the type of biological tissue. The input of the type of biological tissue may be manual input by a doctor. In addition to the type of biological tissue, the state of the biological tissue, for example, wet, dry, etc. can also be detected. Then, labeling of the type of biological tissue estimated above is performed. For example, this is done by adding color to a portion that is a region of a certain biological tissue. In this way, the second recognition unit 112 extracts tissue information, which is information about the biological tissue, from the treatment image. Note that the biological tissue can also be detected by 3D matching with CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) instead of the endoscopic image.

[0048] Next, as shown in S23 of FIG. 7, the estimation unit 113 of the control unit 110 receives the estimation results of the first recognition unit 111 and the second recognition unit 112 and estimates the remaining heat information based on the estimation results. Here, as shown in S24, the estimation unit 113 can also perform the above estimation using information on the energy supply amount in addition to the estimation results of the first recognition unit 111 and the second recognition unit 112. The information on the energy supply amount is mainly information on the energy output setting of the energy device, for example, the energy output amount or output time, and the information can be obtained, for example, from the energy device 310. The information on the energy supply amount can also be obtained from the generator 300 or the like. That is, the estimation unit 113 estimates the remaining heat information of the energy device 310 by inputting the device information, the tissue information, and, if there is information on the energy supply amount, the information into an estimation program described in FIG. 13 described later. The remaining heat information is information on, for example, the temperature, heat amount, etc. of the tip part of the energy device 310. The residual heat information estimated by the estimation unit 113 is called estimated residual heat information. In this manner, the estimation unit 113 of the control unit 110 estimates the estimated residual heat information. For example, the estimation unit 113 estimates the target tissue to be treated by the energy device 310 from the tissue information and device information. The estimation unit 113 can also obtain information about the tissue around the energy device 310 from the estimated target tissue to be treated, and estimates the heat capacity of the heat transfer path based on this. The estimation unit 113 can estimate the estimated residual heat information of the energy device 310 based on the heat capacity of the heat transfer path. The estimation unit 113 may estimate the estimated residual heat information during the energy supply, not just after the energy supply. For example, the estimation unit 113 can estimate the target tissue to be treated by the energy device 310 based on the device information and tissue information. The heat capacity of the heat transfer path can be estimated from the estimated target tissue to be treated, and the residual heat information of the tip portion of the energy device 310 can be estimated based on this.

[0049] 7, the output information determination unit 114 makes a determination about the output information based on the output results of A22A, S22B, and S23 so far. Specifically, the output information determination unit 114 collects the information estimated by the first recognition unit 111, the second recognition unit 112, and the estimation unit 113, and determines the information to be presented to the doctor.

[0050] 7, the presentation unit 115 of the control unit 110 causes the notification unit 230 to perform notification based on the estimated residual heat information, which is the information estimated by the estimation unit 113 in S25. The notification method may be to display an icon on the notification unit 230, or to notify by sound, vibration, or the like. The controller 100 may have a function of outputting an energy output adjustment instruction to the generator 300. For example, the energy output adjustment instruction may be output to the generator 300 according to the input of the surgeon via an interface that allows the surgeon to input. Alternatively, the controller 100 may recognize the biological tissue and the energy device 310 from the endoscopic image by image recognition processing using machine learning or the like as described above, and output an energy output adjustment instruction to the generator 300 based on the recognized information.

[0051] 9 to 12 are examples of images superimposed and displayed by the display unit. In the images shown in FIG. 9 to FIG. 12, the current residual heat temperature of each part of the tip of the energy device 310 is displayed by distinguishing it by color. The image shown in FIG. 9 is an example of the case where the relationship between color and temperature is shown by a bar flap in the upper left of the image. The image shown in FIG. 10 is an example of the case where the relationship between color and temperature is shown by a numerical value in the upper left of the image. The image shown in FIG. 11 shows the temperature by a numerical value directly on the part of the tip of the energy device 310 so that the relationship between color and temperature can be seen. In addition, the display of the part of the tip of the energy device 310 may be displayed in the image so that the shaft, blade, and pad can be seen, as in the image shown in FIG. 12. For example, when the blade part reaches 200 degrees Celsius, the shaft part connected by metal may also heat up to about 100 degrees Celsius, but attention is often not paid to the shaft part in particular, and there is a high risk of thermal damage. Therefore, as shown in Figs. 9 to 12, by displaying the temperature for each part of the tip of the energy device 310, the effect of reducing the risk of thermal damage is increased. In this way, the current residual heat temperature for each part can be displayed, for example, by a bar graph or a numerical value. The temperature state for each part can be identified, for example, by adding a color. Note that the superimposed display of colors may cause a problem of poor visibility of the tip of the energy device 310. In such a case, the display setting may be changed so that the doctor does not select the superimposed display of colors. For example, it is possible to set the blade and pad parts of the tip of the energy device 310 in Fig. 12 not to be superimposed with colors.

[0052] 13 shows an example of the configuration of a learning device 500 that performs machine learning of the estimation process of the energy device 310, the biological tissue, and the residual heat information. The learning device 500 includes a processing unit 510 and a storage unit 520. The learning device 500 is realized by an information processing device such as a PC or a server device. Alternatively, the learning device 500 may be realized by a cloud system in which processing is performed by one or more information processing devices connected via a network.

[0053] The processing unit 510 is a processor such as a CPU, and the storage unit 520 is a storage device such as a semiconductor memory or a hard disk drive. The storage unit 520 stores a learning model 522 and teacher data 521. Here, the teacher data 521 includes a first teacher data 521A, a second teacher data 521B, and a third teacher data 521C. The learning model 522 includes a first learning model 522A, a second learning model 522B, and a third learning model 522C. The processing unit 510 trains the learning model 522 using the teacher data 521 to generate a trained model 121. The teacher data 521 includes image data of a plurality of learning images and learning information that is information about the energy output setting of the energy device 310. The learning images and learning information are each provided with correct answer data. Here, the first teacher data 521A is teacher data about the energy device 310. The second teacher data 521B is teacher data about biological tissue, and the third teacher data 521C is teacher data about residual heat information. The first learning model 522A is a learning model about the energy device 310, the second learning model 522B is a learning model about biological tissue, and the third learning model 522C is a learning model about residual heat information. The learning image includes a treatment image in which one or more biological tissues and one or more energy devices 310 are shown. This treatment image is also called a learning device tissue image. The learning image may also include a treatment image in which one or more biological tissues are shown and the energy device 310 is not shown. This treatment image is also called a learning tissue image. The correct answer data is an annotation in segmentation (area detection), an annotation in detection (position detection), a correct answer label in classification, or a correct answer label in regression (regression analysis). The processing unit 510 inputs a learning image to an inference process by the learning model 522, and provides feedback to the learning model 522 based on the result of the inference process and the error between the correct data. By repeating this process with a large amount of teacher data, a trained model 121 is generated. The generated trained model 121 is then transferred to the storage unit 120 of the controller 100.

[0054] 14 to 16 are diagrams for explaining the details of the learning stage described above. FIG. 14 is an explanatory diagram of the first trained model 122 used for estimating the holding time of the energy device 310 and the biological tissue in the first recognition unit 111. As shown in FIG. 14, the learning device 500 (not shown) corrects the existing first trained model 122 by feeding back the first teacher data 521A, which is labeled with annotations corresponding to the treatment image showing the energy device 310 and the biological tissue, to the first trained model 522A, and transmits the corrected data to the controller 100 as a new first trained model 122. The content of the annotation is information corresponding to the correct answer data such as the type and position of the energy device 310, the configuration and state of the tip part of the energy device 310, and the type of the biological tissue to be treated.

[0055] Fig. 15 is an explanatory diagram of the second trained model 123 used for estimating the type of biological tissue in the second recognition unit 112. As shown in Fig. 15, the learning device 500 (not shown) corrects the existing second trained model 123 by feeding back the second teacher data 521B, which is labeled with an annotation corresponding to the treatment image, to the second trained model 522B, and transmits it to the controller 100 as a new second trained model 123. The content of the annotation is information corresponding to correct data such as the position of the energy device 310 and the type of biological tissue to be treated.

[0056] Fig. 16 is an explanatory diagram of the third trained model 124 used for estimating residual heat information in the estimation unit 113. As shown in Fig. 16, the learning device 500 (not shown) corrects the existing third trained model 124 by feeding back third teacher data 521C, which is labeled with an annotation corresponding to a treatment image, to a third trained model 522C, and transmits the corrected third trained model 124 to the controller 100 as a new third trained model 124. Note that, in the learning stage, the estimation unit 113 may be trained to estimate the temperature during energy supply.

[0057] The correct answer data attached to the learning image includes the type of energy device 310, the residual heat temperature at the tip of the energy device 310, the type of biological tissue to be treated, the time for which the biological tissue is grasped, etc. The correct answer data attached to the learning image may be the residual heat temperature acquired by a thermo camera or a temperature sensor mounted on the tip of the treatment tool. For example, a color indicating a temperature is added to each part of the energy device 310 captured in an image captured by a thermo camera. Then, the image becomes a learning image of the third teacher data 521C, and the residual heat information indicated by a color in the image becomes the correct answer data. The correct answer data may be added by, for example, a doctor.

[0058] The correct answer label attached to the learning information, which is information related to the energy output setting, is, for example, information about the energy output amount, output time, etc., or the history thereof. The correct answer data attached to the learning information can be acquired from an apparatus such as the generator 300. Also, history information of the output energy may be acquired from an apparatus such as the generator 300. The history information is fed back to the third learning model 522C to generate the third trained model 124, thereby improving the estimation accuracy of the residual heat information. In this way, in the learning device 500, a large number of learning device tissue images or learning tissue images are fed back to the learning model 522 as teacher data 521, thereby improving the estimation accuracy of the control unit 110.

[0059] Now, when performing energy treatment in a surgical operation, it is important to protect the living tissue from the heat of the energy device and perform the operation efficiently while maintaining safety. When using an energy device in a surgical operation, it is often difficult for a non-expert to predict the residual heat state of the tip part of the energy device, such as the active blade, pad, and shaft. For example, when a treatment is continued continuously, the temperature of the energy device gradually becomes hot, but it is difficult to predict what the current temperature is. In particular, the ultrasonic device 340 in FIG. 6 described above rises to about 200 degrees Celsius in a short time, so it is difficult for a non-expert doctor to perform the operation efficiently while maintaining safety, and requires skill.

[0060] The medical device disclosed in Patent Document 1 displays the presence or absence of a high-temperature device in the treatment image when a high-temperature energy device is present in the treatment image after the use of an energy treatment tool, and displays a warning image superimposed on the energy device in the image. However, the medical device does not display a specific residual heat temperature in the treatment image, so it cannot respond to cases where the timing of cooling the energy device differs from doctor to doctor. In addition, using a thermo camera or the like to measure temperature has the problem of increasing the number of ports in laparoscopic surgery and increasing the thickness of the sensor-equipped device. For example, doctors cool the energy device by air cooling or by placing the tip of the energy device on wet gauze to cool it down, but it is difficult to adjust the temperature.

[0061] In this regard, according to the present embodiment, the control unit 110 estimates residual heat information such as temperature by machine learning using the transition of the residual heat temperature of the tip portion of the energy device 310 according to the time of current supply from the power source to the energy device 310, the amount of current supply, and the type and state of the tissue to be treated as training data, and presents it to the doctor. For example, machine learning is performed to estimate the current temperature of the energy device based on the energy given to a certain living tissue. In this way, the residual heat information of the tip portion of the energy device 310 can be estimated and displayed in real time. Therefore, the timing of cooling the energy device 310 can be determined regardless of the experience and ability of the doctor. Therefore, even a non-expert doctor can easily avoid thermal damage to the preserved tissue, and can perform surgery more safely and efficiently.

[0062] 5. Second embodiment In the second embodiment, the control unit 110 treats the transition of the heat quantity or temperature after the energy output by the energy device 310 is completed as estimated residual heat information, and causes the notification unit 230 to perform notification based on this. That is, in the second embodiment, after the energy supply by the energy device 310 is completed, estimation is performed by the above-mentioned estimation program as to how the heat quantity or temperature of the tip end of the energy device 310 will transition.

[0063] In the second embodiment, the learned model 121 learns the relationship between the learning images and learning information and the residual heat transition after energy output, and learns to estimate the transition of the amount of heat or temperature after the energy output is completed. That is, in the learning device 500, in order to create a third learned model 121C for estimating the temperature transition of the energy device 310 after the completion of energy output, the residual heat history information during energy supply is used as the third teacher data 521C to correct the third learned model 124 and create a new third learned model 124. Then, as described above, the estimation unit 113 estimates the residual heat information based on the third learned model 124. In this case, the estimation unit 113 estimates the transition of the amount of heat or temperature at the tip of the energy device 310 after the completion of energy output based on information such as the type of biological tissue, the gripping time of the biological tissue, the amount of energy output from the energy device 310, and the output time, which are the third teacher data 521C.

[0064] FIG. 17 is a diagram for explaining the estimation of the temperature transition at the tip portion of the energy device 310. The horizontal axis represents time (s), and the vertical axis represents temperature (° C). Also, each time has a relationship of t3 < t2 < t1 < t0, and the current time is t0. In this case, if the temperatures at times t3, t2, and t1 are acquired at time t0 and plotted as circles on the graph, it will be as shown in FIG. 17, for example. Then, an approximate curve showing the time dependence of the residual heat temperature can be obtained from these data. The approximate curve can be obtained, for example, by using the analysis function of a computer. Further, as shown by the broken line in FIG. 17, by extrapolating the approximate curve to the range of t > t0, the transition of the amount of heat or temperature after the energy output by the energy device 310 is completed can be estimated.

[0065] In this way, the doctor can obtain information not only on the current residual heat information but also on the future temperature transition. Therefore, the doctor can perform the operation smoothly.

[0066] Also, based on the transition of the temperature estimated above, the time until the temperature of the tip of the energy device 310 reaches a desired specified value can be estimated, and the estimated time can be notified by the notification unit 230. That is, when the temperature Tx that may cause thermal damage to the biological tissue is set as a threshold, for example, it is possible to present a time that serves as a guide for how much time must elapse from the current time t0 for the temperature to fall below the threshold Tx and treatment to be resumed. The criterion for determining whether or not thermal damage has occurred is, for example, whether or not protein denaturation, inactivation of intracellular enzymes, etc. have occurred. Note that such a temperature threshold Tx differs depending on the type of biological tissue to be treated, and is, for example, 50 degrees Celsius to 60 degrees Celsius. In addition, when the desired specified value is changed depending on the type of biological tissue to be treated, for example, in the case of biological tissues with a high risk of thermal damage such as the pancreas and duodenum, the specified value can be set low to avoid the occurrence of thermal damage to particularly important biological tissues. In the graph shown in Fig. 17, if the temperature that is the desired specified value is the threshold value Tx, the temperature of the tip of the energy device 310 reaches the threshold value Tx at the time tx when the curve of the estimated temperature transition intersects with T = Tx. Therefore, the control unit 110 obtains the time determined by tx - t0 and causes the notification unit 230 to notify the time, thereby providing an indication of how much time is left before the doctor can safely resume treatment. In this way, the doctor can specifically know the time required before treatment can be resumed.

[0067] 6. Third embodiment In the third embodiment, the control unit 110 causes the notification unit 230 to issue an alert when the heat quantity or temperature of the energy device 310 estimated as the estimated residual heat information is greater than a predetermined threshold. That is, in this embodiment, the control unit 110 estimates, for example, the temperature of the tip portion of the energy device 310, and issues an alert to the doctor when the estimated temperature is greater than a predetermined threshold. The predetermined threshold is, for example, a threshold Tx that is a temperature that may cause thermal damage to biological tissue as described in "5. Second embodiment". The threshold may also be determined as a value of the heat quantity of the tip portion of the energy device 310. Even if the temperature itself of the tip portion of the energy device 310 is low, if the heat quantity is large, there may be a high risk of causing thermal damage to biological tissue due to heat over a long period of time. Therefore, the doctor can select whether to specify the threshold value based on the heat quantity or the temperature depending on the situation.

[0068] The alert of the notification unit 230 may be, for example, a method of displaying a warning on the display unit 232 of the notification unit 230. Here, the display unit 232 is, for example, a display of a personal computer. That is, a warning is issued to the doctor by displaying on the display that the temperature of the tip portion of the energy device 310 exceeds a threshold value. When issuing a warning, for example, the energy device 310 displayed on the display unit 232 may be superimposed on the treatment image by adding a color or the like so that the area exceeding the threshold value can be identified. Furthermore, the control unit 110 may display an estimated result as a display mode on the display unit 232 when the heat quantity or temperature, which is the estimated residual heat information, is higher than a predetermined threshold value.

[0069] The warning display can be displayed by, for example, adding a telop to the display screen of the estimation result shown in the above-mentioned Figs. 9 to 11. Specifically, display examples such as those shown in Figs. 18 and 19 are conceivable. In the example shown in Fig. 18, a telop saying "Warning! Residual heat over 100°C" is displayed in the upper left of the image to warn the doctor that the temperature of the tip of the energy device 310 may cause thermal damage to living tissue. In the example shown in Fig. 19, a telop saying "Warning! Residual heat 100°C in 0 seconds" is displayed. The display method shown in Fig. 19 is different from the display method shown in Fig. 18, and displays, for example, the time required for the temperature to fall below the threshold temperature described above. Therefore, more useful information can be provided to the doctor in proceeding with the surgery, and the surgery can be carried out smoothly. The above-mentioned required time can be obtained, for example, by the estimation method of the temperature transition of the tip of the energy device 310 described in Fig. 17. In addition, in the display shown in Figs. 18 and 19, for example, the above-mentioned telop display can be displayed by blinking, etc., to more effectively call the doctor's attention. The warning is not limited to a display on the display unit 232, but may be, for example, an alarm sound or a vibration of the energy device 310.

[0070] FIG. 20 is a diagram for explaining the process performed by the controller 100 and the like in the third embodiment. In this embodiment, when the estimation unit 113 of the control unit 110 estimates that the temperature or heat quantity of the tip portion of the energy device 310 is greater than the threshold, the notification unit 230 issues an alert. On the other hand, when the estimation unit 113 estimates that the heat quantity or temperature of the tip portion of the energy device 310 is equal to or less than the threshold, the notification unit 230 does not issue an alert. That is, after the energy supply by the energy device 310 is completed, for example, after the energy device 310 is turned off, if the heat quantity or temperature of the tip portion of the energy device 310 exceeds the threshold, an alert is issued. Here, when a transition in the heat quantity or temperature is estimated, the notification of the alert is stopped after the heat quantity or temperature becomes equal to or less than the threshold. In the above process, for example, the memory unit 120 stores table data on residual heat information, and the control unit 110 refers to the table data to output the process of the notification unit 230 corresponding to the estimation result to the notification unit 230 via the I / O device 190. The notification unit 230 then operates according to the output result of the control unit 110. Note that the algorithm for issuing an alert is not limited to the above. According to the third embodiment, the doctor can grasp the residual heat temperature at the tip of the energy device 310, and can perform surgery more safely and efficiently.

[0071] The system of the present embodiment can also be realized as a program. That is, the program of the present embodiment acquires at least one treatment image including an image of at least one energy device 310 and at least one living tissue. Then, from a learning device tissue image, which is an image of at least one energy device 310 and at least one living tissue that receives energy supply and performs energy output, or a learning tissue image, which is an image of at least one living tissue, information on the energy supply amount, which is information on the energy output setting of the energy device 310, is acquired by processing based on the trained model 121 that has been trained to output residual heat information, which is information on the amount of heat or temperature remaining in the energy device 310 after energy is supplied to the energy device 310. Then, based on the treatment image, the information on the amount of energy supply, and the trained model 121, the computer is caused to estimate estimated residual heat information, which is information on the amount of heat or temperature of the energy device 310, and display the estimated residual heat information on the display unit 232. Here, the computer is assumed to be a network terminal such as a personal computer. However, the computer may be a wearable terminal such as a smartphone, a tablet terminal, or a smart watch. By doing so, the same effect as above can be obtained.

[0072] The system of this embodiment can also be realized as an information processing method. That is, the information processing method acquires at least one treatment image including an image of at least one energy device and at least one living tissue. Then, from a learning device tissue image, which is an image of at least one energy device 310 receiving energy supply and performing energy output, and at least one living tissue, or a learning tissue image, which is an image of at least one living tissue, information on the energy supply amount, which is information on the energy output setting of the energy device 310, is acquired by processing based on the trained model 121 that has been trained to output residual heat information, which is information on the amount of heat or temperature remaining in the energy device 310 after energy is supplied to the energy device 310. Then, based on the treatment image, the information on the amount of energy supply, and the trained model 121, estimated residual heat information, which is information on the amount of heat or temperature of the energy device 310, is estimated. By doing so, the same effect as above can be obtained.

[0073] The system 10 of the present embodiment described above includes a storage unit 120 that stores a trained model 121, and a control unit 110. The trained model 121 learns to output residual heat information, which is information about the amount of heat or temperature remaining in the energy device 310 after energy is supplied to the energy device 310, from a training device tissue image, a training tissue image, and training information. The training device tissue image is an image of at least one energy device 310 that receives energy supply and performs energy output, and at least one living tissue. The training tissue image is an image of at least one living tissue. The training information is information about the energy supply amount, which is information about the energy output setting of the energy device. The control unit 110 acquires at least one treatment image including an image of at least one energy device 310 and at least one living tissue. The control unit 110 acquires information about the energy supply amount, which is information about the energy output setting of the energy device 310. The control unit 110 estimates estimated residual heat information, which is information about the amount of heat or the temperature of the energy device 310, based on the treatment image, information about the amount of energy supply, and the trained model 121. The control unit 110 causes the notification unit to issue a notification based on the estimated residual heat information.

[0074] According to this embodiment, residual heat information such as temperature can be estimated and presented to a doctor by machine learning using the time of current supply from the power source to the energy device 310, the amount of current supply, and the transition of the residual heat temperature at the tip of the energy device 310 according to the type and state of the tissue to be treated as teacher data 521. In this way, residual heat information at the tip of the energy device 310 can be estimated and displayed in real time. Therefore, the timing of cooling the energy device 310 can be determined without depending on the experience and ability of the doctor. Therefore, even a non-expert doctor can avoid thermal damage to the preserved tissue and perform surgery more safely and efficiently. The learning device tissue image, learning tissue image, and learning information are described in "4. First embodiment".

[0075] In addition, in this embodiment, the control unit 110 may extract tissue information, which is information regarding the biological tissue imaged on the treatment image, from the treatment image, and estimate estimated residual heat information based on the tissue information and information regarding the amount of energy supply.

[0076] According to this embodiment, the control unit 110 can estimate the type of biological tissue from the treatment image acquired by the endoscope system 200. The treatment image is described in Fig. 8. Also, the learning stage for estimating biological tissue is described in "4. First embodiment".

[0077] In addition, in this embodiment, the control unit 110 may extract device information, which is information regarding the energy device imaged on the treatment image, from the treatment image, estimate the target tissue to be treated by the energy device based on the device information and tissue information, and estimate estimated residual heat information based on the target tissue to be treated.

[0078] According to this embodiment, the control unit 110 can estimate device information related to the energy device 310 from the treatment image acquired by the endoscope system 200, and can estimate residual heat information of the energy device 310 based on the device information and the estimated tissue information. The treatment image is described in FIG. 8. The learning stage for estimating biological tissue is described in "4. First embodiment."

[0079] In this embodiment, the control unit 110 may estimate, as the estimated remaining heat information, a change in the amount of heat or temperature after the energy output by the energy device 310 is completed.

[0080] According to this embodiment, the doctor can obtain not only the current residual heat information but also information on the future temperature change. The method of estimating the heat quantity or temperature change is described in FIG. 17 of "5. Second embodiment".

[0081] In addition, in this embodiment, the trained model 121 may be a trained model 121 that has learned the relationship between the training device tissue image or the training tissue image and training information and the residual heat transition after energy output, and has been trained to estimate the transition of heat quantity or temperature after energy output is completed.

[0082] According to this embodiment, the control unit 110 can estimate the transition of the heat quantity or temperature at the tip of the energy device 310 after the energy output is completed, using the trained model 121 that has trained the relationship between the training image and training information and the transition of the residual heat after the energy output. Details are described in "5. Second embodiment".

[0083] In this embodiment, the control unit 110 may estimate, as the estimated remaining heat information, the time required for the amount of heat or temperature of the energy device 310 to reach a desired specified value after the energy output by the energy device 310 is completed.

[0084] According to this embodiment, the doctor can set the desired amount of heat, etc., and can know specifically the time required for the energy device 310 to reach the desired amount of heat, etc. Therefore, the surgery can be performed smoothly. Details are described in "5. Second embodiment".

[0085] In this embodiment, the control unit 110 may cause the notification unit to issue an alert when the amount of heat or the temperature of the energy device 310 estimated as the estimated residual heat information is greater than a predetermined threshold value.

[0086] According to this embodiment, by setting a threshold value that is a temperature that may cause thermal damage to biological tissue, the doctor can grasp the risk of thermal damage to biological tissue due to residual heat at the tip of the energy device 310. Therefore, surgery can be performed more safely and efficiently. The method of issuing an alert is described in FIG. 20 in "6. Third embodiment."

[0087] In this embodiment, the notification unit 230 is a display unit 232, and the control unit 110 may cause the display unit 232 to display the estimated remaining heat information.

[0088] According to this embodiment, the amount of heat, the temperature, or the time until the amount of heat or the temperature of the energy device 310 reaches a desired specified value is displayed on the display unit 232, enabling the doctor to determine whether or not it is necessary to wait until the temperature of the energy device 310 drops. Details are described in Figures 9 to 12 in "4. First embodiment".

[0089] In addition, in this embodiment, when the heat quantity or temperature of the energy device 310 estimated as the estimated remaining heat information is higher than a predetermined threshold value, the control unit 110 may cause the display unit to display a display indicating the heat quantity or temperature of the energy device estimated as the estimated remaining heat information.

[0090] According to this embodiment, the doctor can know from the display of the display unit 232 whether or not there is a risk of thermal damage to biological tissue due to residual heat at the tip of the energy device 310. Therefore, the doctor can perform surgery more safely and efficiently. Details are described in "6. Third embodiment."

[0091] In this embodiment, the control unit 110 may superimpose a color indicating the amount of heat or temperature of the energy device 310 estimated as the estimated remaining heat information on the energy device 310 displayed on the display unit 232.

[0092] According to this embodiment, the energy device 310 may be a bipolar device 330. Details are described in Figs. 9 to 12 in "4. First embodiment".

[0093] The above processing may be written as a program. That is, the program of this embodiment causes the controller 100 to acquire a treatment image, acquire information on the amount of energy supply by processing based on the trained model 121, estimate estimated residual heat information based on the treatment image, the information on the amount of energy supply, and the trained model 121, and display the estimated residual heat information on the display unit 232.

[0094] The above processing may be described as a method. That is, the method of the present embodiment acquires a treatment image, acquires information on the amount of energy supply by processing based on the trained model 121, and estimates estimated residual heat information based on the treatment image, the information on the amount of energy supply, and the trained model 121.

[0095] Although the present embodiment and its modified examples have been described above, the present disclosure is not limited to each embodiment or its modified example as it is, and in the implementation stage, the components can be modified and embodied within the scope of the gist. In addition, a plurality of components disclosed in each of the above-mentioned embodiments and modified examples can be appropriately combined. For example, some components may be deleted from all the components described in each embodiment or modified example. Furthermore, the components described in different embodiments or modified examples may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the present disclosure. In addition, a term described at least once in the specification or drawings together with a different term having a broader meaning or the same meaning can be replaced with the different term in any part of the specification or drawings. [Explanation of symbols]

[0096] 10 system, 100 controller, 110 control unit, 111 first recognition unit, 112 second recognition unit, 113 estimation unit, 114 output information determination unit, 115 presentation unit, 120 memory unit, 121 trained model, 122 first trained model, 123 second trained model, 124 third trained model, 170 I / O device, 180 I / O device, 190 I / O device, 200 endoscope system, 210 endoscope, 220 main body device, 230 notification unit, 232 display unit, 300 generator, 310 energy device, 320 monopolar device, 321 electrode, 322 insertion unit, 323 operation unit, 325 cable, 330 bipolar device, 331 tip portion, 332 insertion unit, 333 operation unit, 335 cable, 336 Base end, 337 jaw, 338 jaw, 340 ultrasonic device, 341 tip, 342 insertion section, 343 operation section, 344a operation button, 344b operation button, 345 cable, 346 base end, 347 jaw, 348 probe, 500 learning device, 510 processing section, 520 memory section, 521 teacher data, 521A first teacher data, 521B second teacher data, 521C third teacher data, 522 learning model, 522A first learning model, 522B second learning model, 522C third learning model, S1 step, S2Aa step, S2Ab step, S2B step, S3 step, T temperature, Tx threshold, t0 current time, t1 time, t2 time, t3 time

Claims

1. a memory unit that stores a trained model that is input with a learning device tissue image, which is an image of at least one energy device that receives energy supply and outputs energy and at least one biological tissue, or a learning tissue image, which is an image of the learning device tissue image and at least one biological tissue, and learning information regarding the amount of energy supply, which is information regarding the energy output setting of the energy device, and that is trained to output residual heat information, which is information regarding the amount of heat or temperature remaining in the energy device after energy is supplied to the energy device; A control unit, The control unit is acquiring at least one treatment image including an image of the at least one energy device and the at least one biological tissue; obtaining information regarding an energy supply amount, the information being information regarding an energy output setting of the energy device; The treatment image and information regarding the amount of energy supply are input to the trained model, and the trained model is caused to output the residual heat information as estimated residual heat information; The system further comprises a notification unit that performs a notification based on the estimated residual heat information.

2. 2. The system of claim 1, The trained model is a second trained model that is trained to extract tissue information, which is information about biological tissue imaged on the input image, by inputting the learning device tissue image, or the learning device tissue image and the learning tissue image; A third trained model that receives the organization information and the information regarding the energy supply amount and is trained to output the residual heat information; Including, The control unit is inputting the treated image into the second trained model to cause the second trained model to extract the tissue information; A system comprising: inputting the organization information and information regarding the energy supply amount into the third trained model; and causing the third trained model to output the estimated residual heat information.

3. 3. The system according to claim 2, The trained model is a first trained model that is trained to receive the learning device tissue image, or the learning device tissue image and the learning tissue image, and to extract device information that is information about the energy device imaged on the input image; The third trained model is the device information, the tissue information, and information regarding the amount of energy supply are input, and the device is trained to output the residual heat information; The control unit is inputting the treated image into the first trained model to cause the first trained model to extract the device information; A system comprising: inputting the device information, the organization information, and information regarding the energy supply amount into the third trained model; and causing the third trained model to output the estimated residual heat information.

4. 2. The system of claim 1, The system is characterized in that the control unit estimates, as the estimated residual heat information, a change in heat quantity or temperature after energy output by the energy device is completed.

5. 5. The system according to claim 4, The system is characterized in that the trained model is a trained model that has learned the relationship between the training device tissue image or the training tissue image and the training information, and the residual heat change after energy output, and has been trained to estimate the change in heat quantity or temperature after energy output is completed.

6. 6. The system according to claim 5, The system is characterized in that the control unit estimates, as the estimated residual heat information, the time until the heat quantity or temperature of the energy device reaches a desired specified value after energy output by the energy device is completed.

7. 2. The system of claim 1, The system is characterized in that the control unit causes the notification unit to issue an alert when the heat quantity or temperature of the energy device estimated as the estimated residual heat information is higher than a predetermined threshold.

8. 2. The system of claim 1, The notification unit is a display unit, The system is characterized in that the control unit causes the display unit to display the estimated residual heat information.

9. 9. The system according to claim 8, The system is characterized in that the control unit causes the display unit to display a display indicating the heat quantity or temperature of the energy device estimated as the estimated remaining heat information when the heat quantity or temperature of the energy device estimated as the estimated remaining heat information is higher than a predetermined threshold.

10. 9. The system according to claim 8, The system is characterized in that the control unit superimposes a color indicating the heat quantity or temperature of the energy device estimated as the estimated residual heat information on the energy device displayed on the display unit.

11. acquiring at least one treatment image including an image of at least one energy device and at least one biological tissue; obtaining information regarding an energy supply amount, the information being information regarding an energy output setting of the energy device; a learning device tissue image which is an image of at least one energy device that receives energy supply and outputs energy and at least one biological tissue, or a learning tissue image which is an image of the learning device tissue image and at least one biological tissue, and learning information related to the amount of energy supply which is information related to the energy output setting of the energy device, are inputted into a trained model which has been trained to output residual heat information which is information related to the amount of heat or temperature remaining in the energy device after energy is supplied to the energy device, inputting the treatment image and information related to the amount of energy supply into the trained model, and causing the trained model to output the residual heat information as estimated residual heat information; causing a notification unit to perform a notification based on the estimated residual heat information; and A program characterized by causing a computer to execute the above.

12. The program according to claim 11, The trained model is a second trained model that is trained to extract tissue information, which is information about biological tissue imaged on the input image, by inputting the learning device tissue image, or the learning device tissue image and the learning tissue image; A third trained model that receives the organization information and the information regarding the energy supply amount and is trained to output the residual heat information; Including, inputting the treated image into the second trained model to cause the second trained model to extract the tissue information; Inputting the organization information and the information related to the energy supply amount into the third trained model, and causing the third trained model to output the estimated residual heat information; A program characterized by causing a computer to execute the above.

13. The program according to claim 11, The program is characterized in that the program estimates, as the estimated residual heat information, a change in heat quantity or temperature after energy output by the energy device is completed.

14. The program according to claim 13, The program is characterized in that the trained model is a trained model that learns the relationship between the training device tissue image or the training tissue image and the training information, and the residual heat change after energy output, and that is trained to estimate the change in heat quantity or temperature after energy output is completed.

15. The program according to claim 14, The program is characterized in that, as the estimated residual heat information, a time until the amount of heat or temperature of the energy device reaches a desired specified value after energy output by the energy device is completed.

16. acquiring at least one treatment image including an image of at least one energy device and at least one biological tissue; obtaining information regarding an energy supply amount, the information being information regarding an energy output setting of the energy device; a learning device tissue image which is an image of at least one energy device that receives energy supply and outputs energy and at least one biological tissue, or a learning tissue image which is an image of the learning device tissue image and at least one biological tissue, and learning information related to the amount of energy supply which is information related to the energy output setting of the energy device, are inputted into a trained model which has been trained to output residual heat information which is information related to the amount of heat or temperature remaining in the energy device after energy is supplied to the energy device, inputting the treatment image and information related to the amount of energy supply into the trained model, and causing the trained model to output the residual heat information as estimated residual heat information; causing a notification unit to perform a notification based on the estimated residual heat information; and 13. An information processing method comprising:

17. 17. The information processing method according to claim 16, The trained model is a second trained model that is trained to extract tissue information, which is information about biological tissue imaged on the input image, by inputting the learning device tissue image, or the learning device tissue image and the learning tissue image; A third trained model that receives the organization information and the information regarding the energy supply amount and is trained to output the residual heat information; Including, inputting the treated image into the second trained model to cause the second trained model to extract the tissue information; Inputting the organization information and the information related to the energy supply amount into the third trained model, and causing the third trained model to output the estimated residual heat information; 13. An information processing method comprising:

18. 17. The information processing method according to claim 16, An information processing method comprising estimating, as the estimated residual heat information, a change in heat quantity or temperature after energy output by the energy device is completed.

19. 19. The information processing method according to claim 18, The information processing method is characterized in that the trained model is a trained model that has learned the relationship between the training device tissue image or the training tissue image and the training information, and the residual heat change after energy output, and has been trained to estimate the change in heat quantity or temperature after energy output is completed.

20. 20. The information processing method according to claim 19, An information processing method characterized in that the estimated residual heat information includes estimating a time until the heat quantity or temperature of the energy device reaches a desired specified value after energy output by the energy device is completed.

Images