Medical system and method for detecting contact state

Abstract

To provide a medical system and the like that can detect a contact state between a treatment tool and a tissue from an endoscope image.SOLUTION: A medical system 1 includes a memory 120 which stores a trained model 122, and a processor 110. The trained model 122 is a model trained by training data including an endoscopic image and a contact state between a treatment tool and a tissue in the endoscopic image. The endoscopic image is captured by an endoscope which captures an image including the treatment tool and a tissue to be treated. The processor 110 acquires the endoscopic image including the treatment tool and the tissue. The processor 110 uses the trained model to detect, from the endoscopic image, the contact state between the treatment tool and the tissue as first information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a medical system, a contact state detection method, and the like. [Background technology]

[0002] Patent Document 1 discloses a medical instrument that detects contact between a treatment tool and an object using a tactile sensor during an endoscopic procedure. This medical instrument is used together with an endoscope and includes an insertion section that is inserted into a body cavity, and a tactile sensor provided in the insertion section that can detect contact of an object with the insertion section outside the observation field of the endoscope. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-61979 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, contact between the treatment tool and tissue is detected by a tactile sensor mounted on the treatment tool, which may result in increased device costs, an increased size of the treatment tool, or limitations on the device. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to a medical system that includes an endoscopic image captured by an endoscope that captures an image including a treatment tool and a tissue to be treated, and a trained model trained using teacher data including the treatment tool and the tissue in the endoscopic image, and a processor, wherein the processor acquires the endoscopic image including the treatment tool and the tissue, and uses the trained model to detect the contact state between the treatment tool and the tissue from the endoscopic image as first information.

[0006] Another aspect of the present disclosure relates to a contact condition detection method including: capturing an endoscopic image including a treatment tool and tissue to be treated with an endoscope; and detecting the contact condition between the treatment tool and the tissue from the endoscopic image as first information using a trained model trained with teacher data including the endoscopic image and the contact condition between the treatment tool and the tissue in the endoscopic image.

[0007] Yet another aspect of the present disclosure relates to a medical system including: a memory that stores a trained model trained from teacher data including an endoscopic image captured by an endoscope that captures an image including a treatment tool having opening and closing jaws and tissue to be treated, and the open / closed state of the jaw in the endoscopic image; and a processor, wherein the treatment tool is an energy device that treats the tissue by outputting energy from the jaw, and the processor acquires the endoscopic image including the treatment tool and the tissue, performs a first detection using the trained model to detect whether the jaws are open or closed from the endoscopic image, performs a second detection to detect the presence or absence of tissue between the jaws based on electrical information in the energy output, and detects a state in which the tissue is grasped by the jaws as a contact state based on a result of the first detection and a result of the second detection.

[0008] Yet another aspect of the present disclosure relates to a contact state detection method for detecting a contact state between a treatment tool, which is an energy device having jaws that open and close and treats tissue by outputting energy from the jaws, and the tissue to be treated, the contact state detection method including: capturing an endoscopic image including the treatment tool and the tissue with an endoscope; performing a first detection to detect opening and closing of the jaws from the endoscopic image using a trained model trained from teacher data including the endoscopic image and the open and closed states of the jaws in the endoscopic image; performing a second detection to detect the presence or absence of tissue between the jaws based on electrical information in the energy output; and detecting a gripping state of the tissue by the jaws as the contact state based on a result of the first detection and a result of the second detection. [Brief description of the drawings]

[0009] [Figure 1] An example of the support the medical system provides to surgeons. [Diagram 2] An example of a medical system configuration. [Diagram 3] An example of the processing flow of a medical system. [Figure 4] 13 shows a first configuration example of a treatment tool detection unit and a contact detection unit. [Diagram 5] 2 shows a second configuration example of the contact detection unit. [Figure 6] FIG. 4 is a diagram for explaining movement vectors of a treatment tool and tissue. [Figure 7] 13 is a diagram for explaining the "distance difference between the treatment tool and the tissue in the moving direction" acquired by the preprocessing unit. [Figure 8] 13 is a diagram for explaining the "distance difference between the treatment tool and the tissue in the moving direction" acquired by the preprocessing unit. [Figure 9] 13A and 13B are diagrams for explaining the "directional difference between the moving directions of the treatment tool and the tissue" acquired by the preprocessing unit. [Figure 10] 13 is a diagram for explaining "statistics of tissue movement amount distribution subjected to basis transformation based on the treatment tool movement direction" acquired by the preprocessing unit. [Figure 11] 13 is a diagram for explaining "statistics of tissue movement amount distribution subjected to basis transformation based on the treatment tool movement direction" acquired by the preprocessing unit. [Figure 12] 4 is a diagram for explaining “color information” acquired by a preprocessing unit. [Figure 13] First example of setting the area of ​​interest. [Figure 14] Second example of setting the area of ​​interest. [Figure 15] Third example of setting the area of ​​interest. [Figure 16] Third example of setting the area of ​​interest. [Figure 17] Fourth example of setting the area of ​​interest. [Figure 18] Fourth example of setting the area of ​​interest. [Figure 19] FIG. 13 is a diagram for explaining detection of tissue breaks. [Figure 20] FIG. 13 is a diagram for explaining detection of tissue breaks. [Figure 21] 13 shows a first configuration example of a contact detection unit in the second embodiment. [Figure 22] FIG. 13 is a diagram for explaining a second configuration example of the jaw open / close detection unit. [Diagram 23] 13 is a second configuration example of the inter-joint structure detection unit. [Figure 24] 13 shows a third example of the inter-joint structure detection unit. [Diagram 25] 4 shows a fourth example of the configuration of the inter-joint structure detection unit. [Figure 26] 13 shows a second configuration example of the contact detection unit in the second embodiment. [Figure 27] 13 shows an example of contact detection using detection results stored in a memory. [Figure 28] 13 shows a third configuration example of the contact detection unit in the second embodiment. [Figure 29] 13 shows a first configuration example of a contact detection unit in the third embodiment. [Diagram 30] 13 shows a first example of a flow of processing performed by a contact detection unit in the third embodiment. [Diagram 31] 13 shows a second example flow of a process performed by the contact detection unit in the third embodiment. [Diagram 32] 13 shows a second configuration example of the contact detection unit in the third embodiment. [Diagram 33] 13 shows an example of the configuration of a contact detection unit in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In the following disclosure, many different embodiments and examples are provided for implementing different features of the presented subject matter. Of course, these are merely examples and are not intended to be limiting. Furthermore, the disclosure may repeat reference numbers and / or letters in various examples. Such repetition is for the sake of brevity and clarity and does not, per se, require a relationship between the various embodiments and / or configurations described. Furthermore, when a first element is described as being "connected" or "coupled" to a second element, such description includes embodiments in which the first element and the second element are directly connected or coupled to each other, as well as embodiments in which the first element and the second element are indirectly connected or coupled to each other with one or more other intervening elements therebetween.

[0011] 1.Method In a procedure using an endoscope and a treatment tool, it is desirable for a medical system to detect the contact state between the treatment tool and tissue and provide information to the surgeon based on the detection result. There are various situations in which such contact detection is necessary, but here we will explain an example of providing support information to the surgeon in a treatment using an energy device.

[0012] 1 shows an example of support provided to an operator by a medical system. The medical system displays an endoscopic image 232 on a monitor 231. The medical system acquires information on a tissue to be treated from the endoscopic image 232, determines an appropriate energy output according to the information, and presents the output setting 233 to the operator by displaying it on the monitor 231. The operator accepts or rejects the output setting 233 recommended by the medical system. When the output setting 233 is accepted, the medical system outputs energy from the energy device at the output setting 233.

[0013] By performing such automatic adjustment of the energy output, the safety of the treatment can be improved. However, it is troublesome for the surgeon to always be presented with the recommended output setting 233. Therefore, it is desirable to present the information when the surgeon intends to perform treatment. For example, it is desirable to present the information when the treatment tool comes into contact with the tissue, or when the forceps-type surgical treatment tool grasps the tissue. In order to present the information at the appropriate time, it is necessary to accurately recognize the contact state between the treatment tool and the tissue.

[0014] Therefore, the medical system of the present embodiment detects the treatment tool from the endoscopic image and recognizes the contact state between the treatment tool and the tissue based on the detection result. Alternatively, the medical system improves the accuracy of contact detection by combining multiple machine learning models, or by combining a machine learning model and a non-machine learning means. As a result, support information such as information on the contacted tissue is presented to the surgeon only when the treatment tool contacts the tissue, so that the presentation of information at a timing not necessary for the surgeon is suppressed, and the inconvenience is reduced.

[0015] In the above-mentioned Japanese Patent Publication No. 2003-61979, a tactile sensor is mounted on the treatment tool, but the medical system of this embodiment can detect the contact state between the treatment tool and tissue by recognition processing using images. This makes it possible to recognize the contact state without increasing the size of the treatment tool while keeping device costs low. Furthermore, while the above-mentioned Japanese Patent Publication No. 2003-61979 is limited to devices equipped with a tactile sensor, the method of this embodiment does not limit the device. In other words, the method of this embodiment can be applied to various existing treatment tools.

[0016] An example of the contact state detected in this embodiment is given below. The contact state is that the treatment tool is in contact with the tissue, or that the treatment tool is not in contact with the tissue. The contact state may be that the treatment tool is gripping the tissue, or that the treatment tool is not gripping the tissue. When the treatment tool grips the tissue, it is considered that the treatment tool is in contact with the tissue. The contact state may be a transition from contact to non-contact, a transition from non-contact to contact, a transition from gripping to non-grasping, or a transition from non-grasping to gripping. For example, the tissue splitting described later corresponds to a transition from contact to non-contact, or a transition from gripping to non-grasping. The contact state may be that contact is maintained, that non-contact is maintained, that gripping is maintained, or that non-grasping is maintained. The contact state may be a combination of one or more of the above states. The treatment tool includes a shaft and an end effector provided at the tip of the shaft for treating the tissue. At this time, the contact state detected is a contact state between the end effector and the tissue. However, in cases where it is desirable for the shaft not to come into contact with the tissue, for example, the contact state between the shaft and the tissue may be detected.

[0017] The period or timing for detecting the contact state may be arbitrary. For example, when the treatment tool is an energy device, the contact state may be detected when energy is not being output, when energy is being output, or when energy is being output and when energy is not being output.

[0018] 2. First embodiment 2 shows an example of the configuration of a medical system. The medical system 1 includes an endoscope system 200, a controller 100, and an instrument or system 230. Hereinafter, an example will be described in which the medical system 1 is a system using a rigid endoscope for surgical operations. The treatment tool is basically inserted into the abdominal cavity or the like as a device separate from the rigid endoscope. However, the detection method of this embodiment can also be applied to a system using a flexible endoscope for the digestive tract. In that case, the treatment tool passes through a treatment tool channel of the flexible endoscope and protrudes from the tip of the flexible endoscope.

[0019] The endoscope system 200 is a system for capturing images inside a body cavity. The endoscope system 200 includes an endoscope and a main body device.

[0020] An endoscope is a rigid scope that is inserted into a body cavity to capture images of the interior of the body cavity. The endoscope includes an insertion section that is inserted into the body cavity, an operation section that is connected to a base end of the insertion section, a universal cord that is connected to the base end of the operation section, and a connector section that is connected to the base end of the universal cord. An imaging device for capturing images of the interior of the body cavity and an illumination optical system for illuminating the interior of the body cavity are provided at the tip of the insertion section. The imaging device includes an objective optical system and an imaging element that captures an image of a subject formed by the objective optical system. The connector section detachably connects the transmission cable to the main device. An image captured by an endoscope is referred to as a captured image or an endoscopic image.

[0021] The main unit includes a processing device that controls the endoscope, processes the endoscopic image, and displays the endoscopic image, and a light source device that generates illumination light and controls the illumination light. The processing device is composed of a processor such as a CPU, processes the image signal transmitted from the endoscope to generate an endoscopic image, and outputs the endoscopic image to the display and the controller 100. The illumination light emitted by the light source device is guided by a light guide to the illumination optical system of the endoscope, and is emitted from the illumination optical system into the body cavity.

[0022] The device or system 230 is a device or system that operates upon receiving the output of the controller 100. The device or system 230 is, for example, a display that displays a contact state or support information based on the contact state. This display may be provided separately from the display of the endoscope system 200, or may be the display of the endoscope system 200. Alternatively, the device or system 230 may be an energy device and a generator that drives the energy device. The energy device is a device that outputs energy from its tip using high-frequency power or ultrasound, etc., to perform treatment such as coagulation, sealing, hemostasis, incision, excision, or peeling on tissue that the tip contacts. The energy device is a monopolar device that outputs electrical energy from a single end effector, a bipolar device that applies electrical energy between jaws, an ultrasonic device that outputs ultrasonic energy, or a combination device that uses ultrasonic energy and electrical energy in combination. The generator controls the energy output based on the contact state or support information based on the contact state from the controller 100.

[0023] The medical system 1 may further include a treatment tool that is a non-energy device such as forceps or a spatula. The treatment tool that is the target of contact detection may be either an energy device or a non-energy device.

[0024] The controller 100 includes a processor 110, a memory 120, an I / O device 180, and an I / O device 190. Here, an example is shown in which the controller 100 is a device separate from the main body device of the endoscope system 200. The controller 100 may be an information processing device such as a personal computer, or may be a cloud system made up of multiple information processing devices connected via a network. However, the function of the controller 100 may be built into the main body device of the endoscope system 200.

[0025] The I / O device 180 receives an endoscopic image from the endoscope system 200. The I / O device 180 is a cable connector for connecting to a main body device of the endoscope system 200, or a communication circuit for performing communication processing with the main body device.

[0026] The I / O device 190 outputs the contact state information or support information based on the contact state output by the processor 110 to the device or system 230. The I / O device 190 is a cable connector for connecting to the device or system 230, or a communication circuit for performing communication processing with the device or system 230.

[0027] The processor 110 includes hardware. The processor 110 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microcomputer, or a digital signal processor (DSP). Alternatively, the processor 110 may be an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor 110 may be composed of one or more of a CPU, a GPU, a microcomputer, a DSP, an ASIC, and an FPGA. The memory 120 is, for example, a semiconductor memory that is a volatile memory or a non-volatile memory. Alternatively, the memory 120 may be a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device.

[0028] The memory 120 stores a program 121 in which the processing contents of the treatment tool detection unit 130 and the contact detection unit 140 are described. The processor 110 executes the program 121 to execute the processing of the treatment tool detection unit 130 and the contact detection unit 140. For example, the program 121 includes program modules in which the processing of each unit is described, and the processor 110 executes the program modules to execute the processing of each unit.

[0029] The program 121 includes a trained model 122 obtained by machine learning. The trained model is, for example, a neural network trained by deep learning. In this case, the trained model includes a program in which an algorithm of the neural network is described, and weight parameters between nodes of the neural network. The neural network includes an input layer to which input data is input, an intermediate layer that performs arithmetic processing on the data input through the input layer, and an output layer that outputs an inference result based on the arithmetic result output from the intermediate layer. In the learning stage, a learning system configured by an information processing device or a cloud system executes the machine learning process. The learning system includes a processor and a memory that stores the model and the teacher data. The processor generates the trained model 122 by training the model using the teacher data. This trained model 122 is stored in the memory 120 of the controller 100. Note that the controller 100 may also serve as the learning system.

[0030] The program 121 may be stored in a non-transitory information storage medium that is a computer-readable medium. The information storage medium is, for example, an optical disk, a memory card, a hard disk drive, or a semiconductor memory. The semiconductor memory is, for example, a ROM or a non-volatile memory. The processor 110 loads the program 121 stored in the information storage medium into the memory 120, and performs various processes based on the program 121.

[0031] 3 is an example of a flow of processing performed by the medical system. In step S1, an operator operates a treatment tool. The endoscope system 200 captures an endoscopic image showing the treatment tool and a treatment target. The treatment target is a tissue to be treated by the treatment tool. The processor 110 acquires the endoscopic image from the endoscope system 200.

[0032] In step S2, the treatment tool detection unit 130 detects the treatment tool from the endoscopic image. An example of the detection method is detection or segmentation by AI.

[0033] In step S3, the contact detection unit 140 detects the contact state between the treatment tool and the tissue detected in step S2 from the endoscopic image. The image used for detection may be the entire endoscopic image or a part of the endoscopic image. For example, the contact detection unit 140 may set a region of interest including the treatment tool and the surrounding tissue in the endoscopic image based on the detection result of detection or segmentation, and detect the contact state from the image within the region of interest.

[0034] Fig. 4 shows a first configuration example of the treatment tool detection unit and the contact detection unit. Fig. 4 shows the learning stage processing and the inference stage processing. As described above, the learning stage processing is executed by the learning system.

[0035] First, the inference stage will be described. The trained model 122 includes a trained model 122a for treatment tool detection and a trained model 122b for contact detection. The endoscope system 200 captures an endoscopic image in which the treatment tool 10 and the tissue 50 are shown.

[0036] The treatment tool detection unit 130 inputs an endoscopic image to the trained model 122a. The trained model 122a detects the position or area of ​​the treatment tool 10 from the endoscopic image and outputs the detection result 130Q. The detection result 130Q is, for example, a bounding box detected by detection or an area detected by segmentation.

[0037] The contact detection unit 140 inputs the endoscopic image and the detection result 130Q to the trained model 122b, or inputs an image of an attention area of ​​the endoscopic image that is set based on the detection result 130Q to the trained model 122b. The trained model 122b detects the contact state between the treatment tool 10 and the tissue 50 from the input data, and outputs the detection result 140Q.

[0038] The processor 110 performs output using the detection result 140Q. The processor 110 may output the contact state between the treatment tool 10 and the tissue 50 obtained as the detection result 140Q, or may perform information output using the contact state. An example of information output using the contact state will be given below.

[0039] In the first example, the processor 110 uses the contact state detection result 140Q as a trigger for information recognition or information presentation. When the contact between the treatment tool 10 and the tissue 50 is detected, the processor 110 determines the tissue type or tissue state of the contact portion and presents the information to the surgeon by displaying on a monitor or the like. This allows support information to be provided to a non-expert surgeon or the like at an appropriate timing. Alternatively, when the contact between the treatment tool 10 and the tissue 50 is detected, the processor 110 determines the output setting of the energy device according to the tissue type or tissue state of the contact portion and recommends the output setting to the surgeon by displaying on a monitor or the like. This allows support information to be provided at an appropriate timing rather than being displayed all the time. As described in FIG. 1, when there is an approval input from the surgeon, the processor 110 may instruct the generator to output at the recommended setting, and the generator may output from the energy device at the recommended setting.

[0040] In a second example, the processor 110 controls the energy device using the contact state detection result 140Q. When the processor 110 detects contact between the treatment tool 10 and the tissue 50, the processor 110 may transmit an instruction to release the safety lock of the energy device to the generator. That is, since the safety lock state is maintained while non-contact between the treatment tool 10 and the tissue 50 is detected, energy output is not performed even if an output operation of the energy device is performed.

[0041] Next, the learning stage will be described. The learning system generates a trained model 122a by training a model to detect the position or area of ​​a treatment tool from an endoscopic image IMG for training. The endoscopic image IMG for training may be an image captured by an endoscopic system different from the endoscopic system 200 used in the inference stage. The teacher data is a plurality of endoscopic images IMG and annotations indicating the position or area of ​​a treatment tool in each image. The annotation is a bounding box indicating the position of the end effector of the treatment tool in detection, or an area occupied by the end effector of the treatment tool in segmentation.

[0042] The learning system generates a trained model 122b by training the model to detect the contact state between the treatment tool and the tissue from the input data. The input data is an endoscopic image IMG and an annotation indicating the position or area of ​​the treatment tool. Alternatively, the input data is an image of an attention area set based on the annotation in the endoscopic image IMG. The teacher data is a plurality of endoscopic images IMG, annotations indicating the position or area of ​​the treatment tool in each image, and labels indicating the contact state between the treatment tool and the tissue in each image. The labels indicating the contact state here are mainly assumed to be contact, non-contact, and grasping or non-grasping, but labels of various contact states as described above may be used.

[0043] 5 is a second configuration example of the contact detection unit. The contact detection unit 140 performs preprocessing by a preprocessing unit 141 and contact detection using the output of the preprocessing and the trained model 122b. The program that realizes the preprocessing unit 141 may be a rule-based program or a trained model trained by machine learning.

[0044] The block of the preprocessing unit 141 shows information acquired in the preprocessing. The information is hierarchized according to the nature of the information, and the information acquired in the preprocessing is "distance difference between the movement direction of the treatment tool and the tissue", "directional difference between the movement vector of the treatment tool and the tissue", ..., "detection of opening and closing of the jaws, detection of tissue between the jaws". The preprocessing unit 141 acquires one or more pieces of information from the information and inputs them to the trained model 122b. The trained model 122b detects the contact state between the treatment tool and the tissue from the input one or more pieces of information. Among the information described in the block of the preprocessing unit 141, the information shown in the lower layers of "other interpolation information" and "technique for improving the accuracy of contact detection" is used in combination with the information shown in the lower layers of "movement vector" and "feature amount of grip detection". In the learning stage of the trained model 122b, the teacher data is one or more pieces of information that are input data and a label of the contact state corresponding to the input data.

[0045] Details of the information acquired in the pre-processing will be described below, although "detection of opening and closing of the jaws, and detection of tissue between the jaws" will be described later in the second embodiment.

[0046] FIG. 6 is a diagram for explaining movement vectors of a treatment tool and tissue.

[0047] The preprocessing unit 141 acquires a motion vector from an endoscopic image using a technique of optical flow or image registration. A motion vector at each pixel may be acquired, a motion vector at each point of a lattice may be acquired, a motion vector at a feature point of an image may be acquired, or a representative motion vector of a treatment tool and a representative motion vector of tissue may be acquired. The representative motion vector of a treatment tool is, for example, an average motion vector in a treatment tool region detected by segmentation or the like. The representative motion vector of tissue is, for example, an average motion vector in a peripheral region of a treatment tool region detected by segmentation. The average motion vector is not limited to a simple average of motion vectors, but may be calculated by a weighted average. Taking the average motion vector of a treatment tool as an example, the reliability of each motion vector of the motion vector distribution in the treatment tool region is calculated, and the average motion vector is calculated by weighting the motion vectors with weighting based on the reliability. The same applies to the average motion vector of tissue.

[0048] As shown in FIG. 6, it is assumed that the movement vector Vtool of the treatment tool 10 and the movement vectors Vtissue1 and Vtissue2 of the tissue 50 are acquired. In the endoscopic image IMGa in non-contact, the surrounding tissue does not move even if the treatment tool moves, so the correlation between the movement of the treatment tool and the surrounding tissue is small. That is, in non-contact, the correlation between the movement vector Vtool of the treatment tool 10 and the movement vectors Vtissue1 and Vtissue2 of the tissue 50 is small. On the other hand, in the endoscopic image IMGb in contact, when the treatment tool moves, the surrounding tissue moves in the same way, so the correlation between the movement of the treatment tool and the surrounding tissue is large. That is, in contact, the correlation between the movement vector Vtool of the treatment tool 10 and the movement vectors Vtissue1 and Vtissue2 of the tissue 50 is large. From these facts, contact detection is possible by using the movement vector.

[0049] 7 and 8 are diagrams for explaining the "distance difference between the treatment tool and the tissue in the moving direction" acquired by the preprocessing unit.

[0050] 7, the preprocessing unit 141 calculates the average movement vector v of the treatment tool and the average movement vector w of the tissue surrounding the treatment tool, and calculates the Euclidean distance |wv| between these average movement vectors. The smaller the Euclidean distance |wv|, the greater the similarity between the movement vectors v and w, and therefore it is considered that the treatment tool and the tissue are in contact. That is, the trained model 122b increases the probability of the determination result that the treatment tool and the tissue are in contact with each other as the Euclidean distance |wv| is smaller.

[0051] 8, the preprocessing unit 141 acquires the transition of the Euclidean distance |wv| in time series and inputs it to the trained model 122b. The trained model 122b judges that there is no contact when the Euclidean distance |wv| is greater than a threshold, and judges that there is contact when the Euclidean distance |wv| is smaller than the threshold. However, since the trained model 122b makes the judgment based on a network obtained by training, the result is not a simple threshold judgment.

[0052] FIG. 9 is a diagram for explaining the "directional difference between the moving directions of the treatment tool and the tissue" acquired by the preprocessing unit.

[0053] The preprocessing unit 141 calculates a directional difference between the average movement vector v of the treatment tool and the average movement vector w of the tissue surrounding the treatment tool. An example of the directional difference is cosine similarity. If the angle between the average movement vector v of the treatment tool and the average movement vector w of the tissue surrounding the treatment tool is θ, the cosine similarity is cosθ. The closer the cosine similarity cosθ is to 1, the greater the similarity between the movement vectors v and w, and therefore it is considered that the treatment tool and the tissue are in contact. That is, the closer the cosine similarity cosθ is to 1, the higher the probability of the trained model 122b determining that the treatment tool and the tissue are in contact.

[0054] The "movement vector of the treatment tool and tissue" acquired by the preprocessing unit will now be described.

[0055] The trained model 122b uses the movement vector of the treatment tool and the movement vector of the surrounding tissue as features for contact detection. The movement vector of the treatment tool and the movement vector of the surrounding tissue can themselves be useful features for contact detection.

[0056] For example, when the absolute value of the movement vector is small, the influence of noise is considered to be large, and the reliability of contact recognition is considered to be low. For example, when the treatment tool or the treatment tool is actually hardly moving, the influence of hand shaking is relatively large, and it is considered difficult to appropriately determine the contact state from the above-mentioned Euclidean distance or cosine similarity. For this reason, when the absolute value of the movement vector of the treatment tool or the absolute value of the movement vector of the surrounding tissue is small, the trained model 122b determines that the reliability of contact recognition is low. For example, when the above absolute value is small, the trained model 122b may not output the recognition result of the contact state, or may output a result of "unrecognizable".

[0057] Alternatively, when the treatment tool is in contact with tissue, particularly when the treatment tool is gripping tissue, the amount of movement of the treatment tool is considered to be smaller than when the treatment tool is free to move. This is because it is considered that camera shake is reduced or positioning is performed. For example, when the absolute value of the movement vector of the treatment tool is small, the trained model 122b may determine the contact state accordingly. As an example, when the absolute value of the movement vector of the treatment tool is small, the trained model 122b may maintain the recognition result of the contact state obtained previously.

[0058] 10 and 11 are diagrams for explaining "statistics of tissue movement amount distribution subjected to basis transformation based on the treatment tool movement direction" acquired by the preprocessing unit.

[0059] 10, the pre-processing unit 141 calculates the movement direction MDtool of the treatment tool 10 and the movement vector Vtissue of the tissue 50 surrounding the treatment tool. The pre-processing unit 141 resolves the movement vector Vtissue of the surrounding tissue 50 into a component Vp parallel to the movement direction MDtool of the treatment tool 10 and a component Vv perpendicular to the movement direction MDtool of the treatment tool 10. The movement direction MDtool of the treatment tool 10 is the longitudinal axis direction of the treatment tool 10 or the direction of the movement vector of the treatment tool 10. The trained model 122b judges the contact state using the components Vp and Vv.

[0060] FIG. 11 shows histograms of the components Vp and Vv acquired at multiple positions of the surrounding tissue in a certain frame. In the non-contact state, the histograms of the parallel component Vp and the perpendicular component Vv are both approximately symmetrical around "0". On the other hand, in the contact state, the histogram of the perpendicular component Vv is approximately symmetrical around "0", but the histogram of the parallel component Vp is clearly asymmetrical around "0". From this, the contact state can be determined using the components Vp and Vv. For example, the pre-processing unit 141 calculates the kurtosis and skewness of each histogram as statistics, and inputs the kurtosis and skewness to the trained model 122b. The kurtosis is an index indicating the degree of peaking of the histogram. The skewness is an index indicating the degree of asymmetry of the histogram.

[0061] 12 is a diagram for explaining the "color information" acquired by the pre-processing unit. In this example, the treatment tool is an energy device.

[0062] The trained model 122b uses color information of the tissue surrounding the treatment tool as a feature for contact detection. By using the color information, changes such as white scorching or mist of the tissue that occur during energy treatment are captured, and the information is used to complement the judgment of the contact state. The occurrence of white scorching or mist means that energy treatment is in progress, that is, it can be judged that there is "contact."

[0063] As shown in the left diagram of FIG. 12, it is assumed that a region of interest ROIc is set at the tip of a treatment tool in an endoscopic image IMGc before whitening. In the region of interest ROIc, the color of the tissue is strong in red components. In other words, the red component is sufficiently larger than the blue component, and the red component is sufficiently larger than the green component. As shown in the right diagram of FIG. 12, it is assumed that a region of interest ROId is set at the tip of a treatment tool in an endoscopic image IMGd after whitening. The hatched area indicates a whitening portion. In the region of interest ROId, the tissue has turned white, so the difference between each color component is almost eliminated. In other words, the red component is almost the same as the blue component and the green component. In this way, whitening can be judged by the color components. For example, the preprocessing unit 141 obtains the O1 channel of the opponent color space shown in the following formula (1) and inputs it to the trained model 122b.

[0064]

number

[0065] The "ROI setting" acquired by the preprocessing unit will now be explained. ROI is an abbreviation for Region Of Interest.

[0066] The preprocessing unit 141 sets a region of interest in an endoscopic image, calculates features from an image in the region of interest, and inputs the features to the trained model 122b. The region of interest includes a portion of the treatment tool where contact is to be detected and the surrounding tissue. The trained model 122b detects the contact state between the treatment tool and the tissue from the input features. The features are features using the above-mentioned movement vector, color information, or features for grip detection described below. The movement vectors and the like of the treatment tool and the surrounding tissue are basically related to each other around the contact point. Therefore, by setting a region of interest, the detection accuracy of the contact state can be improved.

[0067] FIG. 13 is a first example of setting a region of interest. The treatment tool 10 has a shaft 11 and an end effector 14. The preprocessing unit 141 sets a region of interest ROIe around the end effector 14. FIG. 13 shows an example in which the region of interest ROIe includes the entire end effector 14 and its surrounding area. For example, the end effector 14 is detected from the endoscopic image IMGe by detection processing using machine learning. The preprocessing unit 141 may perform the detection processing, or when the treatment tool detection unit 130 detects the treatment tool by detection, the preprocessing unit 141 may use the detection result of the treatment tool detection unit 130. The preprocessing unit 141 uses the detected region as it is as the region of interest ROIe, or expands or reduces the detected region and uses it as the region of interest ROIe.

[0068] FIG. 14 is a second example of setting the attention area. As shown in the left figure, the preprocessing unit 141 sets the attention area ROIf around the tip of the end effector 14. For example, the area of ​​the treatment tool 10 is detected from the endoscopic image IMGf by segmentation processing using machine learning. The preprocessing unit 141 may perform the segmentation processing, or when the treatment tool detection unit 130 detects the treatment tool by segmentation, the preprocessing unit 141 may use the detection result of the treatment tool detection unit 130. The preprocessing unit 141 detects the tip of the end effector 14 from the area of ​​the detected treatment tool 10, and sets the attention area ROIf based on the tip. An example of tip detection is shown in the right figure. The treatment tool mask MKtool is the area of ​​the treatment tool 10 detected by segmentation. The preprocessing unit 141 obtains the midpoint PC of the intersection line between the edge of the image and the treatment tool mask MKtool, obtains the point PF that is the furthest from the midpoint PC in the treatment tool mask MKtool, and determines that point PF to be the tip of the end effector 14.

[0069] 15 and 16 show a third example of setting the region of interest.

[0070] FIG. 15 shows example images with different optimal positions of the attention region. Endoscopic image IMGg is an example image when tissue is grasped long, and endoscopic image IMGh is an example image when tissue is grasped short. The part of the tissue that has a high correlation with the movement of the treatment tool is the part grasped by the jaw, which is the end effector. Therefore, when tissue is grasped long, the tissue is grasped by more than half of the jaw, so the surrounding area RAg is appropriate as the attention region. On the other hand, when tissue is grasped short, the tissue is grasped only near the tip of the jaw, so the area RAh near the tip of the jaw is appropriate as the attention region.

[0071] As shown in FIG. 16, the preprocessing unit 141 generates a plurality of attention region candidates CR1 to CR9 around the tip of the treatment tool 10. For example, the preprocessing unit 141 sets a region of a predetermined size based on the tip of the treatment tool 10, and generates a plurality of attention region candidates CR1 to CR9 by dividing the region. The number of candidates is not limited to nine. The preprocessing unit 141 sets one or more of the plurality of attention region candidates CR1 to CR9 as the attention region ROIp. FIG. 16 shows an example in which the candidate CR2 is set as the attention region ROIp. For example, the preprocessing unit 141 selects, as the attention region ROIp, a attention region candidate having a movement vector close to the movement vector of the treatment tool. Although an example in FIG. 15 has been described in which the end effector of the treatment tool is a jaw, the method in FIG. 16 can also be applied to a case in which the end effector is not a jaw.

[0072] 17 and 18 show a fourth example of setting a region of interest, in which the end effector of the treatment tool is a jaw.

[0073] Endoscopic image IMGp is ​​an example of an image when tissue is grasped short. The part of the jaw that grasps the tissue is indicated by HLp. If a region of interest that includes the entire jaw is set, most of the tissue included in the region of interest is not grasped, so the correlation between the movement of the jaw and the surrounding tissue is small. Endoscopic image IMGq is an example of an image when tissue is grasped long. The part of the jaw that grasps the tissue is indicated by HLq. If a region of interest that includes only the tip of the jaw is set, most of the area where the correlation between the movement of the jaw and the surrounding tissue is high will not be included in the region of interest.

[0074] Therefore, the preprocessing unit 141 detects the center of the part of the jaw that grasps the tissue, and sets a region of interest based on the center. That is, the preprocessing unit 141 detects the center of the part of the jaw that grasps the tissue HLp, HLq from the endoscopic images IMGp, IMGq, and sets the regions of interest ROIp, ROIq based on the center. This sets a region of interest that includes an area where the movement of the jaw and the surrounding tissue is highly correlated, thereby improving the accuracy of contact detection.

[0075] FIG. 18 shows an example of a method for detecting the center of the part of the jaw that grasps the tissue. The region of the treatment tool 10 is detected from the endoscopic image IMGf by segmentation processing. The pre-processing unit 141 may perform the segmentation processing, or if the treatment tool detection unit 130 detects the treatment tool by segmentation, the pre-processing unit 141 may use the detection result of the treatment tool detection unit 130. The detected region is set as a treatment tool mask MKtool. The pre-processing unit 141 performs expansion processing on the treatment tool mask MKtool, and obtains a difference by subtracting the treatment tool mask MKtool before expansion from the mask after expansion. This allows a surrounding region mask MKperi to be obtained that follows the contour of the treatment tool.

[0076] The pre-processing unit 141 performs edge extraction on the image within the peripheral region mask MKperi of the endoscopic image, and detects a portion with a large edge component as the ridge of the grasped tissue. The pre-processing unit 141 detects the tip of the treatment tool using the method described in FIG. 14 or the like. The pre-processing unit 141 sets the midpoint of the line connecting the ridge of the grasped tissue and the tip of the treatment tool as the reference point for setting the region of interest. If no ridge is detected in the edge extraction, the pre-processing unit 141 may set the region of interest using the first example of the region of interest setting or the like.

[0077] The "correction of the movement vector considering the movement of the camera" performed by the pre-processing unit will be explained. The movement of the camera is an apparent movement amount, so it affects the movement vector of the treatment tool and the movement vector of the tissue. Therefore, by canceling it, it is expected that the accuracy of contact detection will be improved.

[0078] The pre-processing unit 141 detects the movement amount of the camera, and cancels the influence of the camera movement from the movement vector of the treatment tool and the movement vector of the tissue using the movement amount of the camera. For example, the pre-processing unit 141 subtracts the movement vector of the camera from the movement vector of the treatment tool and the movement vector of the tissue. The pre-processing unit 141 calculates a feature amount such as a distance difference using the movement vector after cancellation. As a method for acquiring the movement amount of the camera, various known techniques can be adopted, and for example, the following two examples are considered. In a first example, a motion sensor or a position sensor using an optical sensor, a magnetic sensor, or an inertial sensor is installed on the camera of the endoscope. The pre-processing unit 141 detects the movement amount of the camera based on the sensor output. In a second example, the pre-processing unit 141 estimates the movement amount of the camera by image processing on the endoscopic image. For example, the pre-processing unit 141 may detect a global movement vector representing the movement of the entire image, and estimate the movement amount of the camera based on the global movement vector.

[0079] The "scaling of the movement vector" performed by the preprocessing unit will now be described. The magnitude of the movement vector on the image is affected by both the magnitude of the movement in real space and the distance between the camera and the subject. For this reason, by scaling using depth distance information or information such as the treatment tool width, the effect of the distance between the camera and the subject can be eliminated, and the accuracy of contact detection can be improved.

[0080] The preprocessing unit 141 detects the distance between the camera and the subject. For example, the endoscope has a 3D camera, and the preprocessing unit 141 detects the distance to the subject by using stereo vision by the 3D camera. Alternatively, the preprocessing unit 141 detects the distance from the endoscopic image to the subject by using a known length such as the shaft width of the treatment tool. Taking the shaft width as an example, the preprocessing unit 141 estimates the distance to the subject from the ratio between the width of the shaft shown in the endoscopic image and the known shaft width. The preprocessing unit 141 scales the movement vector of the treatment tool and the movement vector of the tissue to a movement vector at a reference distance by using the detected distance. The preprocessing unit 141 calculates a feature amount such as a distance difference by using the scaled movement vector.

[0081] The "conversion of movement vectors into three dimensions" performed by the preprocessing unit will now be described.

[0082] The pre-processing unit 141 uses the depth information to obtain the movement vector of the treatment tool and the movement vector of the tissue as three-dimensional vectors, and calculates feature quantities such as distance difference using the three-dimensional movement vectors. The pre-processing unit 141 calculates the three-dimensional movement vector by combining, for example, two-dimensional optical flow and depth information. This method is called scene flow. For example, the endoscope has a 3D camera, and the pre-processing unit 141 detects the amount of movement in the depth direction using stereo vision by the 3D camera.

[0083] Using a three-dimensional movement vector provides the following advantages. Although movement in real space is three-dimensional, in image information it is projected onto a two-dimensional plane, so movement information in the depth direction is lost. By using a three-dimensional movement vector, it becomes possible to capture movement in the normal direction of the image with high sensitivity. This improves the reliability of the movement vector, and therefore the accuracy of contact detection.

[0084] The "time series conversion of features" performed by the pre-processing unit will now be described.

[0085] The preprocessing unit 141 inputs the time-series feature values ​​obtained from multiple frames arranged in time series to the trained model 122b. The trained model 122b detects the contact state between the treatment tool and the tissue from the time-series feature values. When performing contact detection, the accuracy of contact detection can be improved by using the feature values ​​in the most recent multiple frames.

[0086] 19 and 20 are diagrams for explaining tissue separation detection. Here, the treatment tool is an energy device and the end effector is a jaw.

[0087] The value of grasping state judgment is that it can detect tissue separation from images. Tissue separation detection is expected to be effective in preventing damage to the probe of ultrasound devices during surgery. In contact detection methods using movement vectors, the feature value changes suddenly in tissue separation scenes, making detection easy. This is explained below.

[0088] As shown in FIG. 19, in the endoscopic image IMGr, the jaws 12 are not gripping the tissue 50. In the subsequent endoscopic image IMGs, the jaws 12 are gripping the tissue, and energy is applied to the tissue 50 between the jaws. At this point, the tissue gripped between the jaws is connected. In the subsequent endoscopic image IMGt, the tissue 50 between the jaws is cut by energy treatment. In this way, the cutting of connected tissue by energy treatment is called tissue cutting. Note that FIG. 19 shows an example in which the treatment tool is a bipolar device, but cutting can also be detected by the method of this embodiment when a hook knife type monopolar device or the like is used.

[0089] Fig. 20 shows an example of the change over time in the Euclidean distance |wv| between the movement vector of the treatment tool and the movement vector of the tissue. When the tissue 50 is grasped by the jaw 12 as shown by IMGs in Fig. 19, |wv| is small. When the tissue 50 is torn apart as shown by IMGt, the correlation between the movement vectors of the jaw 12 and the tissue 50 becomes small. For this reason, |wv| suddenly increases at the moment of the torn apart, as shown by the dashed-dotted circle in Fig. 20. The contact detection unit 140 detects the torn apart of the tissue by detecting the timing at which |wv| suddenly increases.

[0090] 3. Second embodiment In the second embodiment, the end effector of the treatment tool is a jaw. Note that the description of the same parts as in the first embodiment will be omitted. In the second embodiment, for example, the configuration of the medical system 1, the configuration of the controller 100, and the processing performed by the treatment tool detection unit 130 are the same as in the first embodiment.

[0091] Fig. 21 shows a first configuration example of the contact detection unit in the second embodiment. The contact detection unit 140 includes a jaw opening / closing detection unit 144 and an inter-jaw structure detection unit 146. Fig. 21 shows the process in the learning stage and the process in the inference stage.

[0092] First, the inference stage will be described. The trained model 122 described above in Fig. 2 includes a trained model 122c for detecting jaw opening and closing and a trained model 122d for detecting inter-jaw texture.

[0093] The jaw open / close detection unit 144 detects the open / closed state of the jaw shown in the endoscopic image by using the endoscopic image, the detection result 130Q of the treatment tool detection unit 130, and the trained model 122c. The jaw open / close detection unit 144 inputs the endoscopic image and the detection result 130Q to the trained model 122c, or inputs an image of an attention area set based on the detection result 130Q in the endoscopic image to the trained model 122c. The trained model 122c detects the open / closed state of the jaw from the input data and outputs the detection result 144Q.

[0094] The inter-jaw tissue detection unit 146 detects the presence or absence of tissue between the jaws shown in the endoscopic image by using the endoscopic image, the detection result 130Q of the treatment tool detection unit 130, and the trained model 122d. The inter-jaw tissue detection unit 146 inputs the endoscopic image and the detection result 130Q to the trained model 122d, or inputs an image of an attention area set based on the detection result 130Q from the endoscopic image to the trained model 122d. The trained model 122d detects the presence or absence of tissue between the jaws from the input data and outputs the detection result 146Q. The condition for tissue being present between the jaws is that "tissue that covers one of a pair of jaws and does not cover the other jaw is present." The tissue covering the jaw means that a part or all of the jaw is hidden by the tissue in the image.

[0095] The contact detection unit 140 judges the contact state using the detection results 144Q, 146Q. When it is judged that the jaws are closed and tissue is present between the jaws, the contact detection unit 140 judges that the jaws are gripping tissue, that is, the treatment tool and tissue are in contact. The program that judges the contact state using the detection results 144Q, 146Q may be either a rule-based program or a trained model using machine learning. The processor 110 performs output using the detection result of the contact state. The output contents are the same as those in FIG. 4.

[0096] According to this embodiment, the complex image recognition for detecting a contact state from an image is divided into a jaw open / close detection unit 144 and an inter-jaw structure detection unit 146. This improves the recognition accuracy of each trained model compared to a case where a contact state is detected by a single trained model, and therefore an improvement in the recognition accuracy of the entire contact detection can be expected.

[0097] Next, the learning stage will be described. The learning system generates a learned model 122c by training a model to detect the open / closed state of the jaw from the input data. The input data is an endoscopic image IMG and an annotation indicating the position or area of ​​the treatment tool. Alternatively, the input data is an image of an attention area set based on the annotation in the endoscopic image IMG. The teacher data is a plurality of endoscopic images IMG, annotations indicating the position or area of ​​the treatment tool in each image, and labels indicating the open / closed state of the jaw in each image.

[0098] The learning system generates a trained model 122d by training a model to detect the presence or absence of tissue between the jaws from the input data. The input data is an endoscopic image IMG and an annotation indicating the position or area of ​​a treatment tool. Alternatively, the input data is an image of a region of interest set based on the annotation in the endoscopic image IMG. The teacher data is a plurality of endoscopic images IMG, an annotation indicating the position or area of ​​a treatment tool in each image, and a label indicating the presence or absence of tissue between the jaws in each image. The label indicating the presence or absence of tissue between the jaws is attached based on the above-mentioned condition that tissue exists between the jaws. By executing machine learning using such teacher data, an image in which "there is tissue that covers one of a pair of jaws and does not cover the other jaw" is determined to have tissue between the jaws.

[0099] 22 is a diagram for explaining a second configuration example of the jaw open / closed detection unit 144. In this example, the jaw open / closed detection unit 144 judges the open / closed state of the jaw by a rule-based program using three-dimensional information.

[0100] The treatment tool detection unit 130 acquires the three-dimensional shapes of the shaft 11 and the jaw 12 from the endoscopic image. The treatment tool detection unit 130 inputs stereo images acquired by, for example, a 3D camera to the trained model 122a, and the trained model 122a recognizes the three-dimensional shapes of the shaft 11 and the jaw 12 from the input stereo images.

[0101] The jaw open / close detection unit 144 calculates the shaft axis AX from the three-dimensional shape of the shaft 11, and defines a cylinder CY of a specified radius centered on the shaft axis AX. The jaw open / close detection unit 144 judges whether the jaw 12 is contained within the cylinder CY. If the jaw 12 is contained within the cylinder CY, the jaw open / close detection unit 144 judges that the jaw is closed, and if even a part of the jaw 12 is outside the cylinder CY, the jaw is open.

[0102] According to this embodiment, whether the jaw is open or closed is determined using a rule-based program, so the basis for determining whether the jaw is open or closed is easier to understand than when machine learning is used.

[0103] Fig. 23 shows a second example of the configuration of the inter-jaw structure detection unit. As described above, the condition for the presence of a structure between the jaws is that "a structure that covers one of a pair of jaws and does not cover the other jaw exists". In this example, the inter-jaw structure detection unit 146 judges the presence or absence of a structure covering each jaw using each trained model. Fig. 23 shows the processing in the learning stage and the processing in the inference stage.

[0104] First, the inference stage will be described. The treatment tool detection unit 130 detects the first jaw 12a and the second jaw 12b from the endoscopic image. One of the pair of jaws connected to the shaft 11 is the first jaw 12a, and the other is the second jaw 12b.

[0105] 2 includes a trained model 122d1 for detecting a first jaw covering and a trained model 122d2 for detecting a second jaw covering. The trained model 122d1 judges the first jaw 12a, and the trained model 122d2 judges the second jaw 12b.

[0106] The inter-jaw tissue detection unit 146 inputs the endoscopic image and the area of ​​the first jaw 12a detected by the treatment tool detection unit 130 to the trained model 122d1. The trained model 122d1 detects whether the first jaw 12a is covered with tissue 55 from the input data, and outputs the detection result 122d1Q.

[0107] The inter-jaw tissue detection unit 146 inputs the endoscopic image and the area of ​​the second jaw 12b detected by the treatment tool detection unit 130 to the trained model 122d2. The trained model 122d2 detects whether the second jaw 12b is covered with tissue 55 from the input data, and outputs the detection result 122d2Q.

[0108] The inter-jaw tissue detection unit 146 determines that there is tissue between the jaws when one of the detection results 122d1Q and 122d2Q is "not covered by tissue" and the other is "covered by tissue." If the detection result is anything other than that, the inter-jaw tissue detection unit 146 determines that there is no tissue between the jaws.

[0109] Next, the learning stage will be described. The learning system generates a learned model 122d1 by learning a model to detect whether the first jaw 12a is covered with tissue 55 from input data. The input data is an endoscopic image IMG and an annotation indicating the area of ​​the first jaw 12a. The teacher data is a plurality of endoscopic images IMG, an annotation indicating the area of ​​the first jaw 12a in each image, and a covering label indicating whether the first jaw 12a in each image is covered with tissue 55.

[0110] The learning system generates a trained model 122d2 by training a model to detect whether or not the second jaw 12b is covered by tissue 55 from the input data. The input data is an endoscopic image IMG and an annotation indicating the area of ​​the second jaw 12b. The training data is a plurality of endoscopic images IMG, an annotation indicating the area of ​​the second jaw 12b in each image, and a covering label indicating whether or not the second jaw 12b in each image is covered by tissue 55.

[0111] Fig. 24 shows a third example of the configuration of the inter-jaw structure detection unit. Fig. 24 shows only the trained model 122d2 for detecting the second jaw cover, but the trained model 122d1 for detecting the first jaw cover also performs similar detection and learning for the first jaw 12a. Fig. 24 shows the processing in the learning stage and the processing in the inference stage.

[0112] First, the inference stage will be described. The treatment tool detection unit 130 acquires a two-dimensional normal image and a three-dimensional image of the endoscopic image. The three-dimensional image is, for example, a depth map obtained by stereo vision. The treatment tool detection unit 130 detects the first jaw 12a and the second jaw 12b from each of the normal image and the three-dimensional image.

[0113] The inter-jaw tissue detection unit 146 inputs the three-dimensional image, the area of ​​the second jaw 12b detected from the three-dimensional image, the normal image, and the area of ​​the second jaw 12b detected from the normal image to the trained model 122d2. The trained model 122d2 detects whether the second jaw 12b is covered with tissue 55 from the input data, and outputs the detection result 122d2Q.

[0114] Next, the learning stage will be described. The learning system generates a learned model 122d2 by learning a model to detect whether the second jaw 12b is covered by tissue 55 from input data. In the figure, the symbol "DMAP" indicates a three-dimensional image, and the symbol "IMG" indicates a normal image. The input data is a three-dimensional image DMAP, an annotation indicating the area of ​​the second jaw 12b in the three-dimensional image DMAP, a normal image IMG, and an annotation indicating the area of ​​the second jaw 12b in the normal image IMG. The teacher data is a plurality of three-dimensional images DMAP, an annotation indicating the area of ​​the second jaw 12b in each three-dimensional image DMAP, a plurality of normal images IMG, an annotation indicating the area of ​​the second jaw 12b in each normal image IMG, and a covering label indicating whether the second jaw 12b in each image is covered by tissue 55. A covering label may be attached to each of the three-dimensional image and the normal image, or one covering label may be attached to a set of a three-dimensional image and a normal image captured at the same timing.

[0115] Fig. 25 shows a fourth example of the configuration of the inter-jaw structure detection unit. Fig. 25 shows only the trained model 122d2 for detecting the second jaw cover, but the trained model 122d1 for detecting the first jaw cover also performs similar detection and learning for the first jaw 12a. Fig. 25 shows the processing in the learning stage and the processing in the inference stage.

[0116] First, the inference stage will be described. The trained model 122d2 for detecting the second jaw cover includes a trained model 122d2a for a three-dimensional image and a trained model 122d2b for a normal image.

[0117] The inter-jaw tissue detection unit 146 inputs the three-dimensional image and the area of ​​the second jaw 12b detected from the three-dimensional image to the trained model 122d2a. The trained model 122d2a detects whether the second jaw 12b is covered with tissue 55 from the input data.

[0118] The inter-jaw tissue detection unit 146 inputs the normal image and the area of ​​the second jaw 12b detected from the normal image to the trained model 122d2b. The trained model 122d2b detects whether the second jaw 12b is covered with tissue 55 from the input data.

[0119] The inter-jaw tissue detection unit 146 integrates the detection result of the trained model 122d2a and the detection result of the trained model 122d2b to output a final detection result 122d2Q of whether the second jaw 12b is covered with tissue 55. When the detection result of the trained model 122d2a and the detection result of the trained model 122d2b are the same, the inter-jaw tissue detection unit 146 outputs it as the final detection result 122d2Q. When the detection result of the trained model 122d2a and the detection result of the trained model 122d2b are different, the inter-jaw tissue detection unit 146 outputs the detection result with the higher prediction probability or the detection result selected by a weighting set in advance as the final detection result 122d2Q.

[0120] Next, the learning stage will be described. The learning system generates a learned model 122d2a by training a model to detect whether the second jaw 12b is covered with tissue 55 from input data. The input data is a three-dimensional image DMAP and an annotation indicating the area of ​​the second jaw 12b in the three-dimensional image DMAP. The teacher data is a plurality of three-dimensional images DMAP, an annotation indicating the area of ​​the second jaw 12b in each three-dimensional image DMAP, and a covering label indicating whether the second jaw 12b in each image is covered with tissue 55.

[0121] The learning system generates a trained model 122d2b by training the model to detect whether the second jaw 12b is covered by tissue 55 from the input data. The input data is a normal image IMG and an annotation indicating the area of ​​the second jaw 12b in the normal image IMG. The training data is a plurality of normal images IMG, an annotation indicating the area of ​​the second jaw 12b in each normal image IMG, and a covering label indicating whether the second jaw 12b in each image is covered by tissue 55.

[0122] Fig. 26 shows a second configuration example of the contact detection unit in the second embodiment. The following mainly describes the differences from the first configuration example in Fig. 21.

[0123] There are scenes in which the presence or absence of tissue between the jaws cannot be visually confirmed after the jaws are closed due to reasons such as the pair of jaws overlapping in the depth direction. In such scenes, there is a possibility that the inter-jaw tissue detection unit 146 cannot detect the presence or absence of tissue between the jaws. Therefore, a memory function is added to the inter-jaw tissue detection unit 146. After the contact detection unit 140 recognizes that "tissue is present between the jaws," the contact detection unit 140 maintains the recognition result of "tissue is present between the jaws" by using the memory function, even if the situation changes to one in which it is no longer possible to determine the presence or absence of tissue.

[0124] Specifically, the memory 120 stores the detection result 140Q of the inter-jaw tissue detection unit 146. The memory 120 holds, for example, the detection result 140Q output by the inter-jaw tissue detection unit 146 for a certain period of time from the timing of output. The contact detection unit 140 detects the contact state between the treatment tool and the tissue using the detection result 144Q of the jaw opening / closing detection unit 144, the detection result 146Q of the inter-jaw tissue detection unit 146, and the detection result 146Q of the inter-jaw tissue detection unit 146 stored in the memory 120. Specifically, when the inter-jaw tissue detection unit 146 cannot determine the presence or absence of tissue between the jaws, the contact detection unit 140 detects the contact state between the treatment tool and the tissue from the detection result 144Q of the jaw opening / closing detection unit 144 and the detection result 146Q of the inter-jaw tissue detection unit 146 stored in the memory 120.

[0125] FIG. 27 shows an example of contact detection using the detection result stored in the memory. As shown in the upper diagram, at an arbitrary first time, the jaw open / close detection unit 144 judges that the jaws are open, and the inter-jaw tissue detection unit 146 judges that tissue is present between the jaws. This judgment result of "tissue present between the jaws" is stored in the memory 120. As shown in the lower diagram, at a second time after the first time, the jaw open / close detection unit 144 judges that the jaws are closed, and the inter-jaw tissue detection unit 146 judges that the presence or absence of tissue between the jaws is unknown. In this case, the contact detection unit 140 uses the judgment result of "tissue present between the jaws" at the first time stored in the memory 120 to judge that the jaws are gripping tissue, that is, that the treatment tool and the tissue are in contact.

[0126] According to this embodiment, if it is recognized that there is tissue between the jaws when the jaws are opened, the contact state can be detected from the results stored in memory even if it is not possible to determine the presence or absence of tissue between the jaws from the endoscopic image after the jaws are closed.

[0127] Fig. 28 shows a third configuration example of the contact detection unit in the second embodiment. The following mainly describes the differences from the first configuration example in Fig. 21. The current frame is denoted as t, and the frame n frames before that is denoted as tn.

[0128] First, the inference stage will be described. The treatment tool detection unit 130 detects a treatment tool from the endoscopic image of frame t and outputs a detection result 130Q. The detection results 130Q are also output in frames tn, . . . , t-1, and are stored in the memory 120, for example. The jaw open / close detection unit 144 detects the open / close state of the jaws from the endoscopic image of frame t and outputs a detection result 144Q. The trained model 122 in FIG. 2 includes a trained model 122e for detecting inter-jaw tissue. The inter-jaw tissue detection unit 146 inputs the time-series endoscopic images of frames tn, . . . , t-1, t and the detection results 130Q for each image to the trained model 122e. The trained model 122e detects the presence or absence of tissue between the jaws from the input data and outputs the detection result 146Q. As a trained model that handles time-series data, for example, a neural network using Long Short Term Memory (LSTM) is available. The jaw open / close detector 144 may also be configured to detect the open / close state of the jaw from time-series endoscopic images.

[0129] According to this embodiment, even if the situation changes to one in which it is impossible to determine the presence or absence of tissue between the jaws after the tissue grasping operation, the grasping state can be recognized from the time series information. By recognizing the grasping state including the time series data, even if it is impossible to determine the presence or absence of tissue between the jaws from the endoscopic image, it is possible to recognize whether the tissue is being grasped or not from the immediately preceding grasping detection result.

[0130] 4. Third embodiment In the third embodiment, the treatment tool is an energy device. The energy device may be a device using high-frequency power, ultrasound, or both. The energy device may be a bipolar device or a monopolar device. In the third embodiment, contact detection by image and contact detection by electrical information of the energy device are integrated to detect the contact state between the treatment tool and tissue. Note that the description of the same parts as in the first or second embodiment will be omitted. In the third embodiment, for example, the configuration of the medical system 1, the configuration of the controller 100, and the processing performed by the treatment tool detection unit 130 are the same as in the first embodiment.

[0131] First, the generator 235 that drives the energy device 236 will be described. The generator 235 supplies energy to the energy device 236, controls the energy supply, and acquires electrical information. That is, the generator 235 outputs high-frequency power, and the energy device 236 outputs the high-frequency power from an end effector. Alternatively, the end effector of the energy device 236 has an ultrasonic device. The generator 235 outputs a drive signal for the ultrasonic device, and the ultrasonic device of the end effector receives the drive signal and outputs ultrasonic waves.

[0132] The electrical information is electrical impedance obtained when the energy device 236 outputs high-frequency power to the tissue. However, the electrical information is not limited to this. The electrical information may be current, voltage, or phase between current and voltage. Alternatively, the electrical information may be power, amount of power, impedance, resistance, reactance, admittance (reciprocal of impedance), conductance (real part of admittance), or susceptance (imaginary part of admittance). Alternatively, the electrical information may be the above-mentioned change over time, change between each parameter, differential integral between each parameter (when P is a parameter, the differential over time is dP / dt, and the differential by resistance is dP / dR), a value derived by elementary calculation such as sum and difference for each block, or trigger information such as whether each threshold has been crossed.

[0133] Alternatively, the electrical information is mechanical impedance obtained when the energy device 236 outputs ultrasonic waves to the tissue. However, the electrical information is not limited to this. The electrical information may be a change in mechanical impedance over time, or trigger information such as whether each threshold has been crossed.

[0134] Hereinafter, the electrical information is assumed to be electrical impedance or mechanical impedance, which will be simply referred to as impedance. Impedance in the following description can be read as electrical information.

[0135] 29 shows a first configuration example of the contact detection section in the third embodiment. The contact detection section 140 includes an image information determination section 147, an electrical information determination section 142, and a determination section 143.

[0136] The image information determination unit 147 detects the contact state between the treatment tool and the tissue from the endoscopic image using the trained model for contact detection 122b. The method for detecting contact from this image is as described in the first embodiment. Alternatively, the method for detecting contact from the image may be the method described in the second embodiment.

[0137] When the image information determination unit 147 determines that the treatment tool is in contact with the tissue, the electrical information determination unit 142 acquires impedance from the generator 235 and determines the contact state between the treatment tool and the tissue from the impedance. When the jaws are gripping the tissue or the monopolar is in contact with the tissue, an impedance corresponding to the tissue is observed. From this, contact and non-contact can be determined by the impedance. As an example, the electrical information determination unit 142 determines whether the impedance is equal to or greater than a specified value to determine whether the treatment tool is in contact or not. The electrical information determination unit 142 may be realized by a rule-based program or a trained model using machine learning.

[0138] The determination unit 143 integrates the detection result of the image information determination unit 147 and the detection result of the electrical information determination unit 142 to detect the contact state between the treatment tool and the tissue, and outputs the detection result 140Q. The integration process will be described with reference to the flows in Figs. 30 and 31.

[0139] According to this embodiment, contact detection is performed by combining images and impedance, so that it is expected that the accuracy of contact detection is improved compared to contact detection using only images. Furthermore, when contact detection is performed using only electrical or load information, contact detection can only start when the handpiece switch is turned on. This slows down the action after contact detection. The action is, for example, recognition of the type of tissue or automatic adjustment of energy based on the recognition. Such a slow action reduces the value of contact detection. According to this embodiment, contact detection is also performed from images, so that it is possible to speed up the action after contact detection.

[0140] 30 shows a first example of a flow of processing performed by the contact detection unit in the third embodiment. In this flow, when the trained model 122b of the image information judgment unit 147 judges that the treatment tool is in contact with tissue, the judgment unit 143 performs case classification according to the probability that the contact has been judged.

[0141] In step S11, the image information determination unit 147 performs contact detection using the endoscopic image. This process is performed for each frame of the endoscopic image.

[0142] When it is determined in step S11 that the treatment tool is in contact with the tissue, in step S12, the determination unit 143 compares the probability of contact output by the trained model 122b of the image information determination unit 147 with a threshold value x. The threshold value x is, for example, 60%, but may be any value.

[0143] When the probability is equal to or smaller than the threshold value x in step S12, the electrical information determination unit 142 performs contact detection using impedance in step S16. The determination unit 143 adopts the detection result of the contact state output by the electrical information determination unit 142 as the final detection result 140Q.

[0144] When the probability is greater than the threshold value x in step S12, the electrical information determination unit 142 performs contact detection using impedance in step S13. The determination unit 143 compares the detection result output by the image information determination unit 147 with the detection result output by the electrical information determination unit 142.

[0145] When the two detection results match in step S13, in step S14, the determination unit 143 adopts the detection result of the contact state output by the image information determination unit 147 as the final detection result 140Q.

[0146] If the two detection results do not match in step S13, in step S15, the judgment unit 143 compares the contact probability output by the trained model 122b of the image information judgment unit 147 with a threshold value y. The threshold value y is, for example, 95%, but may be any value greater than the threshold value x.

[0147] When the probability is equal to or smaller than the threshold value y in step S15, in step S16, the determination unit 143 adopts the detection result of the contact state output by the electrical information determination unit 142 as the final detection result 140Q.

[0148] When the probability is greater than the threshold value y in step S15, in step S14, the determination unit 143 adopts the contact state detection result output by the image information determination unit 147 as the final detection result 140Q.

[0149] FIG. 31 is a second flow example of processing performed by the contact detection unit in the third embodiment. In this flow, when the contact detection results of the image and impedance differ, the amount of electrical information used for detection is increased. By utilizing multiple pieces of electrical information instead of only a single piece of impedance information, the detection accuracy is improved. In addition, by utilizing multiple pieces of electrical information only when necessary, the recognition flow is efficient.

[0150] A difference from the first flow of FIG. 30 will be described. If the two detection results are not the same in step S13, in step S17, the electrical information determination unit 142 determines the contact state between the treatment tool and the tissue from the plurality of pieces of electrical information. As an example, the plurality of pieces of electrical information are impedance, impedance variation, and current value. At this time, the electrical information determination unit 142 determines that the treatment tool is in contact with the tissue when the impedance is less than a specified value, the impedance variation is less than a specified value, and the current value is equal to or greater than a specified value. After step S17, in step S16, the determination unit 143 adopts the detection result of the contact state output by the electrical information determination unit 142 as the final detection result 140Q.

[0151] In this flow, in the normal state, the electrical information determination unit 142 performs contact detection by determining whether the impedance is equal to or greater than a specified value. The normal state refers to the path from S12 to S16, or the path from S12 to S13, S14. Then, only in the path from S12 to S13, S17, S16 where it is determined that the two detection results do not match, the electrical information determination unit 142 performs contact detection using a plurality of pieces of electrical information.

[0152] 32 shows a second configuration example of the contact detection section in the third embodiment. The contact detection section 140 includes an image information determination section 147, an electrical information determination section 142, and a separation determination section 148.

[0153] 2 includes a trained model 122f for detecting tissue separation from an image. First, the inference stage will be described.

[0154] The image information judgment unit 147 inputs the endoscopic image and the detection result 130Q of the treatment tool detection unit 130 to the trained model 122f, or inputs an image of a region of interest in the endoscopic image that is set based on the detection result 130Q to the trained model 122f. The trained model 122f detects tissue separation from the input data.

[0155] The electrical information determination unit 142 acquires the detection result of the separation of tissue by utilizing the separation detection function of the generator 235. Alternatively, the electrical information determination unit 142 may acquire impedance from the generator 235 and detect the separation of tissue from the impedance. This separation detection may be performed only when the image information determination unit 147 determines that the treatment tool is in contact with the tissue, or may be performed constantly. As an example, the energy device is an ultrasonic device. The generator 235 may detect the load fluctuation of the ultrasonic probe that occurs when the tissue is separated, and the electrical information determination unit 142 may acquire the detection result of the separation of tissue.

[0156] The split determination unit 148 integrates the split detection result output by the image information determination unit 147 and the split detection result output by the electrical information determination unit 142 to output a final split detection result 140Q. The processor 110 performs output using the detection result 140Q. For example, when it is determined that the tissue is split, the processor 110 transmits an instruction to the generator 235 to suppress or stop the energy output, and the generator 235 suppresses or stops the energy output.

[0157] Next, the learning stage of the trained model 122f will be described. The learning system generates the trained model 122f by training the model to detect tissue separation from the input data. The input data is an endoscopic image IMG and an annotation indicating the position or area of ​​a treatment tool. Alternatively, the input data is an image of an attention area set based on the annotation in the endoscopic image IMG. The teacher data is a plurality of endoscopic images IMG, annotations indicating the position or area of ​​a treatment tool in each image, and labels indicating the tissue separation state in each image. The labels indicating the separation state are, for example, a label indicating before separation and a label indicating after separation.

[0158] The impedance-based severance detection function described above is affected by the state of the biological tissue, the type of biological tissue, the way the handle is held, the way the tissue is gripped, and the like. In addition, there may be cases where severance cannot be detected by images alone due to the influence of tissue adhesion. By combining severance detection based on electrical information and severance detection based on images, the frequency of missed detections can be reduced. As a result, excessive temperature rise of the device after tissue incision can be reduced, and the risk of damage to the probe or tissue pad can be reduced.

[0159] In the above method and the first to third embodiments, the medical system 1 includes a memory 120 storing a trained model 122, and a processor 110. The trained model 122 is a model trained using teacher data including an endoscopic image and a contact state between a treatment tool and tissue in the endoscopic image. The endoscopic image is captured by an endoscope that captures an image including the treatment tool and tissue to be treated. The processor 110 acquires an endoscopic image including the treatment tool and tissue. The processor 110 uses the trained model to detect the contact state between the treatment tool and tissue as first information from the endoscopic image.

[0160] According to this embodiment, the contact state between the treatment tool and tissue is detected from the endoscopic image, and various support can be provided to the surgeon using the detection result. For example, support information such as information on the contacted tissue can be presented to the surgeon only when the treatment tool contacts the tissue. This suppresses the presentation of information at a timing that is unnecessary for the surgeon, reducing the inconvenience. In addition, since the contact state between the treatment tool and tissue can be detected by recognition processing using an image, the contact state can be recognized without increasing the size of the treatment tool while keeping device costs low, compared to the case where a tactile sensor or the like is provided on the treatment tool. In addition, since the contact detection from the image does not limit the device used as the treatment tool, it can be applied to various existing treatment tools.

[0161] In this embodiment, the processor 110 may detect a region of interest including a treatment tool from an endoscopic image. The processor 110 may detect a contact state between the treatment tool and tissue from an image of the region of interest in the endoscopic image by using the trained model 122. The setting of the region of interest is described with reference to Figs. 13 to 18 and the like.

[0162] Among the tissues, the tissues that are affected by the treatment tool are the tissues around the treatment tool that is in contact with the tissue. For example, there is a high correlation between the movement vectors of the treatment tool and the tissues around it. Alternatively, the tissues that are affected by the energy treatment are the tissues around the treatment tool. According to this embodiment, a region of interest including the treatment tool is detected, and a contact state is detected from an image of the region of interest, so that the contact state can be detected with high accuracy using the tissues around the treatment tool that are affected by the treatment tool.

[0163] In this embodiment, when the processor 110 detects first information indicating that the treatment tool is in contact with tissue, the processor 110 may perform support processing related to treatment using the treatment tool.

[0164] In the present embodiment, the support process may be a process of presenting the first information, a process of presenting support information related to a treatment, a process of presenting support information related to an energy output setting of a treatment tool that is an energy device, or a process of automatically controlling the treatment tool that is an energy device. An example of the support information is described in FIG. 4 and the like.

[0165] According to this embodiment, the contact state between the treatment tool and tissue is detected from an endoscopic image, and the detection result can be used to provide various types of support to the surgeon. "Support information regarding treatment" is, for example, the type or state of tissue with which the treatment tool has come into contact. "Support information regarding energy output settings" is, for example, recommended output settings determined from an image or the like. "Processing for automatically controlling the treatment tool" is, for example, processing for changing, suppressing, or stopping energy output. Note that the support processing is not limited to the examples given here, as long as it is support using the detection result of the contact state.

[0166] In this embodiment, the processor 110 may obtain an average movement vector of the treatment tool and an average movement vector of the surrounding tissue, which is the tissue around the treatment tool, from the endoscopic image. The processor 110 may detect a contact state between the treatment tool and the tissue from the average movement vector of the treatment tool and the average movement vector of the surrounding tissue, using the trained model 122. Contact detection using the movement vector is described in Figs. 5 to 11 and the like.

[0167] As explained in Fig. 6 etc., when the treatment tool is in contact with the tissue, the correlation between the movement vectors of the treatment tool and the tissue is high, and when the treatment tool is not in contact with the tissue, the correlation between the movement vectors of the treatment tool and the tissue is low. According to this embodiment, this correlation can be utilized to detect the contact state between the treatment tool and the tissue.

[0168] Furthermore, in this embodiment, the processor 110 may detect the contact state between the treatment tool and tissue using at least one of the following (i), (ii), (iii), and (iv): (i) average movement vector of the treatment tool; (ii) average movement vector of the surrounding tissue; (iii) Euclidean distance between the average movement vector of the treatment tool and the average movement vector of the surrounding tissue; (iv) directional difference between the average movement vector of the treatment tool and the average movement vector of the surrounding tissue. These methods are described with reference to Figs. 7 to 9, etc.

[0169] In this embodiment, the processor 110 may also decompose the movement vector of the surrounding tissue into a component parallel to the movement direction of the treatment tool and a component perpendicular to the movement direction of the treatment tool, and detect the contact state between the treatment tool and the tissue using the statistics of the parallel component and the perpendicular component. These methods are described in Figs. 10 to 11, etc.

[0170] According to this embodiment, by using the average movement vector or an amount calculated from the average movement vector, the correlation between the movement vectors of the treatment tool and the tissue can be evaluated and the contact state between the treatment tool and the tissue can be detected.

[0171] In this embodiment, the treatment tool may be an energy device that treats tissue by performing energy output from an end effector. The processor 110 may determine the contact state between the treatment tool and tissue using first information, which is the contact state determined from an endoscopic image, and second information, which is the contact state determined from electrical information in the energy output. Such a method is described in Figs. 29 to 32, etc.

[0172] According to this embodiment, contact detection is performed by combining image-based contact detection and electrical information-based contact detection, so that improved accuracy of contact detection can be expected compared to contact detection performed from only one side.

[0173] In this embodiment, the contact state may be determined as to whether the treatment tool is in contact with the tissue or not. The processor 110 may determine whether to adopt the first information or the second information depending on the estimated probability that the treatment tool is in contact with the tissue in the estimation of the first information using the trained model 122. Such a method is described in Figs. 30 to 31 and the like.

[0174] According to this embodiment, it is possible to determine whether to adopt the result of contact detection based on an image or the result of contact detection based on electrical information, depending on the estimated probability of contact detection based on an image. For example, when the estimated probability of contact detection based on an image is high, the result of contact detection based on an image can be adopted. When the estimated probability of contact detection based on an image is high, it is possible to determine whether to adopt the result of contact detection based on an image or the result of contact detection based on electrical information, while taking into account the result of contact detection based on electrical information.

[0175] In addition, in this embodiment, when the first information and the second information do not match, the processor 110 may determine the contact state from a plurality of pieces of electrical information in the energy output. Such a method is described in FIG. 31 and the like.

[0176] According to the present embodiment, when the result of contact detection based on the image and the result of contact detection based on the electrical information are not consistent, it is assumed that the estimated probability of contact detection is low. In such a case, the contact state can be detected with high accuracy by determining the contact state from a plurality of pieces of electrical information.

[0177] In this embodiment, the treatment tool may have a pair of jaws that open and close. The memory 120 may store the first learned model and the second learned model as the learned model 122. The first learned model may be a model that has been learned to detect the opening and closing of the jaws from an endoscopic image. The second learned model may be a model that has been learned to detect the presence or absence of the tissue between the jaws from an endoscopic image. The processor 110 may perform a first detection to detect the opening and closing of the jaws from an endoscopic image using the first learned model. The processor 110 may perform a second detection to detect the presence or absence of tissue between the jaws from an endoscopic image using the second learned model. The processor 110 may detect the contact state between the treatment tool and the tissue based on the result of the first detection and the result of the second detection. Such a method is described in FIG. 21 to FIG. 28, etc. In the configuration examples of FIG. 21 and FIG. 26, the learned model 122c is the first learned model, and the learned model 122d is the second learned model. In the configuration example of FIG. 28, trained model 122c is the first trained model, and trained model 122e is the second trained model.

[0178] According to this embodiment, the trained model for contact detection is separated into a first trained model for jaw opening / closing detection and a second trained model for jaw structure detection. This improves the estimation accuracy of each model compared to estimating contact detection with a single model, and allows for more accurate contact detection.

[0179] In addition, in this embodiment, the processor 110 may determine that the treatment tool is in contact with tissue when the first detection detects that the jaws are closed and the second detection detects that there is tissue between the jaws.

[0180] In this embodiment, the condition for determining that tissue is present between the jaws in the second detection is that one of the pair of opening and closing jaws is covered with tissue and the other jaw is not covered with tissue.

[0181] By using these judgment conditions, it is possible to determine whether the jaws of the treatment tool are grasping tissue, i.e., whether the treatment tool is in contact with tissue, based on the results of jaw opening / closing detection and the results of tissue detection between the jaws.

[0182] In this embodiment, the treatment tool may be an energy device that treats tissue by performing energy output from an end effector. The processor 110 may present an output setting for the energy output when it detects first information indicating contact between the treatment tool and tissue. The processor 110 may cause the energy device to output energy at the output setting when it receives an output instruction after the presentation. Such a method is described in FIG. 1 and FIG. 4, etc.

[0183] According to this embodiment, only when the treatment tool comes into contact with tissue, support information such as information on the tissue in contact is presented to the surgeon, and the surgeon is requested to input approval or rejection. This prevents the presentation of information at a time when it is not necessary for the surgeon, reducing the inconvenience.

[0184] In this embodiment, the treatment tool may be an energy device that treats tissue by outputting energy from an end effector. The processor 110 may make a decision regarding a change in the setting of the energy output when it detects first information indicating that the treatment tool and tissue have changed from contact to non-contact. The detection of tissue separation is described in Figs. 19 and 20, etc. The "decision regarding a change in the setting of the energy output" is, for example, a decision as to whether or not to change, suppress, or stop the energy output when the tissue is separated.

[0185] According to this embodiment, by detecting a change from contact between the treatment tool and the tissue to non-contact, it is possible to detect that the tissue has been torn apart by the energy treatment. When the treatment tool and the tissue are no longer in contact, a decision can be made regarding changing the energy output settings to assist the surgeon.

[0186] This embodiment may also be implemented as a contact state detection method as follows. That is, the contact state detection method includes capturing an endoscopic image including a treatment tool and tissue to be treated by an endoscope. The contact state detection method includes detecting a contact state between the treatment tool and the tissue as first information from the endoscopic image using a trained model. The trained model is a model trained by teacher data including an endoscopic image and a contact state between the treatment tool and the tissue in the endoscopic image.

[0187] 5. Fourth embodiment In the fourth embodiment, the treatment tool is an energy device. The energy device may be a device using high-frequency power, ultrasound, or both. The energy device may be a bipolar device or a monopolar device. In the fourth embodiment, the contact state between the treatment tool and the tissue is detected by combining jaw opening / closing detection by an image and jaw tissue detection by electrical information. Note that the description of the same parts as in the first, second, or third embodiment is omitted. In the fourth embodiment, for example, the configuration of the medical system 1, the configuration of the controller 100, and the processing performed by the treatment tool detection unit 130 are the same as in the first embodiment.

[0188] 33 shows an example of the configuration of a contact detection unit in the fourth embodiment. The contact detection unit 140 includes a jaw open / close detection unit 144 and an electrical information determination unit 142.

[0189] The jaw open / closed state detection unit 144 detects the open / closed state of the jaw shown in the endoscope image from the endoscope image by using the trained model 122c. This detection method is as described with reference to FIG.

[0190] When the jaw open / close detection unit 144 determines that the jaws are closed, the electrical information determination unit 142 acquires electrical information from the generator 235, detects the contact state between the treatment tool and the tissue using the electrical information, and outputs the detection result 140Q. Specifically, the electrical information determination unit 142 determines the presence or absence of tissue between the jaws using the electrical information. This detection method is as described in the third embodiment.

[0191] When the electrical information determining section 142 determines that tissue is present between the jaws, the contact detecting section 140 determines that the treatment tool is gripping tissue, that is, that the treatment tool is in contact with tissue.

[0192] In the above fourth embodiment, the medical system 1 includes a memory 120 storing a trained model 122 and a processor 110. The trained model 122 is a model trained by teacher data. The teacher data includes an endoscopic image captured by an endoscope that captures an image including a treatment tool having an opening and closing jaw and tissue to be treated, and an open / closed state of the jaw in the endoscopic image. The treatment tool is an energy device that treats tissue by performing energy output from the jaw. The processor 110 acquires an endoscopic image including the treatment tool and tissue. The processor 110 performs a first detection that detects the opening and closing of the jaw from the endoscopic image using the trained model. The processor 110 performs a second detection that detects the presence or absence of tissue between the jaws based on electrical information in the energy output. The processor 110 detects the gripping state of the tissue by the jaws as a contact state based on the result of the first detection and the result of the second detection.

[0193] When considering recognition from an image, it is assumed that jaw opening and closing is easier to detect than contact between a treatment tool and tissue. According to the present embodiment, the contact state can be detected with high accuracy by combining jaw opening and closing detection, which is considered to be more accurate than contact state detection, with inter-jaw tissue detection by impedance.

[0194] Furthermore, in this embodiment, when the processor 110 detects in the first detection that the jaws are closed, the processor 110 performs the second detection.

[0195] According to this embodiment, only the jaw opening / closing detection is performed by the image until the jaws are detected to be closed, and when the jaws are detected to be closed, the inter-jaw tissue detection is performed by the impedance, thereby reducing the computational load.

[0196] Also in this embodiment, processor 110 determines that the jaws are gripping tissue when processor 110 detects that the jaws are closed in the first detection and detects that tissue is between the jaws in the second detection.

[0197] By using such judgment conditions, it is possible to determine whether the jaws of the treatment tool are grasping tissue, i.e., whether the treatment tool is in contact with tissue, based on the results of jaw opening / closing detection and the results of tissue detection between the jaws.

[0198] The present embodiment may also be implemented as a contact state detection method as follows. That is, the contact state detection method is a method for detecting a contact state between a treatment tool and tissue to be treated. The treatment tool is an energy device that has jaws that open and close, and treats tissue by outputting energy from the jaws. The contact state detection method includes capturing an endoscopic image including the treatment tool and the tissue to be treated by an endoscope. The contact state detection method includes performing a first detection to detect the opening and closing of the jaws from the endoscopic image using the trained model 122. The trained model 122 is a model trained by teacher data including an endoscopic image and an open and closed state of the jaws in the endoscopic image. The contact state detection method includes performing a second detection to detect the presence or absence of tissue between the jaws based on electrical information in the energy output. The contact state detection method includes detecting a state in which the tissue is gripped by the jaws as a contact state based on a result of the first detection and a result of the second detection.

[0199] Although the embodiments to which the present disclosure is applied and their modifications have been described above, the present disclosure is not limited to each embodiment or its modifications as it is, and in the implementation stage, the components can be modified and embodied within the scope of the gist of the disclosure. In addition, various disclosures can be formed by appropriately combining multiple components disclosed in each of the above-mentioned embodiments and modifications. For example, some components may be deleted from all the components described in each embodiment or modification. Furthermore, components described in different embodiments or modifications may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the disclosure. In addition, a term described at least once in the specification or drawings together with a different term having a broader meaning or the same meaning can be replaced with that different term in any part of the specification or drawings. [Explanation of symbols]

[0200] 1... medical system, 10... treatment tool, 11... shaft, 12... jaw, 14... end effector, 50... surrounding tissue, 100... controller, 110... processor, 120... memory, 121... program, 122... model, 130... treatment tool detection unit, 140... contact detection unit, 141... pre-processing unit, 142... electrical information judgment unit, 143... judgment unit, 144... jaw opening / closing detection unit, 146... jaw tissue detection unit, 147... image information judgment unit, 148... judgment unit, 180, 190... I / O device, 200... endoscope system, 230... system, 231... monitor, 232... endoscope image, 233... output setting, 235... generator, 236... energy device, IMG... endoscope image, ROIc... region of interest, Vtissue1, Vtissue2... movement vector, Vtool... movement vector, v... average movement vector, w... average movement vector

Claims

1. A memory that stores a trained model trained using training data including an endoscopic image captured by an endoscope that captures an image including a treatment tool and a tissue to be treated, and a contact state between the treatment tool and the tissue in the endoscopic image; A processor; Including, The processor, acquiring an endoscopic image including the treatment tool and the tissue; A medical system characterized in that the contact state between the treatment tool and the tissue is detected as first information from the endoscopic image using the trained model.

2. 2. The medical system according to claim 1, The processor, A medical system comprising: a first image processing unit that detects an area of ​​interest including the treatment tool from the endoscopic image; and a second image processing unit that detects the contact state between the treatment tool and the tissue from an image of the area of ​​interest in the endoscopic image using the trained model.

3. 2. The medical system according to claim 1, The processor, A medical system comprising: a treatment tool; a treatment device that performs, when the first information indicating that the treatment tool is in contact with the tissue is detected, a treatment tool that performs a support process for a treatment using the treatment tool.

4. 4. The medical system according to claim 3, The medical system, characterized in that the support processing is a processing of presenting the first information, a processing of presenting support information regarding the treatment, a processing of presenting support information regarding energy output settings of the treatment tool which is an energy device, or a processing of automatically controlling the treatment tool which is an energy device.

5. 2. The medical system according to claim 1, The processor, From the endoscopic image, an average movement vector of the treatment tool and an average movement vector of the surrounding tissue, which is the tissue around the treatment tool, are calculated; A medical system characterized by detecting the contact state between the treatment tool and the tissue from the average movement vector of the treatment tool and the average movement vector of the surrounding tissue using the learned model.

6. 6. The medical system according to claim 5, The processor, A medical system characterized by detecting the contact state between the treatment tool and the tissue using at least one of the average movement vector of the treatment tool, the average movement vector of the surrounding tissue, the Euclidean distance between the average movement vector of the treatment tool and the average movement vector of the surrounding tissue, and the directional difference between the average movement vector of the treatment tool and the average movement vector of the surrounding tissue.

7. 6. The medical system according to claim 5, The processor, A medical system characterized by decomposing the movement vector of the surrounding tissue into components parallel to and perpendicular to the movement direction of the treatment tool, and detecting the contact state between the treatment tool and the tissue using statistics of the parallel components and the perpendicular components.

8. 2. The medical system according to claim 1, The treatment tool is an energy device that treats the tissue by outputting energy from an end effector, The processor, A medical system characterized in that the contact state between the treatment tool and the tissue is determined using first information, which is the contact state determined from the endoscopic image, and second information, which is the contact state determined from electrical information in the energy output.

9. 9. The medical system according to claim 8, The determination of the contact state is a determination of whether the treatment tool is in contact with the tissue or not, The processor, A medical system characterized in that it determines whether the first information or the second information is to be adopted depending on an estimated probability that the treatment tool and the tissue are in contact in estimating the first information using the trained model.

10. 9. The medical system according to claim 8, The processor, A medical system characterized in that, when the first information and the second information are inconsistent, the contact state is determined from a plurality of electrical information in the energy output.

11. 2. The medical system according to claim 1, The treatment tool has a pair of jaws that open and close, The memory includes: A first trained model trained to detect the opening and closing of the jaws from the endoscopic image and a second trained model trained to detect the presence or absence of the tissue between the jaws from the endoscopic image are stored as the trained models; The processor, performing a first detection of detecting opening and closing of the jaw from the endoscopic image using the first trained model; performing a second detection to detect the presence or absence of tissue between the jaws from the endoscopic image using the second trained model; A medical system comprising: a treatment device and a tissue; a treatment apparatus for treating a tissue; and a treatment method for treating a tissue.

12. 12. The medical system of claim 11, The processor, A medical system characterized in that when the first detection detects that the jaws are closed and the second detection detects that the tissue is between the jaws, it is determined that the treatment tool is in contact with the tissue.

13. 13. The medical system of claim 12, A medical system characterized in that the condition for determining that the tissue is present between the jaws in the second detection is that one of a pair of opening and closing jaws is covered by the tissue and the other jaw is not covered by the tissue.

14. 2. The medical system according to claim 1, The treatment tool is an energy device that treats the tissue by outputting energy from an end effector, The processor, A medical system characterized in that when the first information indicating contact between the treatment tool and the tissue is detected, an output setting for the energy output is presented, and when an output instruction is received after the presentation, the energy output is performed from the energy device at the output setting.

15. 2. The medical system according to claim 1, The treatment tool is an energy device that treats the tissue by outputting energy from an end effector, The processor, A medical system comprising: a first information unit that detects a change in contact between the treatment tool and the tissue from a non-contact state; and a second information unit that detects a change in contact between the treatment tool and the tissue from a non-contact state.

16. capturing an endoscopic image including a treatment tool and tissue to be treated by an endoscope; Detecting the contact state between the treatment tool and the tissue from the endoscopic image as first information using a trained model trained with teacher data including the endoscopic image and a contact state between the treatment tool and the tissue in the endoscopic image; A contact state detection method comprising:

17. A memory that stores a trained model trained using teacher data including an endoscopic image captured by an endoscope that captures an image including a treatment tool having an opening and closing jaw and a tissue to be treated, and an open / closed state of the jaw in the endoscopic image; A processor; Including, The treatment tool is an energy device that treats the tissue by outputting energy from the jaws, The processor, acquiring an endoscopic image including the treatment tool and the tissue; performing a first detection of detecting opening and closing of the jaw from the endoscopic image using the trained model; performing a second detection to detect the presence or absence of tissue between the jaws based on electrical information of the energy output; A medical system comprising: a medical device that detects a state in which the jaws grasp the tissue as a contact state based on a result of the first detection and a result of the second detection.

18. 18. The medical system of claim 17, The processor, A medical system comprising: a medical device that performs the second detection when it is detected in the first detection that the jaws are closed.

19. 18. The medical system of claim 17, The processor, a medical system that determines that the jaws are grasping the tissue when the first detection detects that the jaws are closed and the second detection detects that the tissue is between the jaws.

20. A contact state detection method for detecting a contact state between a treatment tool, which is an energy device having jaws that open and close and which treats tissue by outputting energy from the jaws, and a tissue to be treated, comprising: capturing an endoscopic image including the treatment tool and the tissue with an endoscope; performing a first detection of detecting whether the jaw is open or closed from the endoscopic image by using a trained model trained from teacher data including the endoscopic image and the open / closed state of the jaw in the endoscopic image; performing a second detection to detect the presence or absence of tissue between the jaws based on electrical information of the energy output; Detecting a state of the tissue being grasped by the jaws as the contact state based on a result of the first detection and a result of the second detection; A contact state detection method comprising:

Images