Tissue therapy energy feedback systems and methods

Abstract

A system for imaging and treating tissue within a subject using a tissue therapy device includes a tissue therapy device for delivering therapy to internal tissue, a sensor for sensing electrical parameters at the device, an imaging sensor for obtaining internal imaging information, and controller circuitry with signal-processing capabilities, which can determine changes in the state of the tissue therapy based on sensed electrical parameters, processes imaging information to detect changes in the state of the tissue therapy device, and adjust system output if a change is detected. A method for imaging and treating tissue using a tissue therapy device involves monitoring an end effector of a medical device with an imaging sensor, energizing the end effector with electrosurgical energy for surgery, sensing parameters of the energy, determining changes in the surgical operation state, and adjusting electrosurgical system output if a change in the end effector state is detected.

Description

TISSUE THERAPY ENERGY FEEDBACK SYSTEMS AND METHODSPRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 611,406, filed December 18, 2023, and U.S. Provisional Patent Application Serial No. 63 / 637,633, filed April 23, 2024, the contents of which are incorporated herein by reference.BACKGROUND

[0002] Various different types of energy, e.g., radio frequency (RF) or other electromagnetic energy, plasma energy, or ultrasound energy, can be used for vessel sealing, tissue cutting, tissue cautery, tissue ablation, or tissue coagulation, among other things. The energy can be used alone or in combination with mechanical energy delivery (e.g., using a sharp cutting instrument) or manipulation (e.g., using a forceps). Many types of monopolar and bipolar energy devices exist for different surgical purposes. In some examples of an energy delivery device (“energy device”), a forceps can be utilized for laparoscopic surgery. The forceps can be employed to perform delicate movements of tissue inside a patient, such as by using a gripping assembly and / or a cutting assembly to interact with tissue. The forceps can deliver sealing energy to the gripping assembly, e.g., jaws of the forceps, to seal tissue. For example, electrosurgical sealing forceps can include or use an energy device such as RF, ultrasonic, and microwave vessel sealing devices. Elastin or collagen within tissue clamped by the forceps can be melted by the energy device in order to seal the tissue.

[0003] Examples of electrosurgical systems are described in Pub. No. WO 2022 / 187832A2 titled “Tissue Therapy Energy Delivery Using Optical Imaging” to Yang et al., assigned to Gyrus ACMI, Inc., the entire contents of which are incorporated herein in their entirety by this reference.OVERVIEW

[0004] The present inventor has recognized, among other things, that tissue treatment procedures using energy delivery can benefit from improvements including reduced procedure time, reduced complications from treatment, and reducing the requisite skill level of the surgeon or other practitioner performing the procedure. These improvements scan help reduce or avoid the need for surgical “tricks” or techniques to accommodate changing conditions at a target site during the procedure. Furthermore, these improvements can help remove variation between surgeons and variability of a single surgeon in performing tissuesealing procedures. For example, some electrosurgical systems can include or use an electrosurgical device with a radio frequency (RF) energy delivery system that can provide nearly instantaneous feedback about one or more conditions (e.g., tissue impedance, resistance, phase angle of therapy power delivery, or the like) at the target site. The feedback can be used to adjust electrosurgical therapy energy delivered to the tissue at the target site based on an electrical feedback signal representative of one or more such conditions at the target site. However, a single, common RF electromagnetic energy waveform application process might be applied to different tissue types. Each tissue type can exhibit different characteristics during a particular procedure, such as during vessel sealing, or during monopolar or bipolar cutting or any other tissue treatment. For example, a carotid artery and a renal artery have different vessel sealing characteristics but, in one approach, each of such vessels would likely be controlled by the same therapy energy waveform application process. Similar considerations can exist for other tissues, such as fats, ligaments, or other tissues, each of which can have other specific therapy energy waveforms. Thus, in some cases, the electrical tissue characteristic (e.g., tissue impedance) feedback from the target tissue site alone may not be enough to appropriately adjust energy therapy delivery, or may not be enough to appropriately distinguish between tissue types to allow tailoring or adjustment or other nuances in the way that the electrical energy is delivered to the target tissue site. For example, the present inventor has recognized, among other things, that a carotid artery can benefit from electrical therapy power applied relatively more slowly, e.g., for vessel sealing, while a renal artery can tolerate faster power application without creating tissue “popping” problems. Such tissue popping is a phenomenon in which steam generated within the tissue exits, e.g., from between forceps jaws, applying mechanical pressure and electrical energy together, at such speed that it can cause damage to the vessel wall including at one or more locations that can be away from the target vessel sealing site. Thus, the present inventor has recognized that other forms of feedback, in addition to or other than electrical feedback, can be used to help make different, e.g., better tailored, energy delivery decisions.

[0005] The present inventor has recognized that the aforementioned tissue feedback signal based on impedance, resistance or phase angle can be affected by articulation or manipulation of the therapy device, such as articulation of jaws of a forceps. For example, during the application of RF energy, a controller for a vessel sealing device may look for changes in electrical resistance to determine the state of the tissue. If the resistance is increasing, it is often considered that the tissue is moving to complete its sealing phase, e.g., drying out, and that the energy application is close to completion. Different tissue types might have differentresistance feedback signal profiles, e.g., the resistance feedback signals might increase at different rates. However, for a given tissue type, a similar increase of electrical resistance can possibly be produced by the surgeon slowly opening the device jaws before the tissue has been sealed. In other words, the rise in resistance of the tissue becoming sealed, e.g., drying out, can look similar to the rise in resistance of the jaws opening slowly before the tissue is sealed. This can electrically fool the system controller and make the system controller think that the sealing device has achieved its desired sealing cycle. Then, once the user cuts the tissue, internal hemorrhage can occur since the vessel might not actually be sealed.

[0006] The present disclosure can provide solutions to these and other problems by combining a technique for quantifying the delivered energy, e.g., electrical feedback comprising resistance magnitude or resistance rate of change, and visualized feedback, e.g., video images of the clamping state of jaws of a forceps, to prevent or reduce the occurrence of these risks. The electrical feedback and video feedback can be combined to generate improved automated decisions. For example, a tissue therapy device can incorporate imaging capabilities, such as on the tissue therapy device itself or on an endoscope used in conjunction with the tissue therapy device. A camera can monitor the clamping state of jaws of the tissue therapy device to determine if they are opened or closed or engaged with tissue or not engaged with tissue. The open or closed state of the jaws can facilitate a determination of whether or not a proper seal has been formed in tissue engaged by the jaw. The image output of the camera can be compared to simultaneously obtained electrical feedback, such as resistance, impedance, phase angle feedback, time, etc., to verify the accuracy of the electrical feedback. For example, changes in sensed resistance (AR or R) that indicate or appear to indicate drying of tissue during a proper tissue sealing event can be compared to the open or closed state of the jaws. The open or closed state of the jaws can be used to verify the electrical feedback, e.g., the magnitude or rate of change of a sensed electrical (phase, impedance, resistance, etc.) signal. In examples, one or more of visual, audio and haptic feedback can be provided to a user to take corrective action, or an electrosurgical system can perform automatic corrective actions, among other things.

[0007] In an example, a system for imaging and treating tissue of a subject can comprise a tissue therapy device configured to deliver a tissue therapy output for applying to a tissue therapy to tissue at a location internal to the subject, a sensor configured to sense an electrical parameter at the tissue therapy device, an imaging sensor adapted to obtain imaging information of the tissue therapy device from within the location, and controller circuitry comprising signal-processing circuitry configured to determine a change in state of the tissuetherapy from the electrical parameter that is sensed, image-process the imaging information to determine if a change in state of the tissue therapy device internal to the subject has occurred when the change in state of the tissue therapy occurred, and generate or adjust an output of the system if a change in state of the tissue therapy device has occurred.

[0008] In another example, a method for treating tissue using an electrosurgical system can method comprise monitoring operation of an end effector assembly of a medical device with an imaging sensor, energizing the end effector assembly using an electrosurgical energy to perform a surgical operation, sensing a parameter of the electrosurgical energy at the end effector assembly, determining a change in state of the surgical operation from the parameter that is sensed, determining if there is a change in state of the end effector assembly using the imaging sensor when a change in state of the surgical operation is sensed, and adjusting or generating an output of the electrosurgical system if there is a change in state of the end effector assembly.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 A is a side view of an example of an electrosurgical system comprising forceps and electrosurgical feedback capabilities of the present disclosure.

[0010] FIG. IB is a close-up perspective view of a jaw assembly suitable for use with the forceps of FIG. 1 A in a first, open position.

[0011] FIG. 1C is close-up perspective view of the jaw assembly of FIG. IB in a second, closed position.

[0012] FIG. ID is a schematic illustration of the jaw assembly of FIG. IB and FIG. 1C grasping a target object, such as a blood vessel.

[0013] FIG. IE is a perspective view of an end effector suitable for use with the forceps of FIG. 1 A comprising an integrated imaging device.

[0014] FIG. 2 is a schematic diagram of an electrosurgical system in use by a surgeon illustrating a display screen showing forceps engaging a target object.

[0015] FIG. 3 is a block diagram showing example components of an electrosurgical system having a feedback system capable of performing one or more of the methods, procedures and operations discussed herein.

[0016] FIG. 4A is a schematic diagram of jaws of a forceps engaging unsealed tissue before pressure is applied.

[0017] FIG. 4B is a schematic diagram of the forceps of FIG. 4A applying pressure to collapse the tissue and apply sealing energy to the tissue.

[0018] FIG. 4C is a schematic diagram of the forceps of FIG. 4B disengaged from sealed or potentially sealed tissue.

[0019] FIG. 5 is a chart plotting the application of pressure by a forceps and resistance sensed between jaws of the forceps over time, wherein pressure is applied by the forceps for a time sufficient to seal tissue.

[0020] FIG. 6 is a chart plotting the application of pressure by a forceps and resistance sensed between jaws of the forceps over time, wherein pressure is removed rapidly before a sufficient seal is formed.

[0021] FIG. 7 is a chart plotting the application of pressure by a forceps and resistance sensed between jaws of the forceps over time, wherein pressure is removed slowly before a sufficient seal is formed.

[0022] FIG. 8 A is a line diagram illustrating methods of determining status of an electrosurgical event using images of a medical device and electrical feedback from the medical device to provide feedback to a user and to take corrective action.

[0023] FIG. 8B is a line diagram illustrating methods of determining if a sealing event performed with an electrosurgical forceps has completed using images of jaws of the forceps and electrical feedback from sensors in the forceps.

[0024] FIG. 9 is a schematic diagram of a machine learning model system illustrating inputs for determining surgical procedure event status and outputs for providing user feedback and initiating system corrective action.

[0025] FIG. 10 is a schematic diagram of a computing system for use with the electrosurgical system of FIG. 3.DETAILED DESCRIPTION

[0026] This disclosure relates to devices and methods for treating tissue such as, but not limited to, blood vessels including arteries, capillaries, and veins. In some approaches to tissue treatment, an energy device, such as radio frequency (RF), ultrasonic, and microwave vessel sealing devices, can be used as a part of an electrosurgical sealing forceps. An end effector assembly, such as forceps, can include jaws that can be operated by a user to provide a desired jaw pressure to compress the tissue for electrosurgical sealing. The energy devices can be used to apply energy to heat one or more tissue components, e.g., collagen or elastin of the tissue. The heated and potentially melted tissue components can form a seal in the tissue. Certain electrosurgical sealing forceps can also include a cutting element as part of the end effector, such as a stationary or moveable cutting blade to cut the sealed vessel. Suchdevices can use a relatively high jaw closure force, e.g., high relative to the amount of force needed to deform the tissue, to compress the vessel tissue for sealing. Generally, the desired jaw pressure that a user can apply to perform a successful sealing operation can lie within a relatively small range of acceptable pressures. The desired jaw pressure can be applied for a sufficient length of time while sealing energy is simultaneously applied to form a proper seal. A control system for the electrosurgical sealing forceps can monitor electrical feedback from the jaws to determine if a seal has been completed. In particular, a tissue sealing event aims to prevent blood flow through the tissue by closing off a blood passage enclosed by the tissue and preventing blood flow within the tissue itself. As such, the tissue can be transformed from typical tissue having moisture content of blood and other tissue (elastin or collagen) to dried out tissue having no moisture or a substantially reduced amount of moisture. Thus, a positive indication that tissue has been sealed of is whether or not the tissue has dried out. An indication of the moisture content of tissue can be approximated by the electrical resistance provided by the tissue, with dried out tissue providing much higher electrical resistance than moist tissue. Further description of the formation of a proper or adequate tissue seal is described with reference to FIG. 5.

[0027] A problem with this approach, however, can be that an undesired jaw closure force can cause unwanted tissue outcomes, such as insufficiently sealed tissue or damaged tissue, during the procedure if the tissue is not held at an appropriate target tissue pressure and for a sufficient amount of time. Providing the appropriate target tissue pressure can be challenging for users, for example, if the region between the jaws is not filled with tissue. In such a case, the closure force needed from the end effector to supply the target tissue pressure can vary based on the amount of tissue is in contact with a contact surface area of the jaws. For instance, at a fixed closure force in a case in which the jaws compress a relatively small amount of tissue, a greater tissue pressure is supplied than in a case in which the jaws compress a relatively large amount of tissue at the same fixed closure force. Also, providing the desired tissue pressure can be challenging for users due to varying tissue thickness, morphology, size, volume, density, composition, moisture content, and the like. As mentioned, the “target tissue pressure” can refer to relatively small range of acceptable pressures for the procedure. Supplying the target tissue pressure to perform a successful procedure is typically incumbent on the user. If too little closure force is applied, the tissue is compressed at too small of a tissue and the resultant seals created can have a very low ‘burst pressure’ (e.g., the blood pressure required to prevent the created seal from breaking). This is because the ‘glue’ created from the melted elements of the vessels such as the collagen, is notsqueezed together sufficiently. This insufficient pressurization of the glue can result in a weakness at the pressure-treated joint interface and results in poor seals. If too great of closure force is applied, the tissue pressure is too great and, irrevocable tissue damage can occur such as tearing and ripping of the tissue. Also, if too great of closure force is applied, the tissue pressure is too great and tissue popping can occur when the steam generated within the tissue while applying energy is trapped between the jaws at the high pressure until the jaws are released following the procedure. At this point, the steam within the walls of the vessel can escape at a high velocity and can damage surrounding tissue. This can result in behind-the-seal weaknesses that can later burst during times of patient recovery from anesthesia or post-operatively (due to the increase of a patient’s blood pressure meeting the particular blood pressure level within the vessel that causes failure at the weakened location of the “sealed” blood vessel).

[0028] Furthermore, the present inventor has also recognized that a problem with relying on electrical feedback to determine seal completion is that the application of pressure can be varied by the user and this variation can affect electrically-sensed system feedback and therefore system operation and control that relies on the feedback. For example, applied pressure can vary from one user to another user. Also, applied pressure from a single user can vary from one sealing operation to another sealing operation. With existing energy devices, it is often a challenge to link the use of a device, e.g., how an operator uses the device, to the feedback provided to the capital equipment (e.g., an electrical generator), such that the capital equipment can account for user variation based on the feedback. Specifically, electrical feedback is often used to determine the effect that the device is having on tissue, but assumes that the user is ensuring a ‘steady state’ of the device during this time.

[0029] In incidents of the user activating the energy device to treat tissue where contact or compressive force is required, such as for vessel sealing, a known or consistent force is desired, as to allow the systems feedback control circuit to accurately progress through various stages of an algorithm. Various algorithms used to evaluate system feedback is described in Pub. No. US 2020 / 0352618 Al titled “Reduced Thermal Margin Techniques in Combination Energy Electrosurgical System” to Batchelor, assigned to Gyrus ACMI, Inc., the contents of which are incorporated herein in their entirety by this reference.

[0030] In such cases where the user changes the contact area or pressure, variations in the electrical feedback can occur, causing incorrect decisions to be made by the control system that determines the algorithm for applying the energy. A particular case in the change of the pressure can be the rate at which a user releases pressure on the tissue after attempting tocomplete a sealing procedure. For example, a user may prematurely release pressure on the tissue before a proper seal is formed. In such cases, it is desirable for the electrosurgical system controller to notify the user that an insufficient seal has been formed or was potentially formed. However, variation in the release speed of the pressure can result in different electrical feedback. The variation can be inconsistently or improperly interpreted by the electrosurgical system controller, thereby preventing the electrosurgical system controller from issuing a warning regarding the potential for an improper seal having been formed or taking corrective action to adjust the delivered electrical energy, for example.

[0031] This issue was identified by the present inventor in a vessel sealing system, where the applied clamping electrodes, e.g., forceps jaws, were rapidly opened by the user before a proper or complete seal was formed, resulting in a ‘false positive’, where the rise in electrical impedance (by reducing the contact pressure / area with the tissue), led the system to believe that the tissue state had changed and ‘dried out’ indicating that the sealing process was nearing completion. However, the tissue was not appropriately modified and cutting such tissue would lead to a loss of hemostasis. In other words, the rise in resistance resulting from tissue drying out can be obfuscated by the rise in resistance from the drop in contact surface area with the tissue as the jaws open.

[0032] To overcome this, the present inventor recognized that if the rate of change of the resistance (AR / R) was faster than that expected of tissue, then the generator would report an alarm or other such indicator to the surgeon that an issue with the sealing process had occurred. Further description of such phenomenon and corrective action is described with reference to FIG. 6. A similar feedback regime is described in the aforementioned Pub. No. US 2020 / 0352618 Al to Batchelor.

[0033] Monitoring of the rate of change of resistance or other electrical property can overcome the challenge of fast release of tissue before a proper seal has been formed, but if the user slowly releases the contact of the tissue with the device, the resistance rise could mimic that of appropriate energy application and result in a false positive indication that appropriate energy has been applied and that the user can move to the next step of their process.

[0034] To overcome this, the present disclosure provides a tissue treatment system that can determine if a tissue treatment event, e.g., a tissue sealing event, has occurred based on visualization feedback, such as using images generated by an imaging sensor or camera, in addition to electrical feedback, rather than just relying on the electrical feedback alone. As such, system output can be more accurately adjusted, and output can be provided to a userregarding the potential of an improper or inadequate surgical event occurring. Further description of such phenomenon is described immediately below and with reference to FIG. 7.

[0035] The system of the present disclosure can use information from one or more images to help determine the state of the device in use and whether the electrical feedback data is consistent with a change in the application device or tissue state. For example, images of jaws being positioned around tissue can be compared to electrical feedback such as a resistance signal to determine if the resistance signal is being generated when the jaws are closed. Changes in resistance signals when the jaws are closed can be more indicative of the condition of the tissue being treated, e.g., sealed. If the images of the jaws show that the jaws are opened or opening while the resistance signal is changing can be indicative of the resistance being changed by the opening of the jaws, rather than the physiological state of the tissue.

[0036] The visualization feedback can utilize a tissue image generated by an imaging sensor such as a camera. In an example, the system can use information from the image to help determine changes in resistance based upon the position and movement of a medical device, such as jaws of a forceps. In examples, the relative position between two jaws of a forceps can be determined to establish a baseline position and an associated resistance. In examples, the relative position between the two jaws can be compared over time for comparison to changes in resistance over the same time period. In examples, images of the jaws can be monitored for movements of the two jaws to determine when opening or closing is occurring. The rate of change of the positions of the jaws can be determined to determine “fast” and “slow” openings and closings for comparison to changes in the electrical resistance.

[0037] This can inform the electrosurgical system that the change in resistance is not due to the tissue state and that a feedback signal for controlling the delivered energy can be altered so as to not rely solely on changes in resistance to provide the feedback. In doing this, a slow opening or movement of the application device that is consistent with electrical feedback that might indicate the desired tissue modification had been achieved can be identified as an undesirable event by a control system and can react accordingly, such as by issuing indicia to a user or changing system operation.

[0038] This reaction by the system, e.g., the feedback signal or output provided, could be any one or more of the following: to inform the user that an error has occurred, to change the energy application algorithm to accommodate the change, to indicate to the user to reposition the device, to request the user to ‘start over’ with a different technique, to inform the user thata partial stage has been achieved with the energy application, to disable some or all system capabilities, and to suggest correct use of the device.

[0039] The system feedback and output can be provided via any one or more of the following: visual output signals, auditory output signals, written output signals, iconic output signals, and functional restriction output signals. An example of a functional restriction could be preventing a vessel sealing device from being able to cut, if the system detects anomalies with the device use, even though the electrical requirements had been met.

[0040] Similar functionality can be applied in other energy devices, such as cutting devices and ablation devices that utilize needles to create lesions using resistance feedback as an active controller of the energy delivery. These systems can additionally combine electrical, e.g., resistance feedback with imaging feedback to provide operator feedback or system modifications, such as disabling other portions of a procedure from being performed.

[0041] FIG. 1A, FIG. IB, FIG. 1C, FIG. ID, and FIG. IE illustrate an example of portions of electrosurgical system 100. Electrosurgical system 100 can include or use medical device 102 having end effector assembly 104. As described herein, electrosurgical system 100 can be configured to deliver a tailored energy output to a variety of devices. In the illustrative example, electrosurgical system 100 can provide tailored waveforms to one or more electrodes, such as active electrode 111 (FIG. ID), which can then continue to electrode 113 (FIG. ID). Such tailored energy output can be used to treat the tissue, such as to seal, cut, ablate, fulgurate or desiccate, among other effects. In some examples, end effector assembly 104 can include both of active electrode 111 and return electrode 113, however, the device does not have to be a forceps or be bipolar as shown in the illustrative example. Waveforms delivered by processing unit 122 can be tailored for monopolar and other types of devices as well. For example, an active electrode of a device can be used in conjunction with a remote return electrode, such as, but not limited to a return electrode pad that can be placed, for example, on the body of the patient.

[0042] Furthermore, the tailored energy outputs described herein can be used in systems that deliver any type of energy output that is compatible with a particular end effector or device. In merely a few non-limiting examples provided for the sake of clarity, tailoring an energy output can include tailoring the ultrasonic energy or radiofrequency energy delivered by to / by an ultrasonic forceps, or tailoring the thermal energy delivered to / by forceps or other device for treatments such as cutting, sealing, coagulation, ablation, desiccation, fulguration, and the like.

[0043] End effector assembly 104 can include or use jaw assembly 106. Alternatively, or additionally, end effector assembly 104 can include or use a “J-shaped Hook” type electrode or other electrode types for surgery. Jaw assembly 106 can include first jaw member 108 and second jaw member 110. Second jaw member 110 can be pivotably coupled to first jaw member 108 about a pivot axis or point at pivot point 112 located on or distal of shaft 116. Drives 119 can be included or used in medical device 102, such as housed within handpiece 118 and mechanically coupled to end effector assembly 104. Also, drives 119 can be included at or near the distal end of end effector assembly 104 such directly driving the jaw assembly 106. Drives 119 can be any suitable drive associated or coupled with the end effector in any arrangement, including, e.g., robotic applications. End effector assembly 104 can include cutting device 130 (FIG. IB). In examples, cutting device 130 can comprise a knife, a razor or another type of blade or saw configured to cut, slice or incise tissue positioned between first jaw member 108 and second jaw member 110. Cutting device 130 can be connected to a drive of drives 119 or another drive, such as a motor or linear actuator to move cutting device 130 forward and backward, e.g., distally and proximally between first jaw member 108 and second jaw member 110. In operation, cutting device 130 can be retracted into shaft 116 while first jaw member 108 and second jaw member 110 engage tissue and perform a sealing operation, for example. After the sealing operation, a user of medical device 102 can press an activator, such as trigger 132 (FIG. 1 A) on handpiece 118, to cause cutting device 130 to move distally in between first jaw member 108 and second jaw member 110, thereby slicing through tissue held in place by first jaw member 108 and second jaw member 110. After the slicing operation, cutting device 130 can be retracted back into shaft 116 out of first jaw member 108 and second jaw member 110.

[0044] Electrosurgical system 100 can also include or use sensors 114 for determining a tissue characteristic. For example, sensors 114 can comprise sensors for sensing an electrical parameter or imaging sensors for sensing a video parameter. Sensors 114 can include an electrical sensor for measuring one or more electrical properties such as electrical properties of tissue, e.g., resistance, capacitance, or inductance. In examples, sensors 114 can include or comprise a resistance sensor. More specifically, sensors 114 can also include an electrical sensor or electrode, such as for providing electrical characterization (e.g., resistance) of tissue compressed between first jaw member 108 and second jaw member 110, either during or after or interleaved with application of electrosurgical treatment energy to the tissue. Sensors 114 can be located in the end effector assembly 104, for example, a component of one or more of first jaw member 108, second jaw member 110, or the tissue sealingplates 109, which in this example can include active electrode 111 and return electrode 113. As depicted in FIG. 1 A, sensors 114 can be integrated at or near the end effector assembly 104. In examples, sensors 114 can be separate from the end effector assembly, such as extending from another shaft of medical device 102, a shaft of another instrument, or such a shaft as extending from another device such as a robotic arm or videoscope. In examples, sensors 114 can comprise one or more imaging sensors, e.g., cameras, located at or near the distal end of the end effector assembly, on a jaw of the jaw assembly 106, or integrated with an end effector assembly 104, such as a J-hook (as depicted in FIG. IE). However, imaging sensors or cameras can be attached to other portions of medical device 102, as discussed below.

[0045] As shown in FIG. IB and FIG. 1C, first jaw member 108 and second jaw member 110 of jaw assembly 106 can be movable between a first position “a,” in which first jaw member 108 and second jaw member 110 are spaced apart from each other and a second position “b,” in which first jaw member 108 and second jaw member 110 are positioned closer to each other than in the first position “a.” As depicted in FIG. ID, end effector assembly 104 can grasp target object 124, such as a blood vessel or other target object, such as a part of a body, an anatomical feature, tissue, veins, arteries, or a combination thereof of a human or animal subject. In an example, end effector assembly 104 can be used in electrosurgical system 100 such as to compress one or more of lymphatics, tissue pedicles, arteries, and veins, such as with a diameter or similar cross-sectional dimension ranging from about 0.5mm to about 7mm. Herein, a diameter of a vessel can refer to either of a measured diameter or an average diameter along a length of interest of a vessel of interest. In another example, end effector assembly 104 can be used in electrosurgical system 100 such as to compress lymphatics or arteries with a diameter or other similar cross-sectional dimension greater than 7mm. At least one of first jaw member 108 and second jaw member 110 can include electrodes that can be adapted such as to be electrically connected to an electrosurgical energy source, such as to provide current that can be passed through the electrodes of end effector assembly 104. For example, a therapy current can be passed from first jaw member 108 to second jaw member 110 when tissue is located within the jaw and the therapy current can coagulate blood, cauterize, cut, or a combination thereof. End effector assembly 104 can generally include one or more working assemblies, such as pairs of jaws, and sufficient controls to work the one or more working assemblies. End effector assembly 104 can include parts employed to perform the recited functions and can include shaft 116 or another elongated or other shaped shaft (e.g., a tubular member, a hollow tube, or an assembly of tubes), handpiece 118, one ormore operable mechanisms used to manipulate shaft 116 or to actuate end effector assembly 104, such as actuator 120, or a combination thereof. Handpiece 118 can be an assembly of parts or housing structures that can be assembled to form handpiece 118 structure with a cavity. In an example, shaft 116 and end effector assembly 104 can be included with or mounted to the end of a robotic arm such as to permit robotic stabilization, positioning, manipulation, and operation, instead of being hand-held by a user grasping handpiece 118.

[0046] As depicted in FIG. IE, cameras 117 can be included in or on medical device 102. Cameras 117 can include one or more imaging sensors capable of illuminating and capturing a tissue image, such as using a focal plane array (FPA) imaging sensor array of pixels. Cameras 117 can be imaging sensors located at the distal end of the end effector assembly 104 on shaft extension 128. However, cameras 117 can be positioned further proximally along shaft 116 so that objects or tissue between first jaw member 108 and second jaw member 110 can be more readily viewed. In examples, cameras 117 can comprise one or more cameras that can be integrated at or near a jaw assembly 106 or a J-hook (as shown). Cameras 117 can be mounted on shaft extension 128 to view distally, laterally or proximally. One or more light sources 115 can be included in electrosurgical system 100, such as near cameras 117, such as for illumination of the surgical site. Where cameras 117 are integrated into medical device 102, manipulation of the device can simultaneously or concurrently position both of cameras 117 and the jaw members. Medical device 102 can include one or more features, such as the shape of first jaw member 108 and second jaw member 110, fiducial markers, reflective markers and the like, that can be recognized by cameras 117 to facilitate recognition of one or more features in the image. Also, cameras 117 can be accompanied by an optical fiber bundle or other illumination optics such as to communicate light from an external light source to an internal target, and can include one or more of a spectroscopic imaging or analysis sensor, a hyperspectral imaging sensor, a colorimetric imaging sensor, a video camera sensor, an infrared imaging sensor, an ultrasound imaging sensor, a 3D imaging sensor, a LIDAR imaging sensor, an optical coherence tomography (OCT) imaging sensor, a focal plane array (FPA) imaging sensor, or a fluorescence or other shifted-wavelength response imaging sensor.

[0047] Cameras 117 can be communicatively coupled to processing unit 122. Processing unit 122 can receive data from cameras 117 and, based thereon, can determine a location, relative location, position, movement, rate of change of position of first jaw member 108 and second jaw member 110. In an example, processing unit 122 can use a trained learning model, an algorithmic signal processing approach, or both, such as to recognize or identifyone or more device-recognizable features of the jaws in the tissue image such as one or more shapes, patterns, colors. These features can be cross-referenced to a database (e.g., database 410 of FIG. 9) of image feature data, in an example. Processing unit 122 can further manipulate the image, the features, or both, such as changing from color to greyscale or monochrome or segmenting or outlining of certain features, such as to help improve feature recognition. In examples, depth-of-field analysis circuitry can be included or used by processing unit 122 such as to help obtain representations of 3D features of the tissue.Further description of a trained learning model or other image signal processing of the images to recognize one or more features and determine a state of medical device 102 is discussed in greater detail below with reference to FIG. 9.

[0048] As discussed with reference to FIG. 4A to FIG. 4C, processing unit 122 can use the data from cameras 117 to measure or estimate a position or location of first jaw member 108 and second jaw member 110. For example, device-recognizable features from the specimen image can be used by processing unit 122 such as to measure or estimate these parameters. Also, a pivot angle of first jaw member 108 and the second jaw member 110 can be measured and used such as to help estimate the position and rate of change of position. In an example, the position or rate of change of position of the jaws can be determined by processing unit 122 and can function as threshold inputs for user feedback. For instance, a user of end effector assembly 104 can receive visual or audible or haptic or other feedback from processing unit 122 upon meeting or exceeding the determined rate of change. Also, the user can receive visual or audible or haptic or other feedback when the rate of change exceeds of falls below a predetermined range. In other examples, processing unit 122 can determine the position of the jaws, relative position of the jaws or rate of change of the position of the jaws and use any or all of these parameters to establish, adjust, or modulate the energy waveform such as to help compensate for, e.g., too fast or too slow closing or opening of the jaws.

[0049] There are many different tissue materials that can be activated upon and cut. Such tissues can include fat, mesentery, ligaments, or the like. Such tissues can be activated upon using a different therapy output scheme than what is used for vessel sealing. Thus, the present techniques are not just limited to use in vessel sealing devices, but can be applied elsewhere, including for monopolar and bipolar cutting devices, such as those that are intended to cut tissue hemostatically.

[0050] The present system can include or be coupled to an endoscope or other visualization device, such as can provide illumination to a patient-internal tissue site, a photodetector or multi-pixel imaging array such as to transduce response light from the patient internal tissuesite for optical imaging. The system can further include electrical signal processing circuitry. This can include imaging processing circuitry, such as which can be used to process the transduced signals into an electrical representation of the optical image.

[0051] Processing unit 122 can include or use processing circuitry. Processing unit 122 is not limited to a single unit or circuit. Processing unit 122 can include multiple sources of processing, and some may be physically separate or remotely located from the device. In an example, processing unit 122 can include or use memory circuitry comprising instructions that, when executed by the processing circuitry of processing unit 122, can cause processing unit 122 to perform an inspection operation using sensors 114 and cameras 117. The processing circuitry can include or use image-processing circuitry configured such as to determine a position and rate of change of position of first jaw member 108 and the second jaw member 110 using tissue image from cameras 117 and a trained machine learning model, and compare the position of the jaws to electrical feedback obtained from sensors 114. Further description of the configurations of processing unit 122 are provided with reference to FIG. 3 and FIG. 10.

[0052] FIG. 2 illustrates an example of an electrosurgical system in use by a surgeon. The system can include or use the one or more of sensors 114 and cameras 117. Sensors 114 can be resistance sensors configured to sense resistance between first jaw member 108 and second jaw member 110. Cameras 117 can be capable of capturing a tissue image, and the tissue image can include device-recognizable features of the medical instrument. For example, the camera can a provide real time, inline, or otherwise concurrent video monitoring feed for the surgeon and can use frames of the feed to capture the device-recognizable features of the medical instrument. The video monitoring feed can be viewed on display 126 during a procedure. Display 126 can comprise a video monitor, a television screen, a liquid crystal display, a light emitting diode (LED) display and the like. The video monitoring feed can include or use one or more visual or tactile or audible indicators such as to provide feedback to the surgeon during the procedure. The indicators can provide feedback to the surgeon such as to help the surgeon determine if corrective actions are desired. Further description of feedback that can be provided with the present disclosure can be found with reference to FIG. 3. Additionally, or alternatively, the device-recognizable features can be used by the processing unit 122 to help automatically apply the therapy energy.

[0053] FIG. 3 shows a block diagram of system 150 for performing one or more of the methods, procedures or operations discussed herein. System 150 can comprise controller 152, camera unit 154 and energy application device 156. Controller 152 can comprise centralprocessing unit, e.g., CPU 158, image processor 160 and energy generator source 162. In examples, controller 152 can comprise or include processing unit 122.

[0054] Controller 152 can comprise a machine configured to receive imaging inputs from camera unit 154 and electrical feedback from energy application device 156 to perform analysis thereof to determine the effectiveness of a tissue sealing operation being performed by energy application device 156 and provide output for a user. Controller 152 can comprise capital equipment used in an operating room, such as a power supply, electrical generator, imaging unit, illumination unit and the like. Controller 152 can include CPU 158, which can be configured to communicate with image processor 160 and energy generator source 162. CPU can comprise a component of a larger machine configured to allow CPU 158, image processor 160 and energy generator source 162 to operate together. Further details of controller 152 are described with reference to FIG. 10.

[0055] Camera unit 154 can comprise an imaging device, image processor and the like. Camera unit 154 can be used to obtain images of anatomy and energy application device 156 within a patient. In examples, camera unit 154 can comprise a charge-coupled device (CCD), a solid state device such as a complementary metal oxide semiconductor (CMOS), a focal plane array (FPA) imaging sensor, another imaging sensor or camera described herein or another device known in the art.

[0056] Energy application device 156 can comprise a device suitable for delivering energy to tissue to perform a medical procedure, such as a therapeutic procedure or diagnostic procedure. Energy application device 156 can comprise a device configured for delivering radio frequency (RF) energy, electromagnetic energy, plasma energy, ultrasound energy and the like. In examples, energy application device 156 can comprise jaw assembly 106, including first jaw member 108, second jaw member 110, active electrode 111, return electrode 113 and sensors 114.

[0057] In operation, camera unit 154 and energy application device 156 can be inserted into anatomy of a patient. Jaws of energy application device 156 can be positioned around tissue and can be actuated to apply force to the tissue. Energy application device 156 can be activated to apply energy to the jaws to seal the tissue. For example, a user can push a button or the like on handpiece 118 to provide activation energy from energy generator source 162 to energy application device 156. Thus, energy generator source 162 can provide energy transfer to energy application device 156. Simultaneously, camera unit 154 can observe energy application device 156 and the tissue located therein. Camera unit 154 can provide imaging data to image processor 160. Image processor 160 can render the signal fromcamera unit 154 as an image on display 126. Image processor 160 can additionally identify first jaw member 108 and second jaw member 110 in the images and determine an operative state thereof, e.g., open, closed or intermediate states. Image processor 160 can additionally convey the signal from camera unit 154 to CPU 158. CPU 158 can additionally receive input from energy application device 156. For example, energy application device 156 can provide measurements from one or more sensors associated with energy application device 156 to energy generator source 162, which can subsequently relay the measurements to CPU 158. As such, CPU 158 can receive imaging signals from camera unit 154 and sensor signals from energy application device 156, which can be correlated along a common timeline. As such, changes in sensor output that can be used by CPU 158 to determine if a proper tissue seal has been formed can be compared to images of first jaw member 108, second jaw member 110 and target object 124 to determine if first jaw member 108 and second jaw member 110 were clamped down on target object 124 when sensor output from energy application device 156 indicated a proper seal had been formed. FIG. 4 A - FIG. 4C illustrate stages of first jaw member 108 and second jaw member 110 clamping down on target object 124.

[0058] Controller 152 can comprise output device 170 for providing output and human- perceptible feedback. Output device 170 can comprise display 126, haptic feedback device 172 and audio driver 174. Display 126, also shown in FIG. 2, can comprise indicia 176A through indicia 176E and dial 178. In other examples, dial 178 can comprise a mechanical dial responsive to output of controller 152. Output device 170 can comprise, or can be in communication with, CPU 158, which can also be in communication with sensors 114 and cameras 117 via image processor 160 and energy generator source 162. In examples, sensors 114 and cameras 117 can be in direct communication with output device 170. In examples, output device 170 can be located on or in processing unit 122 (FIG. 1 A).

[0059] Display 126 can comprise an active display unit, such as a liquid crystal display, a plasma screen, an organic light-emitting diode display and the like. Display 126 can comprise a touchscreen device. Display 126 can be programmed to provide a variety of outputs and receive a variety of user-inputs. CPU 158 can be connected to a memory device, such as main memory 604 and static memory 606 of FIG. 10, and can include information related to electrical parameters sensed by sensors 114, such as changes in sensed electrical parameters that can correspond to changes in the status of therapy procedures being performed on different tissue types with different energy types, baseline or reference information for the electrical parameters, and indicia and instructions to be displayed on display 126 if the sensed electrical parameters do not sufficiently correspond to the baselineor reference information, such as warnings that a procedure has or may not have been completed properly and the like. In examples, display 126 can show various textual messages, such as “Warning! An improper surgical operation may have been performed. Re- execute the surgical operation using a different location and / or technique.” In examples, display 126 can provide confirmation that a proper seal has been performed, such as “The completed surgical operation has passed performance checks.” Thus, CPU 158 can comprise signal-processing circuitry that can receive an input from sensors 114 and cameras 117, consult a lookup table stored in memory to find output indicia for the corresponding output of sensors 114 and cameras 117 and display on display 126 or provide another feedback output relating to the adequacy of a procedure being performed and the like. Activation of at least one of indicia 176A through indicia 176E, as well as haptic feedback device 172, audio driver 174 and dial 178, can provide an indication of a confidence level that the procedure being performed has progressed or completed properly, as can be determined by a predictive engine, e.g., Al or ML system, for example, as discussed with reference to FIG. 9.

[0060] In examples, each of indicia 176A through indicia 176E can be activated to indicate a confidence level in a procedure that is being performed has progressed or completed properly. Each of indicia 176A through indicia 176E can comprise a light emitting diode. In examples, indicia 176E at the bottom of output device 170 and indicia 176 A at the top of output device 170 can be activated in opposite manners to indicate opposite ends of a confidence spectrum. Thus, indicia 176E can be activated to show a first level of a confidence. Indicia 176B, indicia 176C and indicia 176D can be activated to indicate varying levels in between the first and second levels such that a continuous spectrum or a gradual changing of light emitting activation can be provided. Indicia 176A through indicia 176E can update in real-time to indicate the confidence in the procedure being performed. Thus, as a surgeon manipulates medical device 102 (FIG.1A) to perform a medical procedure, an indication in the confidence level of the adequacy of the procedure can be provided. In other examples, all of indicia 176A through indicia 176E can be activated, or lit up, and can change colors to indicate the magnitude of the electrical parameter. For example, lighter colors can be used to indicates lower confidence and darker colors can be used to indicate higher confidence. In examples, indicia 176A through indicia 176E and dial 178 can be provided with labels to translate the confidence into actionable feedback for a user.

[0061] In an example, display 126 can include dial 178. Dial 178 can include a scale to indicate different confidence levels. A mechanical or virtual needle can be moved relative toa scale on dial 178 to indicate different levels of confidence. One end of the scale can indicate low confidence and the opposite end of the scale can indicate high confidence.

[0062] In examples, an audible alarm can be used to provide feedback indicating different confidence levels, such as by using audio driver 174. For example, a steady signal can be emitted that changes pitch, volume or tone based on the confidence level. In other examples, an intermittent signal can be emitted that changes intervals based on the confidence level. Thus, low pitch or low volume audible alarms can indicate high confidence and high pitch or high volume audible alarms can indicate low confidence. In examples, no audible alarm can represent high confidence and audible alarm can indicate lower confidence, with high pitch or volume of the audible alarm indicating even lower confidence.

[0063] In examples, a tactile alarm can be used to provide feedback indicating the confidence level, such as by using haptic feedback device 172. For example, a steady vibration can be emitted that changes speed, e.g., frequency, based on the confidence level. In other examples, an intermittent signal can be emitted that changes intervals based on the confidence level. Thus, low speed vibrations or low frequency vibrations can indicate high confidence and high speed vibrations or high frequency vibrations can indicate low confidence. In examples, no vibration can represent high confidence and vibration can indicate lower confidence, with high speed or frequency of the vibration indicating even lower confidence.

[0064] FIG. 4A is a schematic diagram of end effector assembly 104 engaging target object 124. End effector assembly 104 can comprise the same or similar end effector shown in FIG. ID. End effector assembly 104 can comprise jaw assembly 106, which can include first jaw member 108 and second jaw member 110. Second jaw member 110 can be pivotably coupled to first jaw member 108 about pivot point 112. First jaw member 108 can include active electrode 111 and second jaw member 110 can include return electrode 113. First jaw member 108 and second jaw member 110 can include sensors 114. Target object 124 can comprise a blood vessel having tissue surrounding an internal lumen, or another anatomic structure.

[0065] End effector assembly 104 can further comprise imaging device 140, which can comprise an embodiment of cameras 117, camera unit 154 (FIG. 3) or an alternative or addition thereto. Imaging device 140 can be positioned proximally of first jaw member 108 and second jaw member 110 such that target object 124 positioned therebetween can be viewed. Imaging device 140 can have field of view 142 in which first jaw member 108, second jaw member 110 and target object 124 can be located. Imaging device 140 can haveobjective lens 144 that can be angled toward first jaw member 108 and second jaw member 110. Objective lens 144 can comprise a fisheye lens, e.g., a lens having a one-hundred- eighty-degree field of view, to allow imaging device 140 to capture images laterally to imaging device 140.

[0066] First jaw member 108 and second jaw member 110 can be opened to allow target object 124 to be fit therebetween. FIG. 4A shows the tip of first jaw member 108 spaced from the tip of second jaw member 110 distance DI. First jaw member 108 and second jaw member 110 can include marker 146 and marker 148, respectively, that can be recognized by imaging device 140. In examples, marker 146 and marker 148 can comprise indicia or physical features that can be tracked by image processor 160 (FIG. 3) in images from imaging device 140. In examples, marker 146 and marker 148 can comprise fiducial markers, bar codes, quick response (QR) codes and the like. Marker 146 and marker 148 can thus be used to determine the open or closed state of first jaw member 108 and second jaw member 110. As shown in FIG. 4B, first jaw member 108 and second jaw member 110 can be rotated to apply pressure to target object 124. As discussed herein, the ability of active electrode 111 and return electrode 113 to seal target object 124 can depend on the amount of pressure that first jaw member 108 and second jaw member 110 apply to target object 124 and the length of time that the pressure is applied to target object 124.

[0067] FIG. 4B is a schematic diagram of end effector assembly 104 of FIG. 4A applying pressure to target object 124. As mentioned, in order to seal target object, it is desirable to apply pressure to target object 124 with first jaw member 108 and second jaw member 110. As shown, first jaw member 108 and second jaw member 110 can squeeze target object 124 so that walls of the vessel collapse down against each other. As such, when activation energy is applied by active electrode 111 and return electrode 113, target object 124 can be sealed to prevent blood flow therethrough. For example, walls of the vessel can become “glued” together as discussed previously. FIG. 4B shows the tip of first jaw member 108 spaced from the tip of second jaw member 110 distance D2, which is less than distance DI of FIG. 4A. Description of a proper or adequately formed seal is described further with reference to FIG. 5.

[0068] FIG. 4C is a schematic diagram of end effector assembly 104 of FIG. 4B releasing target object 124. First jaw member 108 and second jaw member 110 can be rotated away from each other to separate from target object 124. FIG. 4C shows first jaw member 108 and second jaw member 110 returned to the opened position of FIG. 4 A where the tip of first jaw member 108 is spaced from the tip of second jaw member 110 distance DI. As shown inFIG. 4C, target object 124 is permanently deformed in a sealed state where walls of the vessel remain in contact with each other. Additionally, the tissue is dried out, which prevents blood from flowing through the tissue itself. As such, a proper seal is formed on target object 124 after first jaw member 108 and second jaw member 110 apply activation energy to target object 124 for a sufficiently long time at sufficient pressure. However, the shape alone of target object 124 cannot be used to determine if a proper seal is formed. If insufficient pressure is applied or pressure is not applied for a sufficiently long time, the tissue may not properly deform or may not thoroughly dry out despite possibly looking like a proper seal has been formed. Thus, if a sealing process moves from the state of FIG. 4B to the state of FIG. 4C prematurely, an improper seal can be formed. If when forming the improper seal, first jaw member 108 and second jaw member 110 are moved rapidly apart, controller 152 can analyze output of sensors 114 to determine an excessive rise in sensed resistance to determine that first jaw member 108 and second jaw member 110 were opened prematurely, as discussed with reference to FIG. 6. If when forming the improper seal, first jaw member 108 and second jaw member 110 are moved slowly apart, controller 152 can analyze output of imaging device 140 and / or camera unit 154 (FIG. 3) to determine that first jaw member 108 and second jaw member 110 were opened prematurely, simultaneously with a rise in resistance indicated by sensors 114, as discussed with reference to FIG. 7.

[0069] Image processor 160 (FIG. 3) can be configured to analyze images from imaging device 140 to determine the position of first jaw member 108 and second jaw member 110. For example, image processor 160 can determine changes between distance DI and distance D2 including intermediate distances therebetween. Image processor 160 can track marker 146 and marker 148 to determine position, change in position, rate of change of position and the like between first jaw member 108 and second jaw member 110. Additionally, image processor 160 can monitor and track changes in angle AA to determine a relative position between first jaw member 108 and second jaw member 110.

[0070] When CPU 158 receives electrical feedback indicating that a surgical event has occurred, e.g., a sealing task has been completed, CPU 158 can look at output of image processor 160 to see if first jaw member 108 and second jaw member 110 are at distances close to DI to indicate that first jaw member 108 and second jaw member 110 were properly engaged with target object 124 when the electrical feedback was received. This can be used to verify a proper completion of the task. Thus, the medical procedure can proceed to the next step or to completion.

[0071] When CPU 158 receives electrical feedback indicating that a surgical event has occurred, e.g., a sealing task has been completed, and CPU 158 sees that first jaw member 108 and second jaw member 110 are at distances closer to D2 to indicate that first jaw member 108 and second jaw member 110 were not properly engaged with target object 124 when the electrical feedback was received, this can be indicative of incompletion of the surgical task. Thus, CPU 158 can take interventional action, such as by presenting audio, visual, and / or tactile feedback to a user and / or adjusting output of the electrosurgical system or disabling an aspect of the electrosurgical system.

[0072] FIG. 5 illustrates chart 200 having plot 202 of pressure P applied by a forceps, such as end effector assembly 104 of FIG. ID, versus plot 204 of resistance R sensed between first jaw member 108 and second jaw member 110 over time using, for example, sensors 114. Thus, x-axis 206 can indicate time T, while y-axis 208 can indicate resistance R and pressure P. The relative magnitudes of resistance R and pressure P are not necessarily to scale and are for illustrative purposes. FIG. 5 illustrates pressure P being applied at sufficient magnitude for sufficient time to achieve a proper seal, e.g., a seal where blood flow is prevented through the tissue, as can be indicated by resistance R reaching threshold resistance level 218 at a particular rate.

[0073] At some point in time 210, first jaw member 108 and second jaw member 110 can begin or continue to apply pressure to target object 124. Point in time 210 can correspond to FIG. 4B where first jaw member 108 and second jaw member 110 are closed down on target object 124. Simultaneously, active electrode 111 and return electrode 113 can begin or continue to apply electrical energy to target object 124. For example, energy generator source 162 of FIG. 3 can provide energy to energy application device 156 of FIG. 3. Pressure P can be applied generally uniformly until point in time 212 to achieve a proper seal. The magnitude of pressure P can be controlled by an operator or user of end effector assembly 104 as the user manipulates medical device 102, e.g., actuator 120. The magnitude of resistance R is controlled by various factors, such as the physiological state of the tissue and the state of end effector assembly 104, as discussed herein. The magnitude of resistance R can be determined by CPU 158 (FIG. 3) based on, for example, feedback generated by system 150. For example, the magnitude of resistance R can be provided to CPU 158 by energy application device 156. Seal time 214 comprises a length of time T1 sufficient to achieve a desirable seal of target object 124 for a given pressure P and other variables, such as tissue type. Seal time 214 can correspond to the rise in resistance R to threshold resistance level 218.

[0074] At point in time 210, plot 204 for resistance R can begin to increase at a low rate as the tissue of target object 124 begins to heat up. At point in time 216, plot 202 for resistance R can begin to increase more rapidly as the tissue of target object 124 begins to dry out. Resistance R can rapidly rise as the tissue approaches being completely dried out. Plot 204 is illustrated as a two-stage plot having two linear lines. However, plot 204 can follow one or more curved trajectories where the slope of plot 202 gradually increases at first and then increases more rapidly closer to point in time 212. Plot 204 can follow a path where the rate of change in resistance R slowly increases at first and then more rapidly increases near point in time 212. Plot 202 can extend above threshold resistance level 218 for resistance R. Threshold resistance level 218 can comprise a resistance level, e.g., a change in resistance (AR) or a rate of change of resistance (R), where system 150 (FIG. 3) can be confident a proper seal has been achieved.

[0075] After point in time 212, jaws can be opened by a user to move on to a different portion of performing or finishing a medical procedure. Point in time 212 can correspond to FIG. 4C where first jaw member 108 and second jaw member 110 are opened away from target object 124. As discussed, variability in the amount of time that a user applies pressure P and variability in application of pressure P by the user can impact the seal quality achieved at target object 124. However, after point in time 212, first jaw member 108 and second jaw member 110 can be opened at any speed by the user without affecting the seal quality. Thus, user variation in the opening of end effector assembly 104 does not impact seal quality so long as pressure has been applied by end effector assembly 104 for the duration of seal time 214, which can be indicated by resistance R obtaining a predetermined rate of change, illustrated in FIG. 5.

[0076] Magnitudes of resistance R can be sensed by sensors 114 (FIG. ID) and provided to controller 152 and controller 152 can compare the magnitude and rate of change of resistance R to values of resistances and rates of change of resistance stored in memory (e.g., main memory 604 or static memory 606 of FIG. 10) representative of tissue undergoing a proper seal for different types of tissues, different energy sources, different end effectors and the like. When the increase in the rate of change of resistance R matches predetermined values of an increase in the rate of change of resistance R that indicates a proper seal has been formed, e.g., the tissue has adequately dried out, controller 152 can determine that a proper seal has been formed.

[0077] If a user releases pressure P before point in time 210, the seal can be negatively impacted. As discussed herein, system 150 can be configured to perform analysis ofelectrical signals generated by end effector assembly 104, such as resistance R, to evaluate the formation of a seal on tissue. Furthermore, system 150 can obtain video images of end effector assembly 104 to obtain operating states of end effector assembly 104 at times when the electrical signals are generated.

[0078] FIG. 6 illustrates chart 220 having plot 222 of pressure P applied by a forceps, such as end effector assembly 104 of FIG. ID, versus plot 224 of resistance R sensed between first jaw member 108 and second jaw member 110 over time by sensors 114. Thus, x-axis 206 can indicate time, while y-axis 208 can indicate resistance R and pressure P. FIG. 6 illustrates pressure P being applied at a generally constant magnitude for a first amount of time and then being released before seal time 214 is reached, thereby potentially resulting in an insufficient seal being formed. As discussed below, pressure P is released quickly such that system 150 can detect a rapid rise in the change of resistance R and trigger an alarm to indicate to a user that an incomplete seal may have been formed.

[0079] At some point in time 230, which can be a similar starting point to point in time 210, first jaw member 108 and second jaw member 110 can begin to apply pressure to target object 124. Simultaneously, active electrode 111 and return electrode 113 can begin to apply electrical energy to target object 124. Pressure P can be applied generally uniformly until point in time 232 where the user begins to move first jaw member 108 away from second jaw member 110. Thus, premature time 234 comprises time T2 that is less than seal time Tl, thereby indicating that an improper seal may have been formed. As discussed below, system 150 (FIG. 3) can determine that an improper seal may have been formed by determining a rate of change of resistance R that is abnormally quick. From point in time 232, pressure P can drop quickly to zero where first jaw member 108 and second jaw member 110 are disengaged from target object 124.

[0080] At point in time 230, plot 224 for resistance R can begin to increase at a low rate as the tissue of target object 124 begins to heat up. At point in time 232, plot 224 for resistance R can begin to increase more rapidly as first jaw member 108 moves away from second jaw member 110. As first jaw member 108 and second jaw member 110 move away from each other, first jaw member 108 and second jaw member 110 can begin to disengage with target object 124, thereby reducing the contact surface area with target object 124. Thus, resistance R can begin to rise as the energy delivered by active electrode 111 and return electrode 113 has less surface area to pass through. Resistance R can rapidly rise until first jaw member 108 and second jaw member 110 are completely disengaged from target object 124, such as shown in FIG. 4C, thereby causing an open circuit and a spike in the resistance R to the opencircuit value. After point in time 232, resistance R in plot 224 can rise more rapidly than in plot 204. Again, plot 224 is illustrated as a two-stage plot having two linear lines. However, plot 224 can follow a curved trajectory where the slope of plot 222 gradually increases at first and then increases more rapidly closer to point in time 236.

[0081] Magnitudes of resistance R can be sensed by sensors 114 (FIG. ID) and provided to controller 152 and controller 152 can compare the magnitude and rate of change or resistance R to values of resistances and rates of change of resistance stored in memory (e.g., main memory 604 or static memory 606 of FIG. 10) representative of tissue undergoing a proper seal for different types of tissues, different energy sources, different end effectors and the like. For example, plot 204 can be stored in memory as a baseline or reference resistance plot for end effector assembly 104 and the type of tissue of target object 124. Controller 152 can determine that the rate of change of resistance in plot 224 is greater than in plot 204 and can trigger an alarm to be generated for the user or can generate a control system to enable or disable another component of system 150, as discussed below. For example, controller 152 can issue warnings, instructions and indicia as discussed with reference to output device 170 of FIG. 3.

[0082] FIG. 7 illustrates chart 240 having plot 242 of pressure P applied by a forceps, such as end effector assembly 104 of FIG. ID, versus plot 244 of resistance R sensed between first jaw member 108 and second jaw member 110 over time by sensors 114 (FIG. ID). Thus, x- axis 206 can indicate time, while y-axis 208 can indicate resistance R and pressure P. FIG. 7 illustrates pressure P being applied at a generally constant magnitude for a first amount of time and then being released before seal time 214 is reached, thereby potentially resulting in an insufficient seal being formed. As discussed below, pressure P is released slowly such that system 150 cannot readily detect a difference in the rate of change of resistance R from a baseline condition, such as that of FIG. 5. Thus, system 150 may not trigger an alarm to indicate to a user that an incomplete seal may have been formed. With the present disclosure, system 150 can utilize a camera or imaging sensor, e.g., cameras 117 of FIG. IE or imaging device 140 of FIG. 4 A, to determine the relative positions of first jaw member 108 and second jaw member 110 to evaluate if pressure has been removed from target object 124 prematurely, e.g., before the rate of change of resistance R has properly increased.

[0083] At some point in time 250, which can be a similar starting point to point in time 210, first jaw member 108 and second jaw member 110 can begin to apply pressure to target object 124. Simultaneously, active electrode 111 and return electrode 113 can begin to apply electrical energy to target object 124. Pressure P can be applied generally uniformly untilpoint in time 252 where the user begins to move first jaw member 108 away from second jaw member 110. Thus, premature time 254 comprises time T3 that is less than seal time Tl, thereby indicating that an improper seal may have been formed. Note, premature time 254 can be similar to premature time 234 of FIG. 6. As discussed below, system 150 (FIG. 3) can determine that an improper seal may have been formed by determining a rate of change of resistance R that is abnormally quick. From point in time 252, pressure P can drop quickly to zero where first jaw member 108 and second jaw member 110 are disengaged from target object 124.

[0084] At point in time 250, plot 244 for resistance R can begin to increase at a low rate as the tissue of target object 124 begins to heat up. At point in time 252, plot 224 for resistance R can begin to increase more rapidly as first jaw member 108 moves away from second jaw member 110. As first jaw member 108 and second jaw member 110 move away from each other, first jaw member 108 and second jaw member 110 can begin to disengage with target object 124, thereby reducing the contact surface area with target object 124. Thus, resistance R can begin to rise as the energy delivered by active electrode 111 and return electrode 113 has less surface area to pass through. However, resistance R will rise more slowly than the scenario of FIG. 6 associated with plot 224 due to first jaw member 108 and second jaw member 110 moving slowly. The increase in the rate of change of R from first jaw member 108 and second jaw member 110 opening can be offset by first jaw member 108 and second jaw member 110 opening more slowly than what occurs in FIG. 6. In other words, the increase in the rate of change of resistance R from the jaws opening can be masked by the jaws opening slowly, thereby mimicking a rate of change in resistance for tissue being properly sealed. Thus, after point in time 252, resistance R in plot 244 can rise in a similar fashion to plot 204. System 150 may not, therefore reliably issue an alarm to the user that first jaw member 108 and second jaw member 110 have been opened prematurely.

[0085] Again, plot 244 is illustrated as a two-stage plot having two linear lines. However, plot 244 can follow a curved trajectory where the slope of plot 244 gradually increases at first and then increases more rapidly closer to point in time 256. Plot 244 can continue to rise until first jaw member 108 and second jaw member are completely disengaged from target object 124, such as shown in FIG. 4C, thereby causing an open circuit and a spike in the resistance R to the open circuit value. Such an event can take place before or after seal time 214 is reached, but at that point is irrelevant or not useful to trigger an alarm because first jaw member 108 and second jaw member 110 have already been prematurely opened.

[0086] Magnitudes of resistance R can be sensed by sensors 114 (FIG. ID) and provided to controller 152 and controller 152 can compare the magnitude and rate of change or resistance R to values of resistances and rates of change of resistance stored in memory (e.g., main memory 604 or static memory 606 of FIG. 10) representative of tissue undergoing a proper seal for different types of tissues, different energy sources, different end effectors and the like. For example, plot 204 can be stored in memory as a baseline or reference resistance plot for end effector assembly 104 and the type of tissue of target object 124. As mentioned, controller 152 can be unable to distinguish plot 244 from plot 204 due to similarities in the rate of change of resistance. As such, controller 152 can compare images from cameras 117 (FIG. ID) to resistance or rate of change of resistance values to determine if first jaw member 108 and second jaw member 110 have been opened prematurely to trigger an alarm to be generated for the user or to generate a control system to enable or disable another component of system 150. For example, if the increase in the rate of change R indicating a proper seal has been formed occurs when images of first jaw member 108 and second jaw member 110 are close together, system 150 can confirm a proper seal. However, if the increase in the rate of change R indicating a proper seal has been formed occurs when images of first jaw member 108 and second jaw member 110 are apart from each other, system 150 can take corrective actions, such as disabling cutting device 130 (FIG. IB) or providing warning about an improper seal or instructions for correcting the improper seal on display 126, as discussed herein.

[0087] FIG. 8A is a block diagram of method 300 of determining if a proper surgical operation has been performed and issuing feedback and control signals if it is determined that an improper surgical operation may have been performed. In examples, method 300 of treating tissue can be performed using any one or more of the various electrosurgical systems described herein, including electrosurgical forceps used to seal tissue and needles used to produce lesions. Method 300 is described with reference to operation 302 to operation 310. Method 300 can additionally include fewer or greater operations other than operation 302 to operation 310. Additionally, in other examples, operation 302 through operation 310 can be performed in other sequences.

[0088] At operation 302, visual capture systems can monitor a device status. The video capture system can include camera unit 154 of FIG. 3 for obtaining video imaging of end effectors of medical devices, such as end effector assembly 104. Camera unit 154 can include, for example, one or more of cameras 117 (FIG. ID) and imaging device 140 (FIG. 4A). In examples, the visual capture system can determine a status or operational state of anend effector, such as engaged with tissue or not engaged with tissue, activated or not activated, opened or closed, extended or retracted, curved or straightened. For example, the visual capture system can monitor the relative locations of first jaw member 108 and second jaw member 110. Output of camera unit 154 can be provided to image processor 160 of FIG. 3 where controller 152 can interpret the video images to determine a state or status of the end effector using, in examples, a machine learning model or artificial intelligence engine to facilitate interpretation of the images.

[0089] At operation 304, feedback can be provided to indicate that output of the medical device is completed. For example, an energy input, such as electricity, ultrasonic energy, plasma, vibration, etc., can be provided to the end effector, e.g., forceps or needle, to perform an electrosurgical operation. Sensors, such as resistance sensors, temperature sensors, pressure sensors and the like, can be included in the end effector to obtain electronic feedback in the form of an electrical signal. For example, sensors 114 can be used to obtain electrical information from first jaw member 108 and second jaw member 110. In examples, resistance between active electrode 111 and return electrode 113 can be provided to energy generator source 162 (FIG. 3), which can convey the resistance signal to CPU 158. CPU 158 can interpret the feedback, in the form of an electrical signal, to determine a magnitude of the sensed parameter and a rate of change of the sensed parameter. CPU 158 can determine the completion of various surgical operations by sensing and monitoring the levels of various feedback signals and comparing the obtained feedback signals to baseline or threshold feedback signals known to represent completed surgical tasks when the end effector was in a known operational state, which can be stored in non-transient computer memory devices for CPU 158. In examples, CPU 158 can utilize the machine learning model or artificial intelligence engine of operation 302 to facilitate interpretation of the electronic feedback.

[0090] At operation 306, visual feedback checks can be performed to determine if no related device changes could have caused a completion event. For example, CPU 158 can analyze output of camera unit 154 to determine changes in the end effector state at the time the electronic feedback indicated a surgical operation had completed. In examples, CPU 158 can check for steady state operation of the end effector while the electronic feedback was being obtained that indicated a change in status of the surgical operation. For example, the relative position of first jaw member 108 and second jaw member 110 can be compared in images obtained by the video capture system. The relative position of first jaw member 108 and second jaw member 110 can be compared to the sensed electrical output obtained at operation 304 at the same. If no changes in the device are detected at the time the electronic feedbackindicated a completion of a surgical operation, method 300 can proceed to operation 308. If changes in the device are detected a the time the electronic feedback indicated a completion of a surgical operation, method 300 can proceed to operation 310.

[0091] At operation 308, energy delivery by the device can be concluded or a different stage of energy delivery can be performed. Operation 308 can be completed if CPU 158 determines that no changes in the medical device were detected at the time the feedback signal indicated the surgical operation concluded. Steady state operation of the medical device can indicate that the medical device did not change or unintentionally alter a parameter being sensed by the medical device at operation 304 that produced the electronic feedback. Thus, an operator of the medical device can proceed to a different medical operation or can conclude the procedure.

[0092] At operation 310, a warning can be produced indicating incomplete output or a control signal can be produced that results in a change in operation of the device or system. Operation 310 can be completed if CPU 158 determines that changes in the medical device were detected at the time the feedback signal indicated the surgical operation concluded. Non-steady state operation of the medical device can indicate that the medical device may have change or altered a parameter being sensed by the medical device at operation 304 that produced the electronic feedback. CPU 158 can issue a textual, audio, visual or haptic feedback to alert a user of the medical device that the medical operation may not have completed successfully or as desired. CPU 158 can issue instructions for the user to double check the work previously performed or to reperform the surgical operation at a different tissue sight, e.g., adjacent the improper surgical operation. CPU 158 can additionally or alternatively issue command signals to the medical device to enable safety features, such as locks that prevent operation of a capability of the end effector, which can inhibit the capability of the operator or user of the medical device from performing another surgical operation.

[0093] FIG. 8B is a block diagram of method 350 for determining if an improper tissue sealing operation has occurred with end effector assembly 104 (FIG. ID). Method 350 is described with reference to operation 352 to operation 382. Method 350 can additionally include fewer or greater operations other than operation 352 to operation 382. Additionally, in other examples, operation 352 through operation 382 can be performed in other sequences.

[0094] At operation 352, sensors 114 can sense resistance between first jaw member 108 and second jaw member 110. Additionally, other electrical parameters can be sensed by sensors114, such as impedance and phase angle. Output of sensors 114 can be provided to CPU 158, where CPU 158 can calculate rates of change in these electrical feedback signals.

[0095] At operation 354, cameras 117 can record the relative position of first jaw member 108 and second jaw member 110. Cameras 117 can record distance DI (FIG. 4A) and distance D2 (FIG. 4B) as end effector assembly 104 is operated.

[0096] At operation 356, CPU 158 can merge the output signal of sensors 114 and the output signal of cameras 117 onto a common timeline such that changes in the electrical signals from sensors 114 can be correlated to changes in the operational state of first jaw member 108 and second jaw member 110. For example, CPU 158 can match changes output of sensors 114 to changes in output of cameras 117, if any.

[0097] At operation 358, CPU 158 can compare the rate of change of the resistance sensed by sensors 114 to a baseline rate of resistance change stored in main memory 604, for example, indicating that a tissue sealing event has properly occurred.

[0098] At operation 360, CPU 158 can determine if the sensed rate of change of resistance in the tissue upon which the sealing operation is being performed corresponds to the baseline rate of resistance stored in memory.

[0099] At operation 362, if the sensed rate of change in the tissue upon which the sealing operation is being performed does not match or is not within an acceptable tolerance band of the baseline rate of resistance change, the method can continue to operation 364.

[0100] At operation 364, CPU 158 can determine that in improper seal has been or may have been formed. In examples, CPU 158 can determine that first jaw member 108 and second jaw member 110 were opened to rapidly, as in the case of FIG. 6, when arriving at operation 364 from operation 362. In examples, CPU 158 can confirm that first jaw member 108 and second jaw member 110 were not clamped down on target object 124 when the rate of change of resistance feedback was being obtained indicating that the resistance feedback obtained by sensors 114 was a false positive, as in the case of FIG. 7, when arriving at operation 364 from operation 382.

[0101] At operation 366, CPU 158 can provide an indication to a user on display 126 that an improper seal may have been formed. CPU 158 can provide human-perceptible output to a user at operation 366 informing the user of the potentially improper sealing operation and next steps for continuing the surgical operation. Display 126 can provide visual or textual instructions for the user to return to operation 352 and operation 354 to repeat the tissue sealing operation, potentially at another location or by using a different sealing technique,e.g., holding the forceps in engagement with the tissue for a long period of time or opening the forceps at a steady pace.

[0102] At operation 368, CPU 158 can modify an operational capability of energy application device 156 (FIG. 3) or end effector assembly 104 (FIG. ID). For example, CPU 158 can temporarily prevent active electrode 111 and return electrode 113 from being energized, temporarily prevent cutting device 130 from operating, such as by operating a safety and the like. In examples, cutting device 130 can be prevented from moving by a block or obstruction to trigger 132 that is moved into and out of place by a motor controlled by CPU 158.

[0103] At operation 370, a user can take corrective action. For example, a user can follow instructions provided at operation 366 to perform another sealing operation. CPU 158 can follow method 350 to determine if the corrective surgical operation was successful and, if so, can re-enable the operational capability disabled at operation 368, if appropriate. CPU 158 can allow end effector assembly 104 to continue to another surgical operation or to complete the medical procedure. For example, CPU 158 can allow functionality of cutting device 130 to allow the sealed tissue to be cut, thereby completing a portion of the surgical procedure.

[0104] At operation 372, the medical procedure can continue to another medical operation in the medical procedure or to conclusion of the medical procedure.

[0105] At operation 374, if the sensed rate of change in the tissue upon which the sealing operation is being performed matches or is within an acceptable tolerance band of the baseline rate of resistance change, the method can continue to operation 376. CPU 158 can determine that a proper tissue sealing operation may have occurred, such as by determining that output of active electrode 111 and return electrode 113 increased as would be expected for a complete sealing operation. However, CPU 158 can utilize output of cameras 117 obtained at operation 354 to verify that the sensed electrical parameter occurred as would be expected. For example, baseline or threshold resistance values stored in memory can be obtained or determined based on when jaws of a forceps are closed down on tissue to recreate the conditions that would give rise to the same or similar resistance occurring when first jaw member 108 and second jaw member 110 are closed down on tissue.

[0106] At operation 376, CPU 158 can compare the output of sensors 114 to output of cameras 117 to determine an operational state of end effector assembly 104. In example, the visual capture system can monitor the distance between distal tips of first jaw member 108 and second jaw member 110. Thus, the visual capture system can determine distance DI of FIG. 4 A and FIG. 4C and distance D2 of FIG. 4B.

[0107] At operation 378, CPU 158 can determine that first jaw member 108 and second jaw member 110 were closed.

[0108] At operation 380, CPU 158 can confirm that a valid seal was performed. For example, CPU 158 can confirm that first jaw member 108 and second jaw member 110 were clamped down on target object 124 when the rate of change of resistance feedback was being obtained indicating that the tissue sealing operation was completed. Thus, method 350 can continue to operation 372.

[0109] At operation 382, CPU 158 can determine that first jaw member 108 and second jaw member 110 were open. For example, CPU 158 can confirm that first jaw member 108 and second jaw member 110 were not clamped down on target object 124 when the rate of change of resistance feedback was being obtained indicating that the tissue sealing operation was completed. Thus, CPU 158 can infer or determine that corrective action can be taken to confirm or correct the sealing operation. Method 350 can continue to operation 364. From operation 364, method 300 can eventually continue to operation 368 after corrective action has been taken by the user.

[0110] FIG. 9 is a schematic illustration showing a diagram of an exemplary computer-based clinical decision support system (CDSS), e.g., CDSS 400, that is configured to provide an output indicative of the sufficiency, adequacy or properness of the progress of a medical procedure or operation being performed with an electrosurgical device, such as whether or not a tissue sealing procedure has completed forming a seal. For example, CDSS 400 can determine if an end effector has terminated an operation prematurely, such as if a forceps has opened a jaw assembly before a tissue sealing operation has completed. More specifically, CDSS 400 can determine if a jaw assembly has opened prematurely at a slow rate that results in a resistance signal being generated that mimics a resistance signal of tissue being properly sealed. The output can warn a user of the electrosurgical device that the medical procedure being performed may not have completed as intended or desired and can automatically adjust an operation or output of the electrosurgical device to potentially counteract the premature termination of the medical procedure. An exemplary system output can comprise an assessment of a probably diagnosing an improperly formed tissue seal, as well as warnings perceivable by the user to check the seal that was formed of to perform another sealing procedure. An exemplary system output can comprise disabling cutting device 130 until another sealing operation, e.g., a proper sealing operation, has been performed.[OHl] The output can be based on input from sensors 114 and cameras 117, for example. The artificial intelligence model can analyze electrical output of sensors 114 and cameras 117to identify correlations in changes of electrical output related to changes in jaw assembly 106 that result in or are likely to result in an improper seal being formed. Output of sensors 114 can include magnitudes of electrical parameters, changes in electoral parameters and rates of change of electrical parameters. Output of cameras 117 can include still images and motion video of end effector assembly 104, including the position of first jaw member 108, second jaw member 110 and target object 124 therebetween, as well as the rate of change in position between first jaw member 108, second jaw member 110.

[0112] In various embodiments, CDSS 400 can include input interface 402 through which output of sensors 114 and cameras 117 which are specific to a patient are provided as input features to an artificial intelligence (Al) model, e.g., Al model 404, processor 406 which performs an inference operation in which the output of sensors 114 and cameras 117 are applied to the Al model to generate the feedback signals including operator warnings and system adjustments, and a user interface (UI) through which the feedback signals are communicated to a user, e.g., a clinician, such as display 126 and output device 170 of FIG. 3.

[0113] In some embodiments, input interface 402 may be a direct data link between CDSS 400 and one or more medical devices, e.g., medical device 102 (FIG. 1 A) that generate at least some of the input features. For example, input interface 402 can transmit output of sensors 114 and cameras 117 directly to CDSS 400 during a therapeutic and / or diagnostic medical procedure. Additionally, or alternatively, input interface 402 can be a classical user interface that facilitates interaction between a user and CDSS 400. For example, input interface 402 can facilitate a user interface through which the user can manually enter patient information (heigh, weight, age, etc.), tissue type (blood vessel, artery, carotid artery, renal artery, vein, etc.), procedure type (sealing, cutting, cauterizing, etc.) or output of sensors 114 (e.g., rate of change of resistance) and cameras 117 (images with relative positions of first jaw member 108 and second jaw member 110). Additionally, or alternatively, input interface 402 can provide CDSS 400 with access to an electronic patient record 409 from which one or more input features may be extracted. In any of these cases, input interface 402 is configured to collect one or more of the following input features, as well as others, in association with a specific patient on or before a time at which CDSS 400 is used to assess if first jaw member 108 and second jaw member 110 have prematurely opened before a proper surgical operation, e.g., sealing operation, has been performed:• Procedure type;• Tissue type;• Electrosurgical / medical device type;• Baseline or threshold rates of change of resistance to achieve a proper seal for various combinations of procedure, tissue and electrosurgical device;• Selected activation energy;• Activation energy level;• Length of time of activation energy;• Resistance level achieved;• Rate of change of resistance achieved;• End effector status; and• Relative position of end effector jaws.

[0114] Based on one or more of the above input features, processor 406 can perform an inference operation using the Al model to generate one or more feedback signals discussed herein for generating instructions for a user or controlling a function of the medical device. For example, input interface 402 can deliver the output of sensors 114 and cameras 117 into an input layer of the Al model which propagates these input features through the Al model to an output layer. The Al model can provide a computer system the ability to perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. Al model explores the study and construction of algorithms (e.g., machinelearning algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by building an Al model from example training data in order to make data-driven predictions or decisions expressed as outputs or assessments.

[0115] There are two common modes for machine learning (ML): supervised ML and unsupervised ML. Supervised ML uses prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between the inputs and the outputs. The goal of supervised ML is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs so that the ML model can implement the same relationships when given inputs to generate the corresponding outputs.Unsupervised ML is the training of an ML algorithm using information that is neither classified nor labeled, and allowing the algorithm to act on that information without guidance. Unsupervised ML is useful in exploratory analysis because it can automatically identify structure in data.

[0116] Common tasks for supervised ML are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of commonly used supervised- ML algorithms are Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM).

[0117] Some common tasks for unsupervised ML include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised-ML algorithms are K-means clustering, principal component analysis, and autoencoders.

[0118] Another type of ML is federated learning (also known as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to traditional centralized machinelearning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a common, robust machine learning model without sharing data, thus allowing to address critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0119] The machine learning model can be an artificial neural network in some implementations. Artificial neural networks are artificial in the sense that they are computational entities, inspired by biological neural networks but modified for implementation by computing devices. Artificial neural networks are used to model complex relationships between inputs and outputs or to find patterns in data, where the dependency between the inputs and the outputs cannot be easily ascertained. A neural network typically includes an input layer, one or more intermediate (“hidden”) layers, and an output layer, with each layer including a number of nodes. The number of nodes can vary between layers. A neural network is considered “deep” when it includes two or more hidden layers. The nodes in each layer connect to some or all nodes in the subsequent layer and the weights of these connections are typically learnt from data during the training process, for example through backpropagation in which the network parameters are tuned to produce expected outputs given corresponding inputs in labeled training data. Thus, an artificial neural network is an adaptive version of electrosurgical system 100 that is configured to change its structure (e.g., the connection configuration and / or weights) based on information that flows through thenetwork during training, and the weights of the hidden layers can be considered as an encoding of meaningful patterns in the data.

[0120] A fully connected neural network is one in which each node in the input layer is connected to each node in the subsequent layer (the first hidden layer), each node in that first hidden layer is connected in turn to each node in the subsequent hidden layer, and so on until each node in the final hidden layer is connected to each node in the output layer.

[0121] In an example, the machine learning model can include or use a Convolutional Neural Network (CNN). A CNN is a type of artificial neural network, and like the artificial neural network described above, a CNN is made up of nodes and has learnable weights. However, the layers of a CNN can have nodes arranged in three dimensions: width, height, and depth, corresponding to the 2^2 array of pixel values in each video frame (e.g., the width and height) and to the number of video frames in the sequence (e.g., the depth). The nodes of a layer may only be locally connected to a small region of the width and height layer before it, called a receptive field. The hidden layer weights can take the form of a convolutional filter applied to the receptive field. In some examples, the convolutional filters can be two- dimensional, and thus, convolutions with the same filter can be repeated for each frame (or convolved transformation of an image) in the input volume or for designated subset of the frames. In other examples, the convolutional filters can be three-dimensional and thus extend through the full depth of nodes of the input volume. The nodes in each convolutional layer of a CNN can share weights such that the convolutional filter of a given layer is replicated across the entire width and height of the input volume (e.g., across an entire frame), reducing the overall number of trainable weights and increasing applicability of the CNN to data sets outside of the training data. Values of a layer may be pooled to reduce the number of computations in a subsequent layer (e.g., values representing certain pixels may be passed forward while others are discarded), and further along the depth of the CNN pool masks may reintroduce any discarded values to return the number of data points to the previous size. A number of layers, optionally with some being fully connected, can be stacked to form the CNN architecture. The machine learning model can also be at least one of Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Artificial Neural Network (ANN), or an ensemble model combining the SVM and ANN.

[0122] Al model 404, which can include processing unit 122 (FIG. 1 A), can include or use or be capable of executing a machine learning model trained to recognize or identify various types of medical instruments and their end effectors, e.g., forceps jaws, as well as changes in positions of these medical instruments and end effectors using the specimen image. Almodel 404 can include or use one or more algorithms capable of extracting data from the specimen image, such as data by compression, filtering, edge detection, corner detection, blob detection, ridge detection, Hough transform, image segmentation, optical flow, genetic algorithms (GA), or other techniques for algorithmically analyzing the specimen image. The one or more algorithms can be capable of digital image processing such as to extract relevant data from the specimen image to enable recognition of relevant instrument features. The machine learning model can include or use training data received as an input, such as training data classified from a human user. The model can include or use one or more predictive engines. The predictive engine can include or use several engine parameters such as data sources, algorithms, configuration inputs, or other characteristics of the engine under consideration. The predictive engine can include a data source parameter such as for userinput symptom data. The model can include a plurality of algorithms and source code to generate a predictive engine variant by using the training data to help algorithmically arrive upon a label representing the correct identification of a medical instrument, a position of a medical instrument, or a change in position of the medical instrument, or anatomy or tissue such as vessels and the like. The model can replace the predictive engine variant with a new predictive engine variant based on the training data received as an input, performance of the predictive engine variant, or both. The predictive engine variant can be chosen by an operator such as the human user or can be generated automatically. Selections of the engine parameters can be tagged or replayed by the predictive engine such as to evaluate and tune the predictive engine. In some examples, the operator can determine one or more new engine variants manually such as to troubleshoot, tune, or otherwise override the predictive engine. The predictive engine can include or use training data locally, such as to receive training data exclusively with respect to a plurality of specimen images generated by a single user. Alternatively, or additionally, the predictive engine can include or use training data globally, such as to receive training data collectively with respect to specimen images generated by a plurality of users. The predictive engine can interact with one or more servers which can be capable of data storage, local data communication, global data communication, or any combination thereof. The predictive engine can interact with a website such as for global data communication.

[0123] In some examples, the Al model may be trained continuously or periodically prior to performance of the inference operation by the processor 406. Then, during the inference operation, the patient specific input features provided to the Al model may be propagated from an input layer, through one or more hidden layers, and ultimately to an output layer thatcorresponds to the adjustments to the electrosurgical device and feedback and warnings produced to the user. For example, a rate of change in the increase of resistance sensed in an end effector during performance of a tissue sealing procedure on a specific patient can be compared to rates of change in the increase of resistance for other tissue sealing procedures known to have produced a proper seal. If the rate of change in resistance for the specific patient deviates from the known proper sealing procedures, such as by being too rapid, the Al model can determine that the end effector was released from tissue too quickly. If the rate of change in resistance for the specific patient appears to be within an acceptable tolerance band of the known proper sealing procedures, the Al model can seek to verify that a change in the status or operation of the end effector did not interfere with the sealing procedure. Thus, the Al model can look at imaging of the end effector to ensure that the end effector was adequately or properly engaged with tissue when the rate of change of resistance information was obtained. If the end effector is determined to be adequately or properly engaged with tissue, such as by comparison to database 410 of images of and end effector known to be properly engaged with tissue during as sealing procedure that was properly performed, the Al model can verify that the sealing procedure for the specific patient was conducted properly. If the end effector is determined to be inadequately or improperly engaged with tissue, such as by comparison to database 410 of images of and end effector known to be improperly engaged with tissue during as sealing procedure that was improperly performed, the Al model can generate an output signal. During and / or subsequent to the inference operation, the visual, audible or tactile alarm may be communicated to the user via the user interface (UI) and / or automatically cause an apparatus connected to processor 406 to perform a desired action. For example, the output signal can comprise a signal to activate a visual, audible or tactile alarm for a use. The visual, audible or tactile alarm can provide a confidence level for the tissue sealing operation. The visual, audible or tactile alarm can provide information to the user, such as by specifically informing the user that an inadequate seal was likely to have been performed along with directions to re-execute the sealing procedure, e.g., in the same tissue location, or perform a secondary sealing procedure, e.g., on adjacent tissue. The output signal can modify operation of the electrosurgical device, such as by adjusting output of the sealing energy or by locking the electrosurgical device or a portion of the electrosurgical device, such as cutting device 130, until the user acknowledges the visual, audible or tactile alarm and / or instructions.

[0124] FIG. 10 illustrates generally a block diagram of an example machine 600 upon which any one or more of the techniques (e.g., methodologies or operations) discussed herein can beperformed, such as methods for determining the adequate or proper completion of a surgical operation. Portions of this description can apply to the computing framework of various portions of the electrosurgical systems and devices and clinical decision support systems (e.g., machine learning video analysis systems) in accordance with examples as discussed in this document.

[0125] Controller 152 can comprise an example of machine 600. In examples, controller 152 can operate as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, controller 152 can operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, controller 152 can act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. Controller 152 can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0126] Examples, as described herein, can include, or can operate by, logic or a number of components, or mechanisms. Circuit sets are a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership can be flexible over time and underlying hardware variability. Circuit sets include members that can, alone or in combination, perform specified operations when operating. In an example, hardware of the circuit set can be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set can include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a non-transitory computer readable medium physically modified (e.g., magnetically, electrically, movable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively coupled to the othercomponents of the circuit set member when the device is operating. In an example, any of the physical components can be used in more than one member of more than one circuit set. For example, under operation, execution units can be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0127] Controller 152 (e.g., a computer system) can include, or be connected to, CPU 602 (e.g., a central processing unit (CPU or CPU 158), a graphics processing unit (GPU) (e.g., image processor 160), a hardware processor core, or any combination thereof), main memory 604 and static memory 606, some or all of which can communicate with each other via interlink 608 (e.g., bus). Controller 152 can further include display unit 610 (e.g., a raster display, vector display, holographic display, display 126 (FIG. 2), etc.), alphanumeric input device 612 (e.g., a keyboard), and user interface (UI) navigation device 614 (e.g., a mouse). In an example, display unit 610, alphanumeric input device 612 and navigation device 614 can be a touch screen display. Controller 152 can additionally include storage device 616 (e.g., a drive unit), signal generation device 618 (e.g., a speaker), network interface device 620, and one or more sensors 621, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensors. Controller 152 can include output controller 628, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.). Controller 152 can additionally be connected to sensors 114 and cameras 117 of medical device 102 (FIG. ID).

[0128] Storage device 616 can include machine-readable medium 622 on which is stored one or more sets of data structures or instructions 624 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein, such as the operations described with reference to FIG. 8A and FIG. 8A. Instructions 624 can also reside, completely or at least partially, within main memory 604, within static memory 606, or within CPU 158 during execution thereof by controller 152. In an example, one or any combination of CPU 158, main memory 604, static memory 606, or the storage device 616 can constitute machine readable media.

[0129] While machine-readable medium 622 is illustrated as a single medium, the term “machine readable medium” can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 624. As discussed herein, instructions 624 can include baseline or threshold electrical parameter (e.g., resistance, impedance, phase angle), changes in theelectrical parameters that indicate a surgical operation, e.g., a sealing operation, has completed, and rates of changes of the electrical parameters that indicate a surgical operation has completed when an end effector was in a particular state conducive to completing the surgical operation. Instructions 624 can include such information for different types and combinations of medical devices, activation or treatment energies, e.g., RF or ultrasonic, tissue types and the like. Instructions 624 can comprise instructions related to comparing obtained signals from sensors 114 and cameras 117 to the stored baseline or threshold values to determine if the surgical operations have completed successfully. Instructions 624 can comprise instructions for comparing output of sensors 114 to output of cameras 117 along a common timeline. Instructions 624 can comprise instructions for activating output device 170 (FIG. 3) and controlling or preventing the operation of cutting device 130 (FIG. ID).

[0130] The term “machine readable medium” can include any medium that is capable of storing, encoding, or carrying instructions for execution by controller 152 and that cause controller 152 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples can include solid-state memories, and optical and magnetic media. In an example, a massed machine-readable medium comprises a machine readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed machine-readable media are not transitory propagating signals. Specific examples of massed machine-readable media can include: nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EPSOM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0131] Instructions 624 can further be transmitted or received over communication network 626 using a transmission medium via network interface device 620 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks can include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as WiFi®, IEEE 802.16 family of standards known as WiMax®), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, network interfacedevice 620 can include one or more physical jacks (e.g., Ethernet, coaxial, or phonejacks) or one or more antennas to connect to communication network 626. In an example, network interface device 620 can include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by controller 152, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

[0132] The present disclosure pertains to an advanced tissue therapy energy feedback system designed to enhance the precision and effectiveness of surgical procedures involving tissue treatment, such as vessel sealing, tissue cutting, cautery, ablation, and coagulation. The systems and methods of the present disclosure incorporate novel feedback mechanisms that combine electrical feedback, such as tissue impedance and resistance, with visual feedback from imaging capabilities integrated into the tissue therapy device or used in conjunction with it. This dual feedback approach allows for real-time feedback and corrective actions to be conducted based on the specific conditions at the target site, thereby ensuring that an operation of surgical procedure has been properly performed before advancing to another operation or phase of the procedure. The ability of the system to provide immediate feedback and take automated corrective actions significantly reduces the skill level required by the surgeon, minimizes the variability in surgical outcomes, and decreases the potential for complications. By integrating imaging technology, such as cameras, the system ensures that the electrical feedback is accurately representative of the condition of the tissue, preventing false positives that could lead to incomplete or inadequate treatment. Overall, the systems and methos of the present disclosure represent a significant advancement in the field of electrosurgical systems, offering a more reliable, efficient, and user-friendly solution for tissue therapy.Various Notes

[0133] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventor also contemplates examples in which only those elements shown or described are provided. Moreover, thepresent inventor also contemplates examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0134] In the event of inconsistent usages between this document and any documents so incorporated by reference, the usage in this document controls.

[0135] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0136] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.Examples

[0137] Example 1 is a system for imaging and treating tissue of a subject, the system comprising: a tissue therapy device configured to deliver a tissue therapy output for applying to a tissue therapy to tissue at a location internal to the subject; a sensor configured to sense an electrical parameter at the tissue therapy device; an imaging sensor adapted to obtain imaging information of the tissue therapy device from within the location; and controller circuitry comprising signal-processing circuitry configured to: determine a change in state of the tissue therapy from the electrical parameter that is sensed; image-process the imaging information to determine if a change in state of the tissue therapy device internal to the subject has occurred when the change in state of the tissue therapy occurred; and generate or adjust an output of the system if a change in state of the tissue therapy device has occurred.

[0138] In Example 2, the subject matter of Example 1 optionally includes wherein the controller circuitry is further configured to determine that the tissue therapy has been completed from the change in state of the tissue therapy from the electrical parameter.

[0139] In Example 3, the subject matter of Example 2 optionally includes wherein the controller circuitry is further configured to determine that the tissue therapy device has completed forming a seal in tissue.

[0140] In Example 4, the subject matter of any one or more of Examples 1-3 optionally include wherein the controller circuitry is further configured to sense a rate of change of the electrical parameter.

[0141] In Example 5, the subject matter of Example 4 optionally includes wherein the controller circuitry is further configured to determine if the rate of change is above or below a threshold rate of change indicating that the tissue therapy has been completed.

[0142] In Example 6, the subject matter of any one or more of Examples 1-5 optionally include wherein the controller circuitry is further configured to determine if the tissue therapy device changes position from the imaging information.

[0143] In Example 7, the subject matter of any one or more of Examples 1-6 optionally include wherein the controller circuitry is further configured to determine if the tissue therapy device has released tissue.

[0144] In Example 8, the subject matter of any one or more of Examples 1-7 optionally include wherein the output of the system comprises disabling at least one functionality of the system.

[0145] In Example 9, the subject matter of Example 8 optionally includes wherein the output of the system comprises an ability of the tissue therapy device to cut tissue.

[0146] In Example 10, the subject matter of any one or more of Examples 1-9 optionally include wherein the output of the system comprises at least one of an audio, visual, tactile, written and iconic alarm.

[0147] In Example 11, the subject matter of any one or more of Examples 1-10 optionally include wherein the tissue therapy device comprises forceps comprising a first jaw member pivotably connected to a second jaw member.

[0148] In Example 12, the subject matter of Example 11 optionally includes wherein the controller circuitry is configured to sense resistance between the first jaw member and the second jaw member.

[0149] In Example 13, the subject matter of any one or more of Examples 11-12 optionally include wherein the controller circuitry is configured to determine a relative position between the first jaw member and the second jaw member.

[0150] In Example 14, the subject matter of any one or more of Examples 11-13 optionally include wherein the controller circuitry is configured to determine: a first position wherein first jaw member and the second jaw member are spaced apart from each other; and a second position wherein the first jaw member and the second jaw member are positioned closer to or further apart from each other.

[0151] In Example 15, the subject matter of any one or more of Examples 1-14 optionally include wherein the imaging sensor comprises a camera.

[0152] Example 16 is a method for treating tissue using an electrosurgical system, the method comprising: monitoring operation of an end effector assembly of a medical device with an imaging sensor; energizing the end effector assembly using an electrosurgical energy to perform a surgical operation; sensing a parameter of the electrosurgical energy at the end effector assembly; determining a change in state of the surgical operation from the parameter that is sensed; determining if there is a change in state of the end effector assembly using the imaging sensor when a change in state of the surgical operation is sensed; and adjusting or generating an output of the electrosurgical system if there is a change in state of the end effector assembly.

[0153] In Example 17, the subject matter of Example 16 optionally includes wherein determining the change of state of the surgical operation from the parameter that is sensed comprises determining that the surgical operation has been completed.

[0154] In Example 18, the subject matter of Example 17 optionally includes wherein determining that the surgical operation has been completed comprises determining that the end effector assembly has completed forming a seal in tissue using the electrosurgical energy.

[0155] In Example 19, the subject matter of any one or more of Examples 16-18 optionally include wherein sensing the parameter of the electrosurgical energy at the end effector assembly comprises sensing a rate of change of the parameter.

[0156] In Example 20, the subject matter of Example 19 optionally includes wherein sensing the parameter of the electrosurgical energy at the end effector assembly comprises determining if the rate of change is above or below a threshold rate of change indicating that the surgical operation has been completed.

[0157] In Example 21, the subject matter of any one or more of Examples 16-20 optionally include wherein determining if there is the change in state of the end effector assembly using the imaging sensor comprises determining if the end effector assembly changes position.

[0158] In Example 22, the subject matter of Example 21 optionally includes wherein determining if the end effector assembly changes position comprises determining if the end effector assembly has released tissue.

[0159] In Example 23, the subject matter of any one or more of Examples 16-22 optionally include wherein adjusting or generating the output of the electrosurgical system comprises disabling a capability of the end effector.

[0160] In Example 24, the subject matter of any one or more of Examples 16-23 optionally include wherein adjusting or generating the output of the electrosurgical system comprises adjusting an amount of electrosurgical energy used to energize the end effector assembly.

[0161] In Example 25, the subject matter of Example 24 optionally includes wherein adjusting or generating the output of the electrosurgical system comprises generating at least one of an audio, visual, tactile, written and iconic alarm.

[0162] In Example 26, the subject matter of any one or more of Examples 16-25 optionally include wherein the end effector assembly comprises forceps comprising a first jaw member and a second jaw member.

[0163] In Example 27, the subject matter of Example 26 optionally includes wherein sensing the parameter of the electrosurgical energy at the end effector assembly comprises sensing resistance between the first jaw member and the second jaw member.

[0164] In Example 28, the subject matter of any one or more of Examples 26-27 optionally include wherein monitoring operation of the end effector assembly of the medical device with the imaging sensor comprises determining a relative position between the first jaw member and the second jaw member.

[0165] In Example 29, the subject matter of Example 28 optionally includes wherein monitoring operation of the end effector assembly of the medical device with the imagingsensor comprises observing the end effector assembly move between: a first position wherein first and second jaw members of the end effector assembly are spaced apart from each other; and a second position wherein the first and second jaw members are positioned closer to or further apart from each other.

Claims

CLAIMS:

1. A system for imaging and treating tissue of a subject, the system comprising: a tissue therapy device configured to deliver a tissue therapy output for applying to a tissue therapy to tissue at a location internal to the subject; a sensor configured to sense an electrical parameter at the tissue therapy device; an imaging sensor adapted to obtain imaging information of the tissue therapy device from within the location; and controller circuitry comprising signal-processing circuitry configured to: determine a change in state of the tissue therapy from the electrical parameter that is sensed; image-process the imaging information to determine if a change in state of the tissue therapy device internal to the subject has occurred when the change in state of the tissue therapy occurred; and generate or adjust an output of the system if a change in state of the tissue therapy device has occurred.

2. The system of claim 1, wherein the controller circuitry is further configured to determine that the tissue therapy has been completed from the change in state of the tissue therapy from the electrical parameter.

3. The system of claim 2, wherein the controller circuitry is further configured to determine that the tissue therapy device has completed forming a seal in tissue.

4. The system of claim 1, wherein the controller circuitry is further configured to sense a rate of change of the electrical parameter.

5. The system of claim 4, wherein the controller circuitry is further configured to determine if the rate of change is above or below a threshold rate of change indicating that the tissue therapy has been completed.

6. The system of claim 1, wherein the controller circuitry is further configured to determine if the tissue therapy device changes position from the imaging information.

7. The system of claim 1, wherein the controller circuitry is further configured to determine if the tissue therapy device has released tissue.

8. The system of claim 1, wherein the output of the system comprises disabling at least one functionality of the system.

9. The system of claim 8, wherein the output of the system comprises an ability of the tissue therapy device to cut tissue.

10. The system of claim 1, wherein the output of the system comprises at least one of an audio, visual, tactile, written and iconic alarm.

11. The system of claim 1, wherein the tissue therapy device comprises forceps comprising a first jaw member pivotably connected to a second jaw member.

12. The system of claim 11, wherein the controller circuitry is configured to sense resistance between the first jaw member and the second jaw member.

13. The system of claim 11, wherein the controller circuitry is configured to determine a relative position between the first jaw member and the second jaw member.

14. The system of claim 11, wherein the controller circuitry is configured to determine: a first position wherein first jaw member and the second jaw member are spaced apart from each other; and a second position wherein the first jaw member and the second jaw member are positioned closer to or further apart from each other.

15. The system of claim 1, wherein the imaging sensor comprises a camera.

16. A method for treating tissue using an electrosurgical system, the method comprising: monitoring operation of an end effector assembly of a medical device with an imaging sensor; energizing the end effector assembly using an electrosurgical energy to perform a surgical operation;sensing a parameter of the electrosurgical energy at the end effector assembly; determining a change in state of the surgical operation from the parameter that is sensed; determining if there is a change in state of the end effector assembly using the imaging sensor when a change in state of the surgical operation is sensed; and adjusting or generating an output of the electrosurgical system if there is a change in state of the end effector assembly.

17. The method of claim 16, wherein determining the change of state of the surgical operation from the parameter that is sensed comprises determining that the surgical operation has been completed.

18. The method of claim 17, wherein determining that the surgical operation has been completed comprises determining that the end effector assembly has completed forming a seal in tissue using the electrosurgical energy.

19. The method of claim 16, wherein sensing the parameter of the electrosurgical energy at the end effector assembly comprises sensing a rate of change of the parameter.

20. The method of claim 19, wherein sensing the parameter of the electrosurgical energy at the end effector assembly comprises determining if the rate of change is above or below a threshold rate of change indicating that the surgical operation has been completed.

21. The method of claim 16, wherein determining if there is the change in state of the end effector assembly using the imaging sensor comprises determining if the end effector assembly changes position.

22. The method of claim 21, wherein determining if the end effector assembly changes position comprises determining if the end effector assembly has released tissue.

23. The method of claim 16, wherein adjusting or generating the output of the electrosurgical system comprises disabling a capability of the end effector.

24. The method of claim 16, wherein adjusting or generating the output of the electrosurgical system comprises adjusting an amount of electrosurgical energy used to energize the end effector assembly.

25. The method of claim 24, wherein adjusting or generating the output of the electrosurgical system comprises generating at least one of an audio, visual, tactile, written and iconic alarm.

26. The method of claim 16, wherein the end effector assembly comprises forceps comprising a first jaw member and a second jaw member.

27. The method of claim 26, wherein sensing the parameter of the electrosurgical energy at the end effector assembly comprises sensing resistance between the first jaw member and the second jaw member.

28. The method of claim 26, wherein monitoring operation of the end effector assembly of the medical device with the imaging sensor comprises determining a relative position between the first jaw member and the second jaw member.

29. The method of claim 28, wherein monitoring operation of the end effector assembly of the medical device with the imaging sensor comprises observing the end effector assembly move between: a first position wherein first and second jaw members of the end effector assembly are spaced apart from each other; and a second position wherein the first and second jaw members are positioned closer to or further apart from each other.

Images