Advanced bipolar seal quality prediction

Abstract

A system for assessing the quality of a seal applied to tissue by an electrosurgical instrument is provided. The system may include a surgical device having control circuitry for determining that an end effector of the surgical device is in a closed configuration, determining a presence of tissue disposed between a first jaw and a second jaw of the end effector, movement of the end effector in the closed configuration with tissue present between the first jaw and the second jaw is monitored, movement of the end effector outside a predetermined range is detected based on the movement data, and feedback data is provided based on the detected movement of the end effector.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application No. 63 / 357,177, entitled ADVANCED BIPOLAR SEAL QUALITY PREDICTION, filed on June 30, 2022, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] The present disclosure relates to electrosurgical instruments designed to seal and cut tissue. Summary of the invention

[0004] The following summary is provided to facilitate an understanding of some of the innovative features characteristic of the various aspects disclosed herein and is not intended to be a complete description. A full understanding of the various aspects can be gained by taking the entire specification, claims, and abstract as a whole.

[0005] In one general aspect, the present disclosure relates to a surgical device. The surgical device includes an end effector including a first jaw and a second jaw. The surgical device also includes: a first sensor for detecting tissue disposed between the first jaw and the second jaw; a second sensor for detecting the end effector in a closed configuration; and a third sensor for detecting movement of the end effector. The surgical device also includes a control circuit that is communicatively coupled to the first sensor, the second sensor, and the third sensor. The control circuit includes a processor and a memory, wherein the memory stores instructions that, when executed by the processor, cause the control circuit to determine that the end effector is in a closed configuration based on first sensor data, determine the presence of tissue disposed between the first jaw and the second jaw based on second sensor data, and monitor the movement of the end effector in a closed configuration when tissue is present between the first jaw and the second jaw based on third sensor data. The memory stores additional instructions that, when executed by the processor, cause the control circuit to detect movement of the end effector outside a predetermined range based on the movement, and provide feedback data based on the detected movement of the end effector.

[0006] In at least one aspect, the memory stores further instructions that, when executed by the processor, cause the control circuit to calculate tension on the tissue based on the movement.

[0007] In at least one aspect, the feedback is visual. In at least one aspect, the visual feedback is superimposed on a displayed image of the surgical site.

[0008] In at least one aspect, the feedback is auditory.

[0009] In at least one aspect, the first jaw comprises a clamping arm and the second jaw comprises an ultrasonic blade.

[0010] In at least one aspect, the first jaw comprises an anvil and the second jaw comprises a staple cartridge.

[0011] In another general aspect, the present disclosure relates to a surgical device. The surgical device includes an end effector, the end effector including a first jaw and a second jaw. The surgical device also includes: a first sensor for detecting tissue disposed between the first jaw and the second jaw; a second sensor for detecting that the end effector is in a closed configuration; a first fiducial marker; and a second fiducial marker. The surgical device also includes a control circuit that is communicatively coupled to the first sensor, the second sensor, and a camera. The control circuit includes a processor and a memory, wherein the memory stores instructions that, when executed by the processor, cause the control circuit to receive video data of a surgical site from a camera, determine that the end effector is in a closed configuration based on the first sensor data, and determine the presence of tissue disposed between the first jaw and the second jaw based on the second sensor data. The memory stores additional instructions that, when executed by the processor, cause the control circuit to determine a position of a device tip in the video data based on the first fiducial marker and the second fiducial marker, determine a region of interest in the video data based on the position of the device tip in the video data, and analyze the region of interest of the end effector in the closed configuration when tissue is present between the first jaw and the second jaw. The memory stores further instructions that, when executed by the processor, cause the control circuit to determine tension on the tissue based on the analysis and to provide feedback based on the tension.

[0012] In at least one aspect, the memory stores additional instructions that, when executed by the processor, cause the control circuit to detect movement of the end effector outside of a predetermined range based on the analysis.

[0013] In at least one aspect, the memory stores further instructions that, when executed by the processor, cause the control circuit to determine a device type based on the first fiducial marker and the second fiducial marker.

[0014] In at least one aspect, the feedback is visual. In at least one aspect, the visual feedback is superimposed on a displayed image of the surgical site.

[0015] In at least one aspect, the feedback is auditory.

[0016] In at least one aspect, the first jaw comprises a clamping arm and the second jaw comprises an ultrasonic blade.

[0017] In at least one aspect, the first jaw comprises an anvil and the second jaw comprises a staple cartridge.

[0018] In yet another general aspect, the present disclosure relates to a surgical system comprising a surgical instrument, an RF energy source, and a control circuit. The surgical instrument comprises an end effector to capture tissue. The end effector comprises an electrode to apply radio frequency (RF) energy to the tissue captured by the end effector. The RF energy source is used to provide RF energy to the electrode. The control circuit is used to transmit a control signal to the RF energy source. The control signal causes the RF energy source to provide RF energy to the electrode to apply a seal to the tissue captured by the end effector. The control circuit is further used to predict the quality of the seal and provide feedback to the user based on the prediction.

[0019] In at least one aspect, the control circuit is configured to generate a value associated with the seal and compare the value to a seal threshold. To provide feedback to the user based on the prediction, the control circuit is configured to provide feedback to the user based on a result of the comparison.

[0020] In at least one aspect, the control circuit is configured to forego providing feedback based on the value reaching or exceeding a sealing threshold.

[0021] In at least one aspect, the surgical system further comprises a display. The control circuit is configured to transmit a signal to the display based on the value being below the sealing threshold. The feedback comprises visual feedback on the display. The visual feedback is based on the signal.

[0022] In at least one aspect, the surgical system further comprises an audio feedback module. The control circuit is configured to transmit a signal to the audio feedback module based on the value being below the sealing threshold. The feedback comprises audio feedback via the audio feedback module. The audio feedback is based on the signal.

[0023] In at least one aspect, the surgical system further comprises a tactile feedback module. The control circuit is configured to transmit a signal to the tactile feedback module based on the value being below the sealing threshold. The feedback comprises tactile feedback via the tactile feedback module. The tactile feedback is based on the signal.

[0024] In at least one aspect, the RF energy source includes control circuitry.

[0025] In at least one aspect, the surgical system further comprises a processing unit comprising control circuitry. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The various features of the embodiments described herein, together with their advantages, may be understood from the following description in conjunction with the following drawings:

[0027] Figure 1 A surgical hub paired with a visualization system, a robotic system, and a smart instrument according to at least one aspect of the present disclosure.

[0028] Figure 2 Control circuits for controlling various aspects of a surgical instrument or tool in accordance with at least one aspect of the present disclosure are described.

[0029] Figure 3 Combinatorial logic circuits for controlling various aspects of a surgical instrument or tool in accordance with at least one aspect of the present disclosure are described.

[0030] Figure 4 Sequential logic circuits for controlling various aspects of a surgical instrument or tool in accordance with at least one aspect of the present disclosure are described.

[0031] Figure 5 is a schematic diagram of a surgical instrument for controlling various functions according to at least one aspect of the present disclosure.

[0032] Figure 6 is an illustration of various modules and other components that may be combined to customize a modular energy system in accordance with at least one aspect of the present disclosure.

[0033] Figure 7 A video rating system for surgical documentation is depicted in accordance with at least one aspect of the present disclosure.

[0034] Figure 8 is a surgical view from a camera of a surgical instrument according to at least one aspect of the present disclosure, with the jaws open.

[0035] Fig. 9 is a surgical view from a camera of a surgical instrument according to at least one aspect of the present disclosure, with the jaws clamped on tissue.

[0036] Fig.10 is a surgical view of a camera of a surgical instrument according to at least one aspect of the present disclosure, with the jaws clamped after cutting and sealing tissue.

[0037] Fig.11 is an illustration of a control process for monitoring end effector movement according to at least one aspect of the present disclosure.

[0038] Fig.12 is a graphical representation of the movement of a powered stapler and a non-powered stapler in accordance with at least one aspect of the present disclosure.

[0039] Fig.13 is a surgical view from a camera of a surgical instrument having fiducial markers according to at least one aspect of the present disclosure.

[0040] Fig.14 is a surgical view from a camera with a tissue tension alarm superimposed on the view according to at least one aspect of the present disclosure.

[0041] Fig.15 is a diagram of a control process for calculating tissue tension according to at least one aspect of the present disclosure.

[0042] Fig.16 is a representation of axial tissue tension according to at least one aspect of the present disclosure.

[0043] Fig.17 is a representation of radial tissue tension according to at least one aspect of the present disclosure.

[0044] Fig.18 Flowcharts in accordance with at least one aspect of the present disclosure are described.

[0045] FIG. 19A to FIG. 19C Video tags in accordance with at least one aspect of the present disclosure are described.

[0046] Fig. 20 A flow chart illustrating multi-task learning during model training and validation in accordance with at least one aspect of the present disclosure is illustrated.

[0047] Fig.21 A flow chart illustrating multi-task learning during verification, validation, and clinical use in accordance with at least one aspect of the present disclosure is illustrated.

[0048] Fig. 22 A composite load curve lookup table in accordance with at least one aspect of the present disclosure is described.

[0049] Figure 23 to Figure 25 Wavelet signal processing in accordance with at least one aspect of the present disclosure is described.

[0050] Fig.26 An electrical parameter split into "x" time length segments in accordance with at least one aspect of the present disclosure is illustrated.

[0051] Figure 27 to Figure 29 A specific implementation of a synthetic minority oversampling technique ("SMOTE") in accordance with at least one aspect of the present disclosure is described.

[0052] Fig.30 PCA for identifying relevant features in accordance with at least one aspect of the present disclosure is described.

[0053] Fig.31 An xgboost that allows a model to be tuned back on multiple parameters in accordance with at least one aspect of the present disclosure is described.

[0054] Fig.32A and Fig.32B A model architecture for a CNN model according to at least one aspect of the present disclosure is described.

[0055] Fig.33 A confusion matrix in accordance with at least one aspect of the present disclosure is described.

[0056] Fig.34 ROC curves according to at least one aspect of the present disclosure are illustrated.

[0057] Fig.35 A confusion matrix in accordance with at least one aspect of the present disclosure is described.

[0058] FIG. 36A to FIG. 36D Four confusion matrices in accordance with at least one aspect of the present disclosure are described.

[0059] Fig.37 Visual feedback indicating a good quality seal on a display in accordance with at least one aspect of the present disclosure is described.

[0060] Fig.38 Visual feedback indicating a poor quality seal on a display in accordance with at least one aspect of the present disclosure is described.

[0061] Fig.39 A first embodiment of a seal quality prediction system according to at least one aspect of the present disclosure is described.

[0062] Fig.40 A second embodiment of a seal quality prediction system according to at least one aspect of the present disclosure is described.

[0063] Corresponding reference characters indicate corresponding parts throughout the several views.The exemplification described herein illustrates various embodiments of the invention in one form, and such exemplification should not be construed as limiting the scope of the invention in any way. DETAILED DESCRIPTION

[0064] Many specific details are listed herein to provide a thorough understanding of the overall structure, function, manufacture, and use of the embodiments described in the specification and shown in the drawings. Well-known operations, parts, and elements are not described in detail to avoid obscuring the embodiments described in the specification. The reader will understand that the embodiments described and shown herein are non-limiting examples, so that it can be recognized that the specific structural and functional details disclosed herein can be representative and illustrative. These embodiments can be modified and changed without departing from the scope of the claims.

[0065] The terms "comprise" (and any form of "comprise", such as "comprises" and "comprising"), "have" (and any form of "have", such as "has" and "having"), "include" (and any form of "include", such as "includes" and "including"), and "contain" (and any form of "contain", such as "contains" and "containing") are open-ended linking verbs. Thus, a surgical system, device, or apparatus that "comprises," "has," "includes," or "contains" one or more elements has those one or more elements, but is not limited to having only those one or more elements. Likewise, an element of a system, device, or apparatus that "comprises," "has," "includes," or "contains" one or more features has those one or more features, but is not limited to having only those one or more features.

[0066] The terms "proximal" and "distal" are used herein relative to a clinician manipulating the handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician, and the term "distal" refers to the portion positioned away from the clinician. It should also be understood that for brevity and clarity, spatial terms such as "vertical," "horizontal," "upper," and "lower" may be used herein in conjunction with the accompanying drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and / or absolute.

[0067] Various exemplary devices and methods are provided for performing laparoscopic and minimally invasive surgical procedures. However, the reader will readily appreciate that the various methods and devices disclosed herein can be used in a variety of surgical procedures and applications, including, for example, in conjunction with open surgical procedures. Continuing to refer to this detailed description, the reader will further appreciate that the various instruments disclosed herein can be inserted into the body in any manner, such as through a natural cavity, through an incision or puncture formed in the tissue, etc. The working portion or end effector portion of the instrument can be inserted directly into the patient's body or can be inserted through an access device having a working channel through which the end effector and elongated shaft of the surgical instrument can be advanced.

[0068] refer to Figure 1, the hub 106 is depicted as communicating with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. In some aspects, the visualization system 108 can be a detachable piece of the device. In alternative aspects, the visualization system 108 can be included as a functional module within the hub 106. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, a storage array 134, and an operating room mapping module 133. In some aspects, as Figure 1 As described in , the hub 106 also includes a smoke evacuation module 126, a suction / rinsing module 128, and / or an insufflation module 129. In certain aspects, any of the modules in the hub 106 can be combined with each other into a single module.

[0069] During surgery, energy application to tissue for sealing and / or cutting is usually associated with smoke evacuation, suction of excess fluid and / or flushing of tissue. Fluid lines, power lines and / or data lines from different sources are usually tangled during surgery. Solving this problem during surgery can waste valuable time. Disconnecting the lines may require disconnecting the lines from their corresponding modules, which may require resetting the modules. The hub modular housing 136 provides a unified environment for managing power lines, data lines and fluid lines, which reduces the frequency of tangling between such lines.

[0070] Various aspects of the present disclosure provide a surgical hub for use in a surgical procedure involving application of energy to tissue at a surgical site. The surgical hub includes a hub housing and a combination generator module that can be slidably received in a docking base of the hub housing. The docking base includes data contacts and power contacts. The combination generator module includes one or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed in a single unit. In one aspect, the combination generator module also includes a smoke exhaust component, at least one energy delivery cable for connecting the combination generator module to a surgical instrument, at least one smoke exhaust component for exhausting smoke, fluid, and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke exhaust component.

[0071] In one aspect, the fluid line is a first fluid line and the second fluid line extends from the remote surgical site to an aspiration and irrigation module slidably received in the hub housing. In one aspect, the hub housing includes a fluid interface.

[0072] Certain surgical procedures may require more than one energy type to be applied to tissue. One energy type may be more conducive to cutting tissue, while another different energy type may be more conducive to sealing tissue. For example, a bipolar generator may be used to seal tissue, while an ultrasonic generator may be used to cut sealed tissue. Aspects of the present disclosure provide a solution in which a hub modular housing 136 is used to house different generators and facilitate interactive communication between these generators. One of the advantages of the hub modular housing 136 is that it enables rapid removal and / or replacement of various modules.

[0073] Aspects of the present disclosure provide a modular surgical shell for use in a surgical procedure involving the application of energy to tissue. The modular surgical shell includes a first energy generator module and a first docking station, the first energy generator module being used to generate a first energy for application to tissue, and the first docking station comprising a first docking port, the first docking port comprising a first data contact and a power contact. In one aspect, the first energy generator module is slidably movable into electrical engagement with the power contacts and the data contacts, and wherein the first energy generator module is slidably movable out of electrical engagement with the first power contacts and the data contacts. In an alternative aspect, the first energy generator module is stackably movable into electrical engagement with the power contacts and the data contacts, and wherein the first energy generator module is stackably movable out of electrical engagement with the first power contacts and the data contacts.

[0074] In addition to the above, the modular surgical shell also includes a second energy generator module and a second docking station, the second energy generator module for generating a second energy that is the same as or different from the first energy for application to tissue, and the second docking station includes a second docking port, the second docking port including second data contacts and power contacts. In one aspect, the second energy generator module is slidably movable to electrically engage with the power contacts and the data contacts, and wherein the second energy generator module is slidably movable to not electrically engage with the second power contacts and the data contacts. In an alternative aspect, the second energy generator module is stackably movable to electrically engage with the power contacts and the data contacts, and wherein the second energy generator module is stackably movable to not electrically engage with the second power contacts and the data contacts.

[0075] Additionally, the modular surgical housing further includes a communication bus located between the first docking port and the second docking port to facilitate communication between the first energy generator module and the second energy generator module.

[0076] refer to Figure 1, aspects of the present disclosure are presented as a hub modular housing 136 that allows modular integration of a generator module 140, a smoke evacuation module 126, and a suction / irrigation module 128 and an insufflation module 129. The hub modular housing 136 further facilitates interactive communication between the modules 140, 126, 128, 129. The generator module 140 may be a generator module having integrated monopolar, bipolar, and ultrasonic components supported in a single housing unit that is slidably inserted into the hub modular housing 136. The generator module 140 may be used to connect to a monopolar device 142, a bipolar device 144, and an ultrasonic device 148. Alternatively, the generator module 140 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 136. The hub modular housing 136 may be used to facilitate insertion of multiple generators and interactive communication between the generators docked into the hub modular housing 136 so that the generators will function as a single generator.

[0077] In one aspect, the hub modular housing 136 includes a modular power and communication chassis 149 with external and wireless communication heads to enable removable attachment of modules 140, 126, 128, 129 and interactive communications therebetween.

[0078] Additional information about hubs can be found in U.S. Patent Application Publication No. 2019 / 0201136 and U.S. Patent Application Publication No. 2020 / 0078106, both of which are hereby incorporated by reference in their entirety.

[0079] Figure 2 A control circuit 500 for controlling various aspects of a surgical instrument or tool according to one aspect of the present disclosure is described. The control circuit 500 can be used to implement the various processes described herein. The control circuit 500 may include a microcontroller including one or more processors 502 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 504. The memory circuit 504 stores machine executable instructions that, when executed by the processor 502, cause the processor 502 to execute machine instructions to implement the various processes described herein. The processor 502 can be any of a variety of single-core or multi-core processors known in the art. The memory circuit 504 may include a volatile storage medium and a non-volatile storage medium. The processor 502 may include an instruction processing unit 506 and an operation unit 508. The instruction processing unit can be used to receive instructions from the memory circuit 504 of the present disclosure.

[0080] Figure 3A combinatorial logic circuit 510 for controlling various aspects of a surgical instrument or tool according to one aspect of the present disclosure is illustrated. The combinatorial logic circuit 510 can be used to implement the various processes described herein. The combinatorial logic circuit 510 can include a finite state machine including a combinatorial logic component 512 for receiving data associated with a surgical instrument or tool at an input 514, processing the data by the combinatorial logic component 512, and providing an output 516.

[0081] Figure 4 A sequential logic circuit 520 for controlling various aspects of a surgical instrument or tool according to one aspect of the present disclosure is illustrated. The sequential logic circuit 520 or the combinational logic component 522 may be used to implement the various processes described herein. The sequential logic circuit 520 may include a finite state machine. The sequential logic circuit 520 may include, for example, a combinational logic component 522, at least one memory circuit 524, and a clock 529. The at least one memory circuit 524 may store a current state of the finite state machine. In some cases, the sequential logic circuit 520 may be synchronous or asynchronous. The combinational logic component 522 is used to receive data associated with a surgical instrument or tool from an input 526, process the data by the combinational logic component 522, and provide an output 528. In other aspects, the circuit may include a processor (e.g., a processor 502, Figure 2 ) and a finite state machine to implement various processes herein. In other aspects, the finite state machine may include a combinational logic circuit (e.g., combinational logic circuit 510, Figure 3 ) and a sequential logic circuit 520.

[0082] Figure 5 is a schematic diagram of a surgical instrument 790 for controlling various functions according to one aspect of the present disclosure. In one aspect, the surgical instrument 790 is programmed to control the distal translation of a displacement member such as a closing member 764. The surgical instrument 790 includes an end effector 792, which may include a clamping arm 766, a closing member 764, and a blade 768, which may be interchangeable with one or more RF electrodes 796 (shown in dotted lines) or work in conjunction with one or more RF electrodes. In various embodiments, the blade may include an I-beam, such as the I-beam described elsewhere herein. In various embodiments, the knife may include an ultrasonic knife coupled to an ultrasonic transducer driven by an ultrasonic generator.

[0083] In one aspect, the sensor 788 may be implemented as a limit switch, an electromechanical device, a solid-state switch, a Hall effect device, an MR device, a GMR device, a magnetometer, and the like. In other embodiments, the sensor 638 may be a solid-state switch that operates under the influence of light, such as an optical sensor, an IR sensor, an ultraviolet sensor, and the like. Likewise, the switch may be a solid-state device, such as a transistor (e.g., a FET, a junction FET, a MOSFET, a bipolar transistor, and the like). In other embodiments, the sensor 788 may include a non-electrically conductive switch, an ultrasonic switch, an accelerometer, an inertial sensor, and the like.

[0084] In one aspect, the position sensor 784 can be implemented as an absolute positioning system including a magnetic rotary absolute positioning system implemented as an AS5055EQFT monolithic magnetic rotary position sensor, which can be purchased from Austria Microsystems, AG. The position sensor 784 can interface with the control circuit 760 to provide an absolute positioning system. The position can include a Hall effect element located above a magnet and coupled to a CORDIC processor, which is also known as a bitwise method and Volder algorithm, which is provided to implement a simple and efficient algorithm for calculating hyperbolic and trigonometric functions, which only require addition operations, subtraction operations, bit shift operations, and table lookup operations.

[0085] In some examples, the positioning sensor 784 may be omitted. Where the motor 754 is a stepper motor, the control circuit 760 may track the position of the closing member 764 by gathering the number and direction of steps the motor has been instructed to perform. The positioning sensor 784 may be located in the end effector 792 or at any other portion of the instrument.

[0086] The control circuit 760 may communicate with one or more sensors 788. The sensor 788 may be positioned on the end effector 792 and adapted to operate with the surgical instrument 790 to measure various derived parameters, such as gap distance versus time, tissue compression versus time, and anvil strain versus time. The sensor 788 may include a magnetic sensor, a magnetic field sensor, a strain gauge, a pressure sensor, a force sensor, an inductive sensor (such as an eddy current sensor), a resistive sensor, a capacitive sensor, an optical sensor, and / or any other suitable sensor for measuring one or more parameters of the end effector 792. The sensor 788 may include one or more sensors.

[0087] The RF energy source 794 is coupled to the end effector 792 and is applied to the RF electrode 796 when the RF electrode 796 is disposed in the end effector 792 to replace the blade 768 or to work in conjunction with the blade 768. For example, the blade is made of a conductive metal and can serve as a return path for the electrosurgical RF current. The control circuit 760 controls the delivery of RF energy to the RF electrode 796.

[0088] The control circuit 760 can communicate with a tactile feedback module 870. In some embodiments, the tactile feedback module 870 is positioned in a handpiece, such as within the handpieces 1205, 1207, 1209 of the surgical instruments 1204, 1206, 1208 described in more detail below. In some embodiments, the tactile feedback module 870 is positioned within an input interface with which a user interacts to control various surgical instruments described elsewhere herein. The control circuit 760 can transmit control signals to the tactile feedback module 870 to actuate the tactile feedback module 870 and provide tactile feedback, as described in more detail elsewhere herein.

[0089] The control circuit 760 may be in communication with the audio feedback module 871. The control circuit 760 may transmit control signals to the audio feedback module 871 to cause the audio feedback module 871 to actuate and provide audio feedback to the user, as described in greater detail elsewhere herein.

[0090] Additional details are disclosed in U.S. patent application serial number 15 / 636,096, filed on June 28, 2017, entitled “SURGICAL SYSTEM COUPLABLE WITH STAPLE CARTRIDGE AND RADIO FREQUENCY CARTRIDGE, AND METHOD OF USING SAME,” which is incorporated herein by reference in its entirety.

[0091] Generator Hardware

[0092] As used throughout this specification, the term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that can transmit data through a non-solid medium using modulated electromagnetic radiation. The term does not mean that the associated device does not contain any wires, although in some aspects they may not. The communication module may implement any of a variety of wireless or wired communication standards or protocols, including but not limited to Wi-Fi (IEEE 802.11 series), WiMAX (IEEE 802.16 series), IEEE 802.20, Long Term Evolution (LTE), Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, Bluetooth, and its Ethernet derivatives, and any other wireless and wired protocols designated as 3G, 4G, 5G and above. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter range wireless communications such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, etc.

[0093] As used herein, a processor or processing unit is an electronic circuit that performs operations on some external data source (usually a memory or some other data stream). The term is used herein to refer to a central processing unit (Central Processing Unit) in a system or computer system (especially a System on a Chip (SoC)) that combines multiple specialized "processors".

[0094] As used herein, a system on a chip or system on a chip (SoC or SOC) is an integrated circuit (also referred to as an "IC" or "chip") that integrates all the components of a computer or other electronic system. It may include digital, analog, mixed signal, and often radio frequency functions - all on a single substrate. A SoC integrates a microcontroller (or microprocessor) with advanced peripherals such as a graphics processing unit (GPU), a Wi-Fi module, or a coprocessor. A SoC may or may not include built-in memory.

[0095] As used herein, a microcontroller or controller is a system that integrates a microprocessor with peripheral circuits and memory. A microcontroller (or MCU for a microcontroller unit) can be implemented as a small computer on a single integrated circuit. It can be similar to a SoC; an SoC may include a microcontroller as one of its components. A microcontroller may include one or more core processing units (CPUs) as well as memory and programmable input / output peripherals. Program memory in the form of ferroelectric RAM, NOR flash memory, or OTP ROM, as well as a small amount of RAM, is also often included on the chip. Compared to microprocessors used in personal computers or other general-purpose applications consisting of various discrete chips, microcontrollers can be used for embedded applications.

[0096] As used herein, the term controller or microcontroller may be a stand-alone IC or chip device that interfaces with peripheral devices. This may be a link between the two components of a computer or a controller on an external device that manages the operation of the device (and the connection to the device).

[0097] Any of the processors or microcontrollers described herein may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes: 256KB of single-cycle flash memory or other non-volatile memory (up to 40MHz) on-chip memory, a pre-fetch buffer for improving performance above 40MHz, 32KB of single-cycle serial random access memory (SRAM), a 32KB MCU loaded with Internal read-only memory (ROM) for software, 2KB of electrically erasable programmable read-only memory (EEPROM), one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) simulations, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which can be found in the product data sheet.

[0098] In one example, the processor may include a safety controller including two families based on controllers such as the TMS570 and RM4x also offered by Texas Instruments under the trade name Hercules ARM Cortex R4. The safety controller may be configured to be specifically designed for IEC 61508 and ISO 26262 safety critical applications, etc., to provide advanced integrated safety features while delivering scalable execution, connectivity and memory options.

[0099] The modular device includes a surgical hub (e.g., such as a combination of Figure 1The invention relates to a surgical device or instrument that can be connected to various modules to be connected or paired with a corresponding surgical hub. The modular device includes, for example, intelligent surgical instruments, medical imaging devices, suction / irrigation devices, smoke evacuators, energy generators, ventilators, insufflators, and displays. The modular device described herein can be controlled by a control algorithm. The control algorithm can be executed on the modular device itself, on a surgical hub paired with a specific modular device, or on both the modular device and the surgical hub (e.g., via a distributed computing architecture). In some examples, the control algorithm of the modular device controls the device based on data sensed by the modular device itself (i.e., by sensors in the modular device, sensors on the modular device, or sensors connected to the modular device). The data can be related to the patient being operated on (e.g., tissue properties or insufflation pressure) or the modular device itself (e.g., the rate at which the cutter is advanced, the motor current, or the energy level). For example, the control algorithm of the surgical suturing and cutting instrument can control the rate at which the motor of the instrument drives its cutter through the tissue according to the resistance encountered by the cutter when it advances.

[0100] Figure 6 One form of a surgical system 1200 is described that includes a modular energy system 1000 and various surgical instruments 1204, 1206, 1208 that can be used with the modular energy system, wherein the surgical instrument 1204 is an ultrasonic surgical instrument, the surgical instrument 1206 is an RF electrosurgical instrument, and the multifunctional surgical instrument 1208 is a combination ultrasonic / RF electrosurgical instrument. The modular energy system 1000 may be configured for use with a variety of surgical instruments. According to various forms, the modular energy system 1000 may be configured for use with different surgical instruments of different types including, for example, an ultrasonic surgical instrument 1204, an RF electrosurgical instrument 1206, and a multifunctional surgical instrument 1208 that integrates RF energy and ultrasonic energy delivered separately or simultaneously from the modular energy system 1000. Although in Figure 6 In one form, the modular energy system 1000 is shown as separate from the surgical instruments 1204, 1206, 1208, but in one form, the modular energy system 1000 can be integrally formed with any of the surgical instruments 1204, 1206, 1208 to form an integrated surgical system. The modular energy system 1000 can be configured to enable wired or wireless communication.

[0101] The modular energy system 1000 is used to drive a plurality of surgical instruments 1204, 1206, 1208. The first surgical instrument is an ultrasonic surgical instrument 1204 and includes a handpiece 1205 (HP), an ultrasonic transducer 1220, a shaft 1226, and an end effector 1222. The end effector 1222 includes a clamp arm 1240 and an ultrasonic blade 1228 acoustically coupled to the ultrasonic transducer 1220. The handpiece 1205 includes a trigger 1243 for operating the clamp arm 1240 and a combination of toggle buttons 1234a, 1234b, 1234c for energizing and driving the ultrasonic blade 1228 or other functions. The toggle buttons 1234a, 1234b, 1234c can be used to energize the ultrasonic transducer 1220 using the modular energy system 1000.

[0102] The modular energy system 1000 is also used to drive a second surgical instrument 1206. The second surgical instrument 1206 is an RF electrosurgical instrument and includes a handpiece 1207 (HP), a shaft 1227, and an end effector 1224. The end effector 1224 includes electrodes in the clamping arms 1242a, 1242b and returns through the electrical conductor portion of the shaft 1227. The electrodes are coupled to and powered by a bipolar energy source within the modular energy system 1000. The handpiece 1207 includes a trigger 1245 for operating the clamping arms 1242a, 1242b and an energy button 1235 for actuating an energy switch to power the electrodes in the end effector 1224.

[0103] The modular energy system 1000 is also used to drive a multifunctional surgical instrument 1208. The multifunctional surgical instrument 1208 includes a handpiece 1209 (HP), a shaft 1229, and an end effector 1225. The end effector 1225 includes an ultrasonic blade 1249 and a clamping arm 1246. The ultrasonic blade 1249 is acoustically coupled to an ultrasonic transducer 1220. The ultrasonic transducer 1220 can be separate from the handpiece 1209 or integrated with the handpiece. The handpiece 1209 includes a trigger 1247 for operating the clamping arm 1246 and a combination of toggle buttons 1237a, 1237b, 1237c for energizing and driving the ultrasonic blade 1249 or other functions. Toggle buttons 1237a , 1237b , 1237c may be used to power the ultrasonic transducer 1220 using the modular energy system 1000 , and power the ultrasonic blade 1249 using a bipolar energy source also contained within the modular energy system 1000 .

[0104] The modular energy system 1000 may be configured for use with a variety of surgical instruments. According to various forms, the modular energy system 1000 may be configured for use with different surgical instruments of different types, including, for example, an ultrasonic surgical instrument 1204, an RF electrosurgical instrument 1206, and a multi-function surgical instrument 1208 that integrates RF energy and ultrasonic energy delivered separately or simultaneously from the modular energy system 1000. Although Figure 6 In one form, the modular energy system 1000 is shown as being separate from the surgical instruments 1204, 1206, 1208, but in another form, the modular energy system 1000 can be integrally formed with any of the surgical instruments 1204, 1206, 1208 to form an integrated surgical system. Additional aspects of a generator and surgical instrument for digitally generating an electrical signal waveform are described in U.S. Patent Publication No. US-2017-0086914-A1, which is incorporated herein by reference in its entirety.

[0105] Additional information about modular energy systems can be found in U.S. patent application serial number 17 / 217394, filed on March 30, 2021, entitled “METHODFOR MECHANICAL PACKAGING FOR MODULAR ENERGY SYSTEM,” which is hereby incorporated by reference in its entirety.

[0106] Tissue tension

[0107] Having described the general implementation of the modular energy system 1000 and some surgical instruments that may be coupled thereto, the present disclosure now turns to the issue of tissue tension in surgical devices and the ability to calculate tissue tension. Tissue tension is an important parameter for the performance of both energy devices and endomechanical devices, and if not used properly, this tissue tension may negatively affect the quality of the seal or suturing. In other words, tissue tension affects all surgical devices that are clamped on tissue, such as staplers, clip applicators, ultrasonic devices, suturing devices, bipolar devices, monopolar devices, and graspers. Having poor control on the device may result in greater tissue tension, which may damage the tissue being treated and negatively affect the surgical outcome. Currently, many surgical devices (such as handheld surgical devices) do not have the ability to determine tissue tension.

[0108] The present disclosure provides various solutions for calculating tissue tension and alerting the user of the current tissue tension. This can allow surgeons to maintain low tissue tension, thereby helping them produce beneficial surgical outcomes. The control circuit of the surgical device can be coupled to a sensor that provides information to the control circuit. The information can allow the control circuit to determine tissue tension and alert the user. The control circuit can calculate tissue tension quantitatively or qualitatively. Some qualitative examples can be "low", "medium", "high", "lower" / "decreasing" or "higher" / "increasing". The user can be alerted to tissue tension through real-time visual or auditory feedback. In some instances, feedback about tissue tension can also be provided to the user after surgery.

[0109] Figure 7 A postoperative surgical video rating system is described that allows an unbiased observer to use the rating system 1600 to rate how well a surgeon performed in a surgical video recording. The rating system 1600 covers bimanual dexterity 1610, efficiency 1620, force sensitivity 1630, and robotic control 1640. The goal of the rating system 1600 is to provide surgeons with unbiased feedback about their performance so that the surgeon can improve surgical outcomes. Providing surgeons with real-time feedback about tissue tension during surgery can help surgeons improve their force sensitivity 1630 rating. Having high tissue tension can especially cause tissue tearing and / or damage to nearby structures, which can reduce the surgeon's force sensitivity rating 1630. If the surgeon is suturing, high tissue tension can cause suture breakage to occur frequently, thereby also reducing the force sensitivity rating 1630.

[0110] Figures 8 to 10 A view 1390 is shown provided to a surgeon while the surgeon is performing a surgical procedure. The surgical device 1300 has a shaft 1310 that is inserted into a patient. The shaft 1310 terminates at an end effector 1320. The end effector 1320 has a first jaw 1322 and a second jaw 1324. The first jaw 1322 is configured to rotate from an open configuration to a closed configuration relative to the second jaw 1324. The shaft 1310 of the surgical device 1300 has a label 1312 thereon detailing the type of surgical device. For example, in Figures 8 to 10In the embodiment, the surgical device is an ultrasonic surgical device having a tag 1312 ultrasound. For the surgical device 1300, the first jaw 1322 can be a clamping arm and the second jaw 1324 can be a blade. However, if the surgical device is a stapler, the first jaw can be an anvil and the second jaw can accommodate a staple cartridge. The surgical device 1300 can include various sensors that provide information to a control circuit that controls the surgical device 1300. For example, the control circuit that controls the surgical device can be the hub 106, the control circuit 500, or the control circuit 760. In one example, the control circuit can be part of the surgical device. In an alternative example, the control circuit can be separate and communicatively coupled to the surgical device.

[0111] The surgical device 1300 may include a tissue sensor for detecting tissue disposed between the first jaw 1322 and the second jaw 1324. In one example, the tissue sensor may be a capacitive sensor that can detect tissue between the jaws 1322, 1324. In an alternative example, the tissue sensor may detect continuity between the two jaws 1322, 1324 and determine whether there is tissue between the jaws 1322, 1324. For example, the surgical device may use a return electrode to calculate continuity. In an example where the surgical device is a bipolar electrosurgical device, continuity may be calculated directly. The tissue sensor allows the control circuit to determine when there is tissue between the jaws 1322, 1324.

[0112] The surgical device 1300 may include a jaw closure sensor for detecting the end effector in a closed configuration. In one example, the jaw closure sensor may be a jaw closure switch, wherein the switch is activated when the jaws 1322, 1324 are closed. In an alternative example, the jaw closure sensor may be a jaw closure mechanism positioning sensor, wherein the positioning sensor may be used to determine the position of the first jaw 1322 during the closing process. For example, the positioning sensor may be used to determine when the jaws 1322, 1324 are closed.

[0113] The surgical device 1300 may include a motion sensor to detect the motion of the end effector 1320 or the motion of the device 1300. For example, the motion sensor may detect the end effector motion, including orientation, position, velocity, and acceleration. In various examples, the motion sensor may include an accelerometer, a gyroscope, an inertial measurement unit, etc. The motion sensor may also be a combination of multiple sensors that provide motion of the end effector. One or more motion sensors may be mounted in or on the surgical device 1300.

[0114] The control circuitry may also determine when the surgical device 1300 is in the hands of the surgeon. In some instances, the control circuitry may use a motion sensor to determine when the surgical device 1300 is in use. In an alternative instance, the control circuitry may use video analytics from a camera viewing the surgical site to determine when the surgical device 1300 enters the field of view. For example, the video analytics may detect a tag 1312 on the device 1300 when the device 1300 is in the field of view. As another example, the video analytics may detect the geometry of the device 1300 to detect when the device 1300 is in the field of view.

[0115] refer to Figure 8 , the end effector 1320 of the surgical device 1300 is in an open configuration at a first time t1 during a surgical procedure. Fig. 9 , the end effector 1320 has closed on the tissue 1330 at a second time t2 during the surgical procedure. The first time t1 occurs before the second time t2. The end of the device 1300 is located at 1326. Fig. 9 As can be seen in FIG. 1 , there is tension on the tissue from a ridge 1332 of the tissue. Point 1340 is a location on the tissue that can be monitored to see if the tissue moves after the tissue is segmented. Point 1350 is a point on the surgical device 1300 that can be monitored to see how the device moves after the tissue is segmented. Fig.10 , tissue has been cut and sealed at a third time t3 in the surgical procedure. The second time t2 occurs before the third time t3. It can be seen that after the tissue segmentation, the device 1300 suddenly moves. This is shown by point 1350 moving to point 1352. Additionally, the point on the tissue moves from 1340 to 1342, which indicates that the tissue also moves. When the tissue is clamped in the device 1300, these movements can indicate tissue tension. The movement of the device 1300 can be captured and / or measured and the event marked for surgical analysis.

[0116] according to Fig.11, the control circuit can monitor the movement of the device as an indicator of tissue tension. In various instances, the control circuit can save computing resources by monitoring the device for tissue tension when the device is clamped on tissue. The control circuit process 1700 begins at 1702. The control circuit determines whether the surgical device is in use 1704. If the surgical device is in use, the control circuit continues to determine whether the end effector is in a closed configuration 1706. If the surgical device is in a closed configuration, the control circuit continues to determine whether the jaws are clamped on tissue 1708. If the jaws are clamped on tissue, the control circuit continues to monitor the movement of the device 1710. If the surgical device is not in use, the end effector is not in a closed configuration, or the jaws are not clamped on tissue, the control circuit will start 1702 process 1700 again. This allows the control circuit to monitor the movement of the device only when the surgical device is clamped on tissue, such as when Fig. 9 The control circuitry can monitor the movement of the device and look for any gross or sudden movement that may indicate tissue tension 1712. For example, the control circuitry can detect movement of the end effector outside of a predetermined range. The movement of the device can be calibrated based on a known distance from the motion sensor to the end of the device tip.

[0117] This monitoring of movement includes monitoring the movement of the device at the end of activation when tissue is separated or released. The end of activation can be detected by changes in the tissue sensor. The end of activation can also be detected by the user deactivating or changing the frequency of the ultrasonic device. Throughout the process of monitoring the movement of the device, the control circuit can determine whether there is tension 1714 on the tissue. For example, sudden movement of the device after tissue segmentation can indicate tissue tension. The control circuit then continues to provide feedback to the user in the heads-up display on tissue tension and / or sudden or rough movement. For example, the feedback can be superimposed on the camera view 1390. The feedback can also be auditory. In either instance, the feedback can be provided to the surgeon in real time, allowing the surgeon to correct their movements to benefit the surgical outcome.

[0118] This feedback can also be provided to the surgeon after the procedure. For example, the movement of the device throughout the surgical procedure can be provided to the user. Fig.12 , a diagram 1400 showing the movement of the tip of a surgical device during a surgical procedure can be provided to the surgeon. This diagram can provide the surgeon with a view of how they move the device and whether there are sudden or rough movements. Diagram 1400 shows the movement of a powered surgical stapler 1420 performing a surgical procedure and the movement of a non-powered surgical stapler 1410 performing a similar surgical procedure. It can be seen that the powered surgical stapler provides 37% less movement due to the push button firing, allowing the surgeon to be more consistent between surgical procedures.

[0119] Additionally, sudden or rough movements can be captured, measured and events marked for surgical analysis. Rating system 1600 can also be used to provide surgeons with an overall rating of their performance after surgery. In various instances, motion feedback can be divided into categories such as movement, movement when contacting tissue, movement when clamped on tissue, and movement immediately after tissue segmentation or release. These categories can allow surgeons to observe the movement of the device during different aspects of surgery and see where they need to improve. Motion feedback can also be displayed during specific aspects of surgery. Whether the surgeon continues to have sudden movements. In general, mobile feedback can be used to guide and enhance surgeon skills in real time or in a postoperative report.

[0120] refer to Fig.13 , surgical device 1300 may include one or more reference markers 1362, 1364, and 1366. In various examples, reference markers 1362, 1364, and 1366 may be rings around axis 1310. Fig.13 , each fiducial marker is a black ring around axis 1310, wherein fiducial marker 1362 is away from fiducial marker 1364, which is away from fiducial marker 1366. In an alternative example, one or more fiducial markers can be any mark on device 1300 that can be used by the control circuit. Fiducial markers 1362, 1364, and 1366 can be used to provide device identification. For example, the size and spacing of fiducial markers 1362, 1364, and 1366 can be used by the control circuit to identify a surgical device or surgical device type. Additionally, device type identity information can be obtained by means of Bluetooth or direct wired communication with an OR system (e.g., an energy generator). However, when there are multiple surgical devices in the camera field of view, one or more fiducial markers on the device can help identify the device. Label 1312 can also be used to identify multiple devices in the field of view. It is also possible to use data matrices, QR codes, etc. to identify multiple devices in the field of view.

[0121] The control circuitry can use the video data from the camera to calculate tissue tension during the surgical procedure. When performing real-time video analysis, there may be issues with computational burden. However, this burden can be mitigated by analyzing small locations within the video image and performing the analysis only within a specific time window based on events of interest. For example, an event of interest may be when tissue is clamped in the jaws 1322, 1324 of the end effector 1320.

[0122] refer to Fig.14, fiducial markers 1362, 1364, and 1364 may be used to define a vector (orientation and direction) in three-dimensional space, which may be used to define the position of the end 1326 of the device 1300. In various examples, the maximum range of each ring may be used to generate a vector in three-dimensional space, such as the ends 1362a, 1362b of fiducial marker 1362, the ends 1364a, 1364b of fiducial marker 1364, and the ends 1366a, 1366b of fiducial marker 1366. The position of the end 1326 of the device 1300 in the video data may be calculated based on a priori knowledge of the distances between the fiducial markers 1362, 1364, and 1366 and the end 1326 of the device 1300. In an alternative example, the control circuit may perform an analysis of the video data to locate the end of the surgical instrument based on the color and geometry of the surgical device. The position of the end 1326 of the surgical device 1300 may be used to calculate the region of interest 1370 in the video image data. The region of interest 1370 can greatly reduce the computational burden by reducing the amount of space in the video that requires video analysis. Additionally, the computational burden can be further reduced by only performing video analysis when the surgical device 1300 is clamped on tissue.

[0123] The control circuit can perform video analysis to Fig.15 to calculate tissue tension. In various instances, the control circuitry can save computing resources by performing video analysis on a subset of the video data defined by a region of interest surrounding the end of the device. Additionally, the control circuitry can choose to only perform video analysis when the device is clamped on tissue, further reducing the computational burden. The control circuitry process 1800 begins by receiving current video data from a camera 1802 viewing the surgical site. The control circuitry determines whether the end effector is in a closed configuration 1804. If the end effector is in a closed configuration, the control circuitry proceeds to determine whether the jaws are clamped on tissue 1806. If the jaws are clamped on tissue, the control circuitry proceeds to calculate the distal tip position in the video data 1808. If the end effector is not in a closed configuration or the jaws are not clamped on tissue, the control circuitry will begin process 1800 again by receiving the next batch of video data from the camera 1802. This allows the control circuitry to perform video analysis only when the surgical device is clamped on tissue, such as when Fig.14 middle.

[0124] This can be accomplished by using one or more fiducial markers on the device, such as fiducial markers 1362, 1364, and 1366 ( Fig.14)) to calculate the distal tip position in the video data. One or more fiducial markers can be used to generate a vector in three-dimensional space. The position of the device tip in the video data can be calculated based on a priori knowledge of the distance between the one or more fiducial markers and the tip of the device. In an alternative example, a motion sensor can be used to calculate the tip of the device in three-dimensional space.

[0125] Once the device tip location is known, the control circuitry can generate a region of interest around the device tip location in the video data 1810. An example region of interest in the video data is Fig.14 1370 is shown in FIG. 13. This process reduces the amount of data that needs to be processed by video analysis because the region of interest is only a subset of the video data. Once the region of interest is known, the control circuit can perform video analysis on the video image data to calculate tissue tension. Once the tissue tension is known, the control circuit can provide tissue tension feedback to the user. For example, if tissue tension is detected, a color-coded tension alarm (such as tissue tension alarm 1380) can be superimposed on the video monitor. The color scale can relate to a quantitative or qualitative rating system. For example, the color scale can relate to a qualitative scale of "low", "medium", or "high" tissue tension. In additional or alternative examples, the tissue tension alarm can be audible.

[0126] refer to Fig.14 , video analysis can calculate tissue tension by observing tissue 1330 in the region of interest 1370. Fiducial markers 1362, 1364, and 1366 provide video information about the position and orientation of the device tip 1326, which is used to "see" tension. Video analysis can look for ridges 1332 in the tissue 1330 on one or both sides of the clamped end effector 1320. The ridges 1332 of the tissue 1330 can indicate the presence of tension and the potential degree of tension. This information can then be provided to the surgeon through visual and / or auditory feedback.

[0127] Fig.16 and Fig.17 Two graphs of tissue tension are shown. Fig.16 shows axial tissue tension, and Fig.17Radial tissue tension is shown. Video analysis can use at least one fiducial marker added to the device, such as fiducial markers 1362, 1364, or 1366, to establish at least two points 1522, 1524 on the grasped tissue adjacent to the marker at some set relative distance. Using these points 1522, 1524, the system can establish two lines 1526, 1528, one line 1526 extending toward the grasped tissue on the left side of the end effector and one line 1528 extending toward the grasped tissue on the right side. The two lines intersect at point 1512, and the angle between the two lines can be used to assess the extent to which the tissue is pulled axially or radially by the device. Fig.16 Axial tension is shown, with angle a1 showing no tension, angle a2 showing slight tension, and angle a3 showing greater tension. Fig.17 Radial tension is shown, where angle b1 shows no tension, angle b2 shows slight tension, and angle b3 shows greater tension. When greater tension is applied to the tissue, the degree to which angles "a" and "b" change increases. For example, the change from angle a1 to angle a2 is less than the change from angle a1 to a3. Similarly, the change from angle b1 to angle b2 is less than the change from angle b1 to b3.

[0128] Tissue tension information can be translated to the device user via visual (e.g., low-medium-high scale) or auditory (e.g., if too much tension is applied, an alarm tone plays) indicators. Relative changes within the video image range can be used to see tension development, such as how the angle and length of the edge changes. This information can be provided to the surgeon in real time to enable them to manage tissue tension during surgery with the goal of minimizing tissue tension.

[0129] Advanced Bipolar Seal Quality Prediction

[0130] Advanced bipolar ("ABP") tools are electrosurgical tools used in surgery for soft tissue dissection and vascular sealing, and provide the major benefit of reducing OR time and minimizing bleeding. These tools use radiofrequency current to form the heat required for sealing, but the resulting temperature is generally insufficient to cause tissue cutting. These ABP tools further utilize algorithms to ensure that the seal is complete, and utilize mechanically sharp knives, ultrasonic knives, or any other suitable knives to cut tissue after sealing. Combining sealing and cutting into a single device enhances the versatility of the ABP tool, thereby increasing the flexibility of the ABP tool, reducing the need for instrument exchange to enter and leave the body, and improving surgical efficiency, ease of use, and workflow.

[0131] Although dissection and sealing produce the desired results most of the time, some activations of the ABP tool may still result in minor or major intraoperative bleeding. The surgeon can use information about the quality of the seal to inform subsequent surgical maneuvers that will help prevent or mitigate bleeding, thereby reducing patient bleeding and improving surgical efficiency.

[0132] The present disclosure provides a solution that can predict poor seal quality with high bleeding probability in real time before bleeding occurs. These predictions can be performed using electrical measurements and other measurements throughout the activation and additional device data during or after energy activation. When poor seal quality is detected, the system can issue a warning (auditory, visual or tactile or a combination thereof) that allows the surgeon to decide what appropriate next step should be taken to avoid bleeding, such as not mechanically cutting tissue with a blade. The surgeon can also choose to reapply energy or perform other actions before transection to prevent bleeding.

[0133] Seal quality prediction is based on an inference model that predicts the likelihood of bleeding in near real time based on data from the generator. The inference module is based on a pattern recognition model trained on annotated real-world data from surgeries on humans. The module can be deployed on digital hardware in the operating room with direct access to device data.

[0134] In some embodiments, the system utilizes time series electrical data (current, voltage, impedance, power, sealing cycle phase, total energy, etc.) throughout the activation of the instrument, as well as device manufacturing data (clamping force, multi-point jaw gap measurements, device electrical impedance, etc.) as the basis for making predictions about seal quality. Information about seal quality is communicated to the surgeon, who can then select an appropriate response. The surgical workflow is described in the "Concepts" section below.

[0135] In order to develop the algorithm, data is needed, which is described in the "Data Collection and Annotation" section below. This data is preprocessed into features in real time through many different mathematical transformations and scaling operations described in the "Feature Generation" section below. As an example, preprocessing can produce hundreds or thousands of features, which are fed into machine learning models that take inputs and generate numerical predictions, such as a variable scale of 0 to 1. In one embodiment, a threshold is applied to this output, such as>0.5, as an example, this output can be transformed into a binary prediction for transmission to the user. Machine learning models are described in the "Machine Learning ("ML") Model" section below.

[0136] In some embodiments, a threshold value can be selected for a particular model to appropriately balance the sensitivity and specificity of the prediction. In order to optimize the efficacy of the solution in the context of surgical workflow, it is important to offset false positives (i.e., false alarms that lead to alarm fatigue) and false negatives (i.e., missed bleeding events that lead to bleeding). This is described in the "Measurement" section below. Finally, this information must be transmitted to the surgeon. The electrosurgical system can provide its own interface or work in conjunction with another system. This is described in the "Integration and Interface" section below.

[0137] concept

[0138] As mentioned above, in some embodiments, the system utilizes time series electrical data throughout the activation of the instrument (current, voltage, impedance, power, sealing cycle phase, total energy, etc.) as well as device manufacturing data (clamping force, multi-point jaw gap measurements, device electrical impedance, etc.) as the basis for making predictions about seal quality. Information about seal quality is communicated to the surgeon, who can then select an appropriate response.

[0139] Reference now Fig.18 , a flow chart 2000 according to at least one aspect of the present disclosure is provided. During a surgical procedure, the clamping jaws of an electrosurgical tool (such as surgical instruments 1204, 1206, 1208) can grasp tissue 2002. The jaws and grasped tissue can be viewed on a display (such as display 135, display 711, or any other suitable display described elsewhere herein). Once the desired tissue has been grasped, a generator such as generator module 140, RF energy source 794, modular energy system 1000, or any other suitable energy generator described elsewhere herein can provide RF energy to the electrodes of the electrosurgical tool, which can then apply RF energy to the tissue to seal the tissue. Based on the application, a control system (such as processor module 132, processor 502, control circuit 760, or any other suitable control system described elsewhere herein) can predict 2003 the quality of the seal and determine an appropriate response, as will be described in more detail below. Based on the prediction, the control system can provide feedback to the surgeon indicating the type quality of the seal, such as visual feedback, auditory feedback, or tactile feedback, or a combination thereof.

[0140] In one aspect, when the control system predicts 2004 that the seal is a good quality seal, the control system can provide feedback to the surgeon indicating this situation. Based on the good quality seal indication, the surgeon can confidently continue to use the knife of the electrosurgical tool (such as using knife 768, ultrasonic knife 1249, ultrasonic knife 1228 or any other knife or knife described elsewhere herein) to cut 2006 the sealed tissue. In various other embodiments, when the control system predicts 2004 that the seal has a good seal quality, the control system may not provide feedback to the surgeon. The lack of feedback can notify the surgeon that the electrosurgical tool performs as expected, that is, the seal applied has good quality. The lack of feedback provides the advantage of preventing the surgeon from being distracted by feedback that does need to take corrective action. Once the tissue has been cut by the knife, the surgeon can proceed 2008 to the next surgical task.

[0141] In one aspect, when the control system predicts 2010 that the seal is a poor quality seal, the control system can provide feedback to the surgeon indicating this situation. Based on the poor seal quality indication, the surgeon can decide 2012 the appropriate course of action to be taken. In one aspect, the surgeon can pause 2014 the actuation of the electrosurgical tool to evaluate the sealed tissue before continuing. In one aspect, the surgeon can release 2016 the tissue captured by the jaws of the electrosurgical tool and re-grasp the tissue or grasp different tissue. In one aspect, the surgeon can apply 2018 additional treatment energy to the grasped tissue, thereby further sealing the tissue before continuing. In one aspect, the surgeon can prepare 2020 auxiliary tools, such as different energy instruments that can be used to form hemostasis within the tissue.

[0142] Data collection and annotation

[0143] As described above, in order to develop an algorithm that can achieve the above-described predictor of seal quality, data is needed. To predict the quality of the seal, the control system can utilize a machine learning algorithm that is trained, optimized, and tested using data (both input data and output data) obtained from a variety of sources, such as clinical sources, preclinical sources, or bench sources, or a combination thereof. In one aspect, the closer the data represents actual use, the more changes that can be introduced during the training of the algorithm, and the more robust the algorithm will perform during use in a surgical procedure.

[0144] In some embodiments, the data used to train the algorithm can be obtained from clinical, real-world sources. In one aspect, the data can be clinical, real-world data (both input data and output data) obtained from the same or similar hardware components used on humans. Utilizing clinical data allows the algorithm to be trained with data that will very closely match the data encountered by the algorithm during surgery. In one aspect, as an example, the input data can include time series electrical data from an electrosurgical generator, internal system event data, and device manufacturing data.

[0145] In various embodiments, the output data includes output obtained from surgical videos that were recorded and subsequently annotated in post-processing. Video annotation is the process of applying labels to surgical videos so that an algorithm can extract structured information from the videos. In one aspect, such as FIG. 19A to FIG. 19C Tags such as those shown in table 2050 of can be added to the video by the operator based on the tag definitions and the operator's experience. In various embodiments, the tags can be stored in a memory such as storage array 134, memory 504, or any other suitable memory described elsewhere herein. In various embodiments, the tags can be applied to the surgical video by the operator at an input interface such as a computer, a touch screen monitor, etc.

[0146] For seal quality prediction, it is important to know whether the tissue bleeds after each activation (i.e., "hemostasis outcome"). Therefore, in some embodiments, a "bleeding" or "drying" label is applied to each activation, which is used as the basic fact for training the algorithm. In some embodiments, the bleeding to dry label is converted into a binary value for training the algorithm. In other embodiments, the qualitative estimate of bleeding (e.g., dry, exudative, mild, severe) can be encoded as ordinal data for multi-class algorithm training. Other labels obtained by video annotation (such as, for example, tissue thickness, tissue type, tissue adhesion) can be used during training to assist multi-task learning. In one aspect, multi-task learning is a machine learning algorithm trained to predict multiple outputs, which improves the accuracy of seal quality prediction by utilizing additional tissue data not included in the bleeding / drying label.

[0147] In some embodiments, the data used to train the algorithm can be obtained from bench testing. Augmenting real-world data with bench tissue data improves algorithm accuracy by extending conditions beyond human clinical use. For example, creating bench data with devices that have low-tolerance components and high-tolerance components (at the edge of tolerance or out of tolerance) enhances the robustness of the algorithm by allowing the bench data to learn to better compensate for these factors and identify patterns in electrical data for out-of-tolerance devices. This improves the accuracy of predictions even on nominal component tolerance devices.

[0148] In some embodiments, the data used to train the algorithm can be obtained from preclinical, live animal testing. Similar to adding benchtop test data, as described above, live animal data can further expand the training and test data sets. Adding animal data has the benefit of achieving good algorithm performance during development and subsequently passing validation and verification activities.

[0149] As mentioned above, the machine learning model can be trained to predict multiple outputs using multi-task learning to improve the accuracy of seal quality prediction. Fig. 20 , according to at least one aspect of the present disclosure, a flowchart 2100 of multi-task learning during model training and verification is provided. Model 2102 can receive features (inputs) 2104 described in more detail above, and can output hemostasis 2106 of the tissue, tissue type 2108, and whether there is tissue adhesion 2110. In various embodiments, as described above, these outputs can include outputs obtained from surgical videos that are recorded and then annotated in post-processing. In various embodiments, the hemostasis of the tissue 2106 can be assigned a numerical value within a numerical range based on the quality of the seal. For example, the numerical range can include a minimum value (such as 0) and a maximum value (such as 4). These outputs can be fed to a training module 2112, which can then be fed back into the model 2102 (i.e., back propagated) to further train the model 2102 and improve the accuracy of the seal quality prediction.

[0150] Reference now Fig.21 , according to at least one aspect of the present disclosure, a flowchart 2200 of multi-task learning during validation, verification, and clinical use is provided. Model 2202 can receive features (inputs) 2204 described in more detail above, and can output hemostasis 2206 of the tissue, tissue type 2208, and whether there is tissue adhesion 2210. In various embodiments, the hemostasis of the tissue 2206 can be assigned a value within a range of values ​​based on the quality of the seal. For example, the range of values ​​can include a minimum value (such as 0) and a maximum value (such as 4). Based on the value, the system can make a prediction 2212 of the quality of the seal.

[0151] Feature Generation

[0152] As mentioned above, an algorithmic model may receive features (inputs) that may be used to train a machine learning algorithm.

[0153] In various embodiments, the features may include algorithm-based features. In one aspect, the features may be Fig. 222300 to control the ABP generator algorithm. In various embodiments, the lookup table 2300 may be stored in a memory, such as a memory, such as the memory array 134, the memory 504, the memory 524, or any other suitable memory described herein. During activation, a given lookup table location, CLC code (segment), and index (sub-segment) value is determined based on a combination of time and impedance. By segmenting the electrical data based on which part of the control algorithm the activation is in, it can be determined how well the algorithm is performing.

[0154] For feature generation, the generator file is split into CLC codes and subsequently into smaller chunks inside each CLC code. This can be done in Fig. 22 2304, 2306, 2308 is a new CLC code number, and the vertical lines show how each CLC code is further segmented. Summary statistics (maximum, minimum, mean, standard, maximum / minimum ratio) are then taken from these smaller blocks and used as features for the machine learning model. This is done for a combination of electrical parameters time, voltage, current, impedance, power and energy (t, V, I, Z, P, E). In various embodiments, global (fully activated) summary statistics are also used as features. In various embodiments, the amount of time spent in each CLC index is also used as a feature. The resulting vectors are then combined and stacked to create a vector for each activation.

[0155] In various embodiments, features may include wavelet-based features. In one aspect, wavelets (continuous and discrete) are a signal processing technique that takes a "mother" wave, scales it, and translates it across a given signal. Unlike sine waves, which are not localized in time, wavelets are localized in time. This allows the wavelet transform to obtain time information in addition to frequency information. Because wavelets are localized in time, a signal can be multiplied with the wavelet at different locations in time, a process also known as convolution, as can be seen in Fig.23 After this operation is completed for the original (mother) wavelet, the original (mother) wavelet can be scaled so that it becomes larger and the process is repeated. Fig.24 The resulting transform 2320 shown is similar to a Fast Fourier Transform ("FFT"); however, the transform provides context for the signal in both the frequency and time domains.

[0156] As mentioned above, wavelets may include discrete wavelets and continuous wavelets. A discrete wavelet transform ("DWT") and a continuous wavelet transform ("CWT") of the original signal are performed and used as features. In some embodiments, for both the DWT and the CWT, the mother wavelet used is "db4". In some embodiments, for the CWT, 128 scalings of the mother wavelet are used, and the resulting coefficients and frequencies of the transform are stacked and zero-filled to create an even length feature array. In some embodiments, for the DWT, summary statistics and entropy calculations of the resulting coefficients are used instead of pure coefficients. An example of a wavelet transform (power spectrum) of a signal is shown in FIG. Fig.25 This is shown in the curve graph 2330 in .

[0157] In various embodiments, the features may include time-based features. Fig.26 As shown in the graph 2400 of FIG. 2400 , the ABP electrical parameters voltage, current, impedance, power and energy (V, I, Z, P, E) can be divided into “x” time length segments 2402. Summary statistics (maximum, minimum, average, normal, maximum / minimum ratio) are then taken from these segments. The resulting vectors are then stacked to create one vector, as shown in the graph 2400 .

[0158] In various embodiments, the features may include raw signal features. The raw signal from the generator may be parsed into 5 vectors - voltage, current, impedance, power and energy (V, I, Z, P, E) - which may then be stacked on top of each other and zero padded, with zeros added to the end of each signal to ensure that each vector for each activation is the same length. In one aspect, each signal may have 540 numbers, where 540 is the theoretical maximum length of an activation (5.4 seconds).

[0159] Machine Learning ("ML") Models

[0160] As mentioned above, preprocessing can produce hundreds or thousands of features that are fed into machine learning models that take inputs and generate numerical predictions. In one aspect, with machine learning ("ML") models, the main idea of ​​sampling-based approaches is to modify the distribution of events so that rare classes are well represented in the training samples. Assuming that bleeding conditions represent only about 10% of all conditions, various class balancing techniques are applied to the machine learning models.

[0161] In the case of undersampling, a random sample is taken from the majority class (i.e., non-bleeding events). However, a potential problem with undersampling is that some of the useful non-bleeding instances may not be selected for training, and the classifier will not be optimal. In the case of oversampling, replication of events is taken from the minority class, i.e., bleeding cases. However, a potential problem with oversampling is that this technique can lead to overfitting of noisy data that will be replicated many times. This can lead to poor model generalization.

[0162] Therefore, the present disclosure provides a hybrid method of oversampling and undersampling, which is referred to herein as the synthetic minority oversampling technique ("SMOTE"), which uses feature space similarity to create artificial minority class data. SMOTE is described in more detail in the paper entitled "SMOTE: Synthetic Minority Over-sampling Technique" by Notesh Chawla et al., published in June 2002, which is hereby incorporated by reference herein. The advantage of SMOTE is that when synthetic instances are generated rather than copies of instances, SMOTE alleviates overfitting caused by random oversampling. In addition, there is no information loss. In one aspect, reference Fig.29 As shown in the representation 2430, SMOTE can be implemented by taking the difference between a sample point and one of its nearest neighboring sample points, multiplying the difference by a random number between 0 and 1, and adding the multiplied number to the feature vector. This results in the selection of a random point along the line segment between two specific features.

[0163] Reference now Fig. 27 , providing a scatter plot 2410 of the data set, which shows a large number of points belonging to the majority class (black dots, "0") and a small number of points scattered in the minority class (bright dots, "1"). SMOTE is used to oversample the minority class and undersample the majority class, and then plotted, resulting in Fig.28 Scatter diagram 2420 shown in .

[0164] 224 summary statistics were obtained using a variety of methods involving time-based, algorithmic, wavelet, and raw features. Principal component analysis ("PCA") was performed to extract hidden (potentially lower dimensional) structure from a high dimensional feature space. PCA is an orthogonal projection or transformation of the data into a (possibly lower dimensional) subspace such that the variance of the projected data is maximized. Identifying the axes is called principal component analysis and can be obtained using classical matrix computational tools (eigen or singular value decomposition). References Fig.30 , where PCA is shown to transform features from one state 2440 to a second state 2442 such that only latent features need be retained.

[0165] In some aspects, for the ML model, a predictive model comprising a hybrid of a linear classifier and a tree-based classifier can be used. The model framework has an input of a data frame of size [n, 224], where 224 represents summary statistics for electrical signals obtained from feature engineering. In one aspect, preprocessing is implemented using SMOTE or PCA, both of which are described above, as examples. Tissue classification includes hemorrhage or dryness.

[0166] For logistic regression, a binary classification model using a logistic (sigmoid) function to model probability was used, including a regularization model where α equals 0.5 and β equals 0.5. Hyperparameter tuning was used to obtain optimal values ​​for α and β. A support vector machine ("SVM") algorithm was used to find a hyperplane in N-dimensional space (N - number of features) that discriminately classifies the data points. A linear SVM was fit using L2 regularization. Nonlinearity was modeled using a radial kernel function. References Fig.31 , using Xgboost 2450, which is a machine learning library built around an efficient implementation of boosting for tree models such as GBM. Xgboost allows the model to be tuned back on multiple parameters such as the number of trees (300), maximum depth (3, 6, 8), metric (binary logloss), L1 regularization (0, 1, 2), L2 regularization (1, 2), number of iterations (100, 150, 250), and boosting type ("Dart", "gbtree"). More information about XGBoost is described in the paper entitled "Comparative Analysis of Artificial Neural Network and XGBoost Algorithm for PolSAR Image Classification" published by Nimra Menon et al. in November 2019, which is hereby incorporated by reference in its entirety.

[0167] Other models used are CNN models which are deep learning classification models that can be trained to classify patterns by extracting informative aspects of the signal via learnable weights and biases describing multiple layers of convolutions. In one aspect, the 1D CNN model can include an input of five [1×540] vectors describing the electrical signal. Preprocessing can be a [1×7] median filter, standard scaling. Reference Fig.32A and Fig.32B, the model architecture 2460 may include Conv1D(8)>Dropout(0.2)>Max(2)>Conv1D(16)>Dropout)(0.2)>Max(2)>Conv1D(32)>Dropout(0.2)>Max(0.2)>Conv1D(32)>Dropout(0.2)>Max(0.2)>Flatten>Dense(100)>Dropout(0.2)>Dense(2).

[0168] The other model used is a CNN+Dense multi-head architecture model as a deep learning classification model that can be trained to classify patterns by extracting informative aspects of the signal through learnable weights and biases describing multiple layers of convolutions and combining this information with other inputs. These other inputs are processed by a neural network with multiple densely connected layers. The outputs from multiple heads are combined and processed by a final stack of dense layers.

[0169] In one aspect, the model may include inputs of two [1×540] vectors describing the electrical signal and one [1×5] vector describing the mechanical parameters. Preprocessing may be a [1×7] median filter, standard scaling for the electrical signal, and standard scaling for the mechanical parameter vector.

[0170] measure

[0171] As described above, a threshold can be selected for a particular model to appropriately balance the sensitivity and specificity of the prediction. To optimize the efficacy of the solution in the context of surgical workflow, it is important to counteract false positives (i.e., false alarms that lead to alarm fatigue) as well as false negatives (i.e., missed bleeding events that lead to bleeding).

[0172] Reference now Fig.33 , a confusion matrix table 2470 that can be used to compare predicted results with actual results according to at least one aspect of the present disclosure is provided. In one aspect, the confusion matrix can be used to train the ML algorithm. For example, in the case where the tissue is predicted to be dry and it is actually dry, a true negative label "TN" is provided. In the case where the tissue is predicted to be dry and it is actually bleeding, a false negative label "FN" is provided. In the case where the tissue is predicted to be bleeding and it is actually dry, a false positive label "FP" is provided. In the case where the tissue is predicted to be bleeding and it is actually bleeding, a true positive label "TP" is provided.

[0173] The accuracy of the system is determined by the proportion of true results in the total number of cases examined. The equation for accuracy is provided as the number of TP markers and TN markers divided by the number of total actual number of cases. Given that the data in this case is unbalanced (approximately 10% bleeding events), accuracy fails to correctly capture the precision and sensitivity values ​​for bleeding events.

[0174] The ratio of predicted positives to true positives determines the accuracy of the system. The equation for accuracy is provided as the number of TP markers divided by the sum of TP markers and FP markers. In this case, the classifier showed a high accuracy value, however, the classifier failed to predict all hemorrhages.

[0175] The proportion of actual positives that are corrected for classification determines the call of the system. The equation for the call is provided as the number of TP markers divided by the sum of TP markers and FN markers. Optimizing the classifier model based on this metric produces a low precision score.

[0176] The F1 score helps to obtain a call precision score within the clinical viability threshold. The F1 score is a number between 0 and 1 and is the harmonic mean of precision and call. The equation for the F1 score is provided as 2*call*precision / (call+precision).

[0177] The area under the curve ("AUC") receiver operating characteristic ("ROC") curve indicates how well the probabilities from the positive class are separated from the probabilities from the negative class. The characteristic curve is plotted between the true positive rate values ​​and the false positive rate values, as Fig.34 As shown in the example curve graph 2480 in FIG. Assuming that the data set is skewed, the AUC ROC curve continues to score higher despite the lower accuracy of the minority class. For example, if the model overfits the majority class, the model may not be able to predict the minority class but still produce a high true positive rate ("TPR") value and a low false positive rate ("FPR") value. This will produce a high AUC score.

[0178] The target metric was developed in conversations with a variety of surgeons across North America and APAC. When discussing acceptable false positive rates (alarm fatigue) and false negative rates (missed bleeding events) with surgeons, Fig.35 The confusion matrix 2490 shown in FIG. 24 is used to help facilitate discussion and reach numerical target values. After explaining the overall concept of seal quality prediction to the surgeon, it is explained that there is a tradeoff between false positive rate and false negative rate for a given machine learning model. The surgeon then walks through FIG. 36A to FIG. 36D The four confusion matrices 2500, 2510, 2520, 2530 with various false positive and false negative rates are shown in and were asked to give their opinion on each and which scenario is preferred.

[0179] After all discussions were completed, responses were aggregated to determine clinical viability thresholds. It was determined that the tolerance for "alarm fatigue" (ie, false positives) was approximately 2 to 4 times higher than for "missed bleeding events" (ie, false negatives).

[0180] Integration and interfaces

[0181] As mentioned above, information must be communicated to the surgeon so that once the quality of the seal has been predicted, the surgeon can decide on an appropriate course of action. Communicating the prediction of seal quality can be done in many ways and a balance must be struck between being distracting or obtrusive to the surgeon, but also minimizing the possibility of missing information.

[0182] In one aspect, it would be advantageous to display a non-obtrusive message on a monitor such as monitor 14, the primary laparoscopic (lap) monitor, or any other monitor or display described elsewhere herein when a poor seal is predicted and not display any message when a good seal is predicted. Given that a good seal will occur most of the time, not displaying seal quality information in these cases will help minimize information overload and distraction.

[0183] In various embodiments, a visual method of displaying poor seal quality (display 2540, 2550 on the screen, such as Fig.37 and Fig.38 ) including some or all of binary feedback (good vs. bad or sealed vs. unsealed), continuous feedback (seal quality percentage, 0% to 100%, which seal quality percentage can be accompanied by a recommended threshold visually indicated by a color change), and / or discrete feedback (red, yellow, green indicator icons).

[0184] The first embodiment of the seal quality prediction system for the advanced bipolar device 2600 is described in Fig.39 , wherein seal quality inference is performed within an advanced bipolar generator 2610, which is similar to the generator module 140, the energy source 794, or the modular energy system 1000, or any other generator described elsewhere herein. The inference output is transmitted to a real-time image processing unit 2620 that overlays the seal quality information onto a laparoscopic video image 2650 captured using a laparoscopic camera 2630 and a video camera processing unit 2640, which is similar to the processor module 132, the processor 502, the control circuit 760, or any other suitable image processing unit described elsewhere herein. An advantage of this arrangement is that it will be compatible with any 3rd party laparoscopic video system.

[0185] Fig.40A second embodiment of a seal quality prediction system for an advanced bipolar device 2600 is provided in. The second embodiment is similar to the first embodiment described above, except that the advanced bipolar generator 2610 sends raw electrical data and activation metadata to a real-time inference and image processing unit (RTIIP) 2700 instead of to a real-time image processing unit 2620. Seal quality inference is performed within the RTIIP 2700, which then superimposes the seal quality information onto the laparoscopic video image. An advantage of this arrangement is that the RTIIP 2700 can be provided as a stand-alone product that is compatible with the energy system (generator and device). The inference feature can be provided as a software add-on. This can also eliminate the need for design changes to the energy system, which can have regulatory and R&D cost advantages.

[0186] In various embodiments, the system can also provide an auditory member for transmitting seal quality information. In some embodiments, an audio feedback module (such as any suitable audio feedback module described elsewhere herein) can emit a first sound effect during the sealing phase of device activation, emit a second sound effect in the case of high-quality seals, and emit a third sound effect in the case of poor-quality seals. Various other embodiments are contemplated, wherein no sound effects are emitted in the case of high seal quality so as not to distract the surgeon. In contrast, the audio feedback module only provides feedback in the case of poor seal quality.

[0187] In various embodiments, an energy device (such as instrument 112, surgical instrument 790, or surgical instruments 1204, 1206, 1208) can be designed with a haptic device (such as a piezoelectric motor) that emits a buzzing or clicking sound when a poor seal quality is predicted. Various embodiments are contemplated in which the haptic device provides haptic feedback in the case of a high-quality seal. Various other embodiments are contemplated in which the haptic device does not provide haptic feedback in the case of a high-quality seal so as not to distract the surgeon.

[0188] U.S. Patent Nos. 10,624,691, 10,842,523, 11,291,510, 11,311,342, 11,259,830, 11,304,699, 11,109,866, 11,298,129, 11,229,437, 11,241,235 and U.S. Patent Application Publication Nos. 2019 / 0206562, 20 No. 19 / 0200981, No. 2019 / 0208641, No. 2019 / 0201594, No. 2019 / 0201045, No. 2019 / 0200844, No. 2019 / 0201136, No. 2019 / 0206569, No. 2019 / 0201137, No. 2019 / 0125459, No. 2019 / 0125458, No. 2019 / 0125455 No. 2019 / 0125454, No. 2019 / 0274706, No. 2019 / 0201046, No. 2019 / 0201047, No. 2019 / 0104919, No. 2019 / 0125361, No. 2019 / 0200977, No. 2019 / 0298350, No. 2019 / 0206564, No. 2019 / 0206565, No. 2020 / 01 The entire disclosures of No. 00830, No. 2020 / 0078070, No. 2020 / 0078076, No. 2020 / 0078106, No. 2020 / 0100825, No. 2021 / 0196334, No. 2021 / 0196354, No. 2021 / 0196302, No. 2020 / 0345353, and No. 2022 / 0031315 are hereby incorporated by reference into this document.

[0189] Although various devices have been described herein in conjunction with certain embodiments, many modifications and variations of these embodiments may also be implemented. In one or more embodiments, specific features, structures, or characteristics may be combined in any suitable manner. Therefore, without limitation, specific features, structures, or characteristics shown or described in conjunction with one embodiment may be combined in whole or in part with features, structures, or characteristics of one or more other embodiments. In addition, where materials for certain components are disclosed, other materials may also be used. In addition, according to various embodiments, a single component may be replaced with multiple components, and multiple components may also be replaced with a single component to perform a given one or more functions. The above-described specific embodiments and the following claims are intended to cover all such modifications and variations.

[0190] The device disclosed herein can be designed to be discarded after a single use, or it can be designed to be used multiple times. However, in either case, the device can be used again after being conditioned after at least one use. Conditioning may include any combination of the following steps, including but not limited to disassembling the device, cleaning or replacing specific parts of the device, and then reassembling the device. Specifically, the conditioning facility and / or surgical team can disassemble the device, and after cleaning and / or replacing specific parts of the device, the device can be reassembled for subsequent use. Those skilled in the art will understand that conditioning the device can utilize various techniques for disassembly, cleaning / replacement, and reassembly. The use of such techniques and the resulting repair device are within the scope of this application.

[0191] The devices disclosed herein can be processed before surgery. First, new or used instruments can be obtained and cleaned as needed. Then, the instruments can be sterilized. In one sterilization technique, the instrument is placed in a closed and sealed container (such as a plastic or TYVEK bag). The container and the instrument can then be placed in a radiation field that can penetrate the container, such as gamma radiation, X-rays and / or high-energy electrons. Radiation can kill bacteria on the instrument and in the container. The sterilized instrument can then be stored in a sterile container. The sealed container can keep the instrument sterile until the container is opened in a medical facility. The device can also be sterilized using any other technology known in the art, including but not limited to beta radiation, gamma radiation, ethylene oxide, plasma peroxide and / or steam.

[0192] While the present invention has been described as having an exemplary design, the present invention can be further modified within the spirit and scope of this disclosure. Therefore, this application is intended to cover any variations, uses or adaptations of the invention using its general principles.

[0193] The above specific embodiments have been described in various forms of devices and / or methods using block diagrams, flow charts and / or examples. As long as such block diagrams, flow charts and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation in such block diagrams, flow charts and / or examples can be implemented individually and / or collectively by a variety of hardware, software, firmware or any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein can be implemented equivalently in integrated circuits as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as any combination thereof, in whole or in part, and according to the present disclosure, designing circuits and / or writing software and / or hardware codes will be within the technical scope of those skilled in the art. In addition, those skilled in the art will recognize that the mechanisms of the subject matter described herein can be distributed as one or more program products in a variety of forms, and the illustrative forms of the subject matter described herein are applicable, regardless of the specific type of signal-bearing medium used for actual distribution.

[0194] Instructions for programming logic to perform various disclosed aspects may be stored in a memory in the system, such as a dynamic random access memory (DRAM), a cache, a flash memory, or other memory. In addition, the instructions may be distributed via a network or by other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, but is not limited to a floppy disk, an optical disk, a compact disk read-only memory (CD-ROM), and a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible, machine-readable storage device used when transmitting information on the Internet via an electrical signal, an optical signal, an acoustic signal, or other form of propagation signal (e.g., a carrier wave, an infrared signal, a digital signal, etc.). Therefore, a non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0195] As used in any aspect of this document, the term "control circuitry" or "control system" may refer to, for example, hard-wired circuitry, programmable circuitry (e.g., a computer processor including one or more separate instruction processing cores, a processing unit, a processor, a microcontroller, a microcontroller unit, a controller, a digital signal processor (DSP), a programmable logic device (PLD), a programmable logic array (PLA), a field programmable gate array (FPGA)), a state machine circuitry, firmware that stores instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be implemented collectively or individually as circuitry that forms part of a larger system, such as an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smart phone, etc. Thus, as used herein, "control circuitry" includes, but is not limited to, electronic circuitry having at least one discrete circuit, electronic circuitry having at least one integrated circuit, electronic circuitry having at least one application specific integrated circuit, electronic circuitry forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program to at least partially implement the methods and / or devices described herein, or a microprocessor configured by a computer program to at least partially implement the methods and / or devices described herein), electronic circuitry forming a memory device (e.g., forming a random access memory), and / or electronic circuitry forming a communication device (e.g., a modem, a communication switch, or an optoelectronic device). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog or digital form, or some combination thereof.

[0196] As used in any aspect herein, the term "logic" may refer to an application, software, firmware, and / or circuitry for performing any of the foregoing operations. Software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets and / or data hard-coded (e.g., non-volatile) in a memory device.

[0197] As used in any aspect of this document, the terms "component," "system," "module" and the like may refer to a computer-related entity, hardware, a combination of hardware and software, software, or software in execution.

[0198] As used in any aspect herein, an "algorithm" refers to a self-consistent sequence of steps leading to a desired result, wherein the "steps" refer to manipulations of physical quantities and / or logical states which may (but need not) take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. References to such signals are often made to bits, values, elements, symbols, characters, terms, numbers, and the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to such quantities and / or states.

[0199] Unless otherwise expressly indicated in the above disclosure, it is understood that discussions using terms such as "processing," "computing," "calculating," "determining," and "displaying" in the above disclosure refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms it into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0200] One or more components may be referred to herein as "for", "may be configured to", "operable / operate to", "suitable / adaptable to", "capable of", "conformable / conformable to", etc. Those skilled in the art will recognize that, unless the context requires otherwise, "for" may generally encompass active state components and / or inactive state components and / or standby state components.

[0201] Those skilled in the art will recognize that, in general, the terms used herein, and in particular in the appended claims (e.g., the body of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including but not limited to", the term "having" should be interpreted as "having at least", the term "comprising" should be interpreted as "including but not limited to", etc.). Those skilled in the art will also understand that if a specific number of an introduced claim recitation is intended, such intent will be explicitly stated in the claim, and in the absence of such recitation, no such intent is present. For example, to aid understanding, the following appended claims may contain use of the introductory phrases "at least one" and "one or more" to introduce claims. However, the use of such phrases should not be construed as implying that the introduction of a claim recitation by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to claims containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"); this also applies to the use of definite articles used to introduce claim recitations.

[0202] In addition, even if a specific number of claim recitations is explicitly recited, those skilled in the art will recognize that such recitation should generally be interpreted to mean at least the recited number (e.g., a bare recitation of "two recitations" without other modifiers generally means at least two recitations, or two or more recitations). In addition, in those cases where a convention similar to "at least one of A, B, and C, etc." is used, generally, such construction is intended to have the meaning that those skilled in the art will understand the convention (e.g., "a system having at least one of A, B, and C" will include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those cases where a convention similar to "at least one of A, B, or C, etc." is used, generally, such construction is intended to have the meaning that those skilled in the art will understand the convention (e.g., "a system having at least one of A, B, or C" will include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). It should also be understood by those skilled in the art that, generally, unless the context indicates otherwise, transitional words and / or phrases presenting two or more alternative terms in the detailed description, claims, or drawings should be understood to include the possibility of including one of the terms, any one of the terms, or both of the terms. For example, the phrase "A or B" will generally be understood to include the possibility of "A" or "B" or "A and B".

[0203] For the appended claims, those skilled in the art will appreciate that the operations described therein may generally be performed in any order. In addition, although various operational flow charts are presented in one or more sequences, it should be understood that the various operations may be performed in other orders than the order shown, or the various operations may be performed simultaneously. Unless the context otherwise dictates, examples of such alternative orderings may include overlapping, interleaved, interrupted, reordered, incremental, preparatory, supplementary, simultaneous, reverse, or other altered orderings. In addition, unless the context otherwise dictates, terms such as "responsive to," "related to," or other past tense adjectives are generally not intended to exclude such variants.

[0204] It is worth mentioning that any reference to "one aspect", "an aspect", "an example", or "an example" means that the specific features, structures, or characteristics described in conjunction with the aspect are included in at least one aspect. Therefore, the phrases "in one aspect", "in an aspect", "in an example", or "in an example" appearing in various places throughout the specification do not necessarily refer to the same aspect. In addition, specific features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0205] Any patent application, patent, non-patent publication or other public material referred to in this specification and / or listed in any Application Data Sheet is incorporated herein by reference to the extent that the incorporated material is inconsistent herewith. Therefore, and to the extent necessary, the disclosure explicitly set forth herein supersedes any conflicting material incorporated herein by reference. Any material or portion thereof that is allegedly incorporated herein by reference but conflicts with existing definitions, statements or other public material set forth herein will be incorporated only to the extent that there is no conflict between the incorporated material and the existing public material.

[0206] The terms "comprise" (and any form of "comprise", such as "comprises" and "comprising"), "have" (and any form of "have", such as "has" and "having"), "include" (and any form of "include", such as "includes" and "including"), and "contain" (and any form of "contain", such as "contains" and "containing") are open-ended linking verbs. Thus, a system that "comprises", "has", "includes", or "contains" one or more elements possesses those one or more elements, but is not limited to possessing only those one or more elements. Likewise, an element of a system, device, or apparatus that "comprises", "has", "includes", or "contains" one or more features possesses those one or more features, but is not limited to possessing only those one or more features.

[0207] The terms "substantially," "about," or "approximately" as used in this disclosure refer to the acceptable error for a particular value as determined by one of ordinary skill in the art, which depends in part on how the value is measured or determined. In certain embodiments, the terms "substantially," "about," or "approximately" refer to within 1, 2, 3, or 4 standard deviations. In certain embodiments, the terms "substantially," "about," or "approximately" refer to within 50%, 20%, 15%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, or 0.05% of a given value or range.

[0208] In summary, many of the benefits resulting from the use of the concepts described herein have been described. For purposes of illustration and description, one or more of the above specific embodiments have been provided. These specific embodiments are not intended to be exhaustive or limited to the precise forms disclosed by the invention. The invention may be modified or varied in accordance with the above teachings. The one or more forms selected and described are intended to illustrate the principles and practical applications so that one of ordinary skill in the art can utilize various forms and various modifications suitable for the intended specific use. The claims submitted herewith are intended to define the full scope.

Claims

1. A surgical device comprising: An end effector, the end effector comprising: a first jaw; and Second jaw; a first sensor configured to detect tissue disposed between the first jaw and the second jaw; a second sensor for detecting the end effector in a closed configuration; a third sensor for detecting movement of the end effector; and a control circuit, the control circuit being communicatively coupled to the first sensor, the second sensor, and the third sensor, the control circuit comprising a processor and a memory, wherein the memory stores instructions that, when executed by the processor, cause the control circuit to: determining, based on the first sensor data, that the end effector is in a closed configuration; determining the presence of tissue disposed between the first jaw and the second jaw based on second sensor data; monitoring movement of the end effector in the closed configuration with tissue present between the first jaw and the second jaw based on third sensor data; detecting movement of the end effector outside of a predetermined range based on the movement; and Feedback data is provided based on the detected movement of the end effector.

2. The surgical device according to claim 1, wherein: The memory stores further instructions that, when executed by the processor, cause the control circuit to calculate tension on the tissue based on the motion.

3. The surgical device according to claim 1 or claim 2, wherein: Feedback is visual.

4. The surgical device according to claim 3, wherein: The visual feedback is superimposed on a displayed image of the surgical site.

5. A surgical device according to any preceding claim, wherein: The feedback is auditory.

6. A surgical device according to any preceding claim, wherein: The first jaw includes a clamping arm and the second jaw includes an ultrasonic blade.

7. A surgical device according to any preceding claim, wherein: The first jaw includes an anvil and the second jaw includes a staple cartridge.

8. A surgical device comprising: An end effector, the end effector comprising: a first jaw; and Second jaw; a first sensor configured to detect tissue disposed between the first jaw and the second jaw; a second sensor for detecting that the end effector is in a closed configuration; first fiducial marker; a second fiducial marker; and a control circuit, the control circuit being communicatively coupled to the first sensor, the second sensor, and the camera, the control circuit comprising a processor and a memory, wherein the memory stores instructions that, when executed by the processor, cause the control circuit to: receiving video data of a surgical site from the camera; determining, based on first sensor data, that the end effector is in the closed configuration; determining the presence of tissue disposed between the first jaw and the second jaw based on second sensor data; determining a position of a device tip in the video data based on the first reference marker and the second reference marker; determining a region of interest in the video data based on the position of the device terminal in the video data; analyzing the region of interest of the end effector in the closed configuration with tissue present between the first jaw and the second jaw; determining tension on the tissue based on the analyzing; and Feedback is provided based on the tension.

9. The surgical device according to claim 8, wherein: The memory stores further instructions that, when executed by the processor, cause the control circuit to detect movement of the end effector outside of a predetermined range based on the analysis.

10. The surgical device according to claim 8 or claim 9, wherein: The memory stores further instructions that, when executed by the processor, cause the control circuit to determine a device type based on the first and second fiducial markers.

11. The surgical device according to any one of claims 8 to 10, wherein: The feedback is visual.

12. The surgical device according to claim 11, wherein: The visual feedback is superimposed on a displayed image of the surgical site.

13. The surgical device according to any one of claims 8 to 12, wherein: The feedback is auditory.

14. The surgical device according to any one of claims 8 to 13, wherein: The first jaw includes a clamping arm and the second jaw includes an ultrasonic blade.

15. The surgical device according to any one of claims 8 to 14, wherein: The first jaw includes an anvil and the second jaw includes a staple cartridge.

16. A surgical system comprising: a surgical instrument comprising an end effector for capturing tissue, wherein the end effector comprises an electrode to apply radio frequency (RF) energy to the tissue captured by the end effector; an RF energy source for providing RF energy to the electrode; and A control circuit, the control circuit being used for: transmitting a control signal to the RF energy source, wherein the control signal causes the RF energy source to provide RF energy to the electrode to apply a seal to the tissue captured by the end effector; predicting the quality of the seal; and Feedback is provided to the user based on the prediction.

17. The surgical system of claim 16, wherein: In order to predict the quality of the seal, the control circuit is used to: generating a value associated with the seal; and comparing the value to a sealing threshold; In order to provide feedback to the user based on the prediction, the control circuit is used to provide feedback to the user based on the result of the comparison.

18. The surgical system of claim 17, wherein: The control circuit is configured to forgo providing feedback based on the value reaching or exceeding the sealing threshold.

19. The surgical system of claim 17 or claim 18, wherein: The surgical system also includes a display, wherein the control circuit transmits a signal to the display based on the value being below the sealing threshold, wherein the feedback includes visual feedback on the display, and wherein the visual feedback is based on the signal.

20. The surgical system of any one of claims 17 to 19, wherein: The surgical system also includes an audio feedback module, wherein the control circuit transmits a signal to the audio feedback module based on the value being below the sealing threshold, wherein the feedback includes audio feedback via the audio feedback module, and wherein the audio feedback is based on the signal.

21. The surgical system of any one of claims 17 to 20, wherein: The surgical system also includes a tactile feedback module, wherein the control circuit transmits a signal to the tactile feedback module based on the value being below the sealing threshold, wherein the feedback includes tactile feedback via the tactile feedback module, and wherein the tactile feedback is based on the signal.

22. The surgical system of any one of claims 16 to 21, wherein: The RF energy source includes the control circuit.

23. The surgical system of any one of claims 16 to 22, wherein: The surgical system also includes a processing unit that includes the control circuit.

Images