Dual CMOS array imaging

Abstract

The present invention discloses an image acquisition system, which includes: a plurality of illumination sources, each of which is configured to emit light having a specified central wavelength; a first light sensing element, which has a first field of view and is configured to receive illumination reflected from a portion of a surgical site; a second light sensing element, which has a second field of view and is configured to receive illumination reflected from a second portion of the surgical site; and a computing system. The computing system is configured to receive data from the first light sensing element and from the second light sensing element, calculate imaging data based on the data received from the first light sensing element and from the second light sensing element, and transmit the imaging data for reception by a display system. The second field of view may overlap with at least a portion of the first field of view. The control system of the image acquisition system may function similarly.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application Serial No. 62 / 649,291, filed on March 28, 2018, entitled “USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT,” the disclosure of which is incorporated herein by reference in its entirety.

[0003] This application also claims the benefit of priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application Serial No. 62 / 611,341, filed on December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM,” U.S. Provisional Patent Application Serial No. 62 / 611,340, filed on December 28, 2017, entitled “CLOUD-BASED MEDICAL ANALYTICS,” and U.S. Provisional Patent Application Serial No. 62 / 611,339, filed on December 28, 2017, entitled “ROBOT ASSISTED SURGICAL PLATFORM,” the disclosure of each of which is incorporated herein by reference in its entirety. Background Art

[0004] The present disclosure relates to various surgical systems. Surgical procedures are typically performed in a surgical operating room or room in a medical facility such as, for example, a hospital. A sterile field is typically formed around the patient. The sterile field may include appropriately dressed scrubbed team members, as well as all furniture and fixtures in the area. Various surgical devices and systems are utilized when performing surgical procedures. Summary of the invention

[0005] In some aspects, a minimally invasive image acquisition system may include: a plurality of illumination sources, wherein each illumination source is configured to emit light having a specified central wavelength; a first light sensing element having a first field of view and configured to receive illumination reflected from a first portion of a surgical site when the first portion of the surgical site is illuminated by at least one of the plurality of illumination sources; a second light sensing element having a second field of view and configured to receive illumination reflected from a second portion of the surgical site when the second portion of the surgical site is illuminated by at least one of the plurality of illumination sources; and a computing system, wherein the computing system is configured to receive data from the first light sensing element, receive data from the second light sensing element, calculate imaging data based on the data received from the first light sensing element and the data received from the second light sensing element, and transmit the imaging data for receipt by a display system. Further, the second field of view overlaps at least a portion of the first field of view.

[0006] In one aspect of the minimally invasive image acquisition system, the first field of view has a first angle and the second field of view has a second angle, and the first angle is the same as the second angle.

[0007] In one aspect of the minimally invasive image acquisition system, the first field of view has a first angle and the second field of view has a second angle, and the first angle is different from the second angle.

[0008] In one aspect of the minimally invasive image acquisition system, the first light sensing element has an optical component configured to adjust the first field of view.

[0009] In one aspect of the minimally invasive image acquisition system, the second light sensing element has an optical component configured to adjust the second field of view.

[0010] In one aspect of the minimally invasive image acquisition system, the second field of view overlaps the entirety of the first field of view.

[0011] In one aspect of the minimally invasive image acquisition system, the first field of view is completely surrounded by the second field of view.

[0012] In one aspect of the minimally invasive image acquisition system, a first light sensing element and a second light sensing element are at least partially disposed within an elongated camera probe.

[0013] In one aspect of the minimally invasive image acquisition system, each of the plurality of illumination sources is configured to emit light having a designated central wavelength that is within the visible spectrum.

[0014] In one aspect of the minimally invasive image acquisition system, at least one illumination source of the plurality of illumination sources is configured to emit light having a designated central wavelength that is outside the visible spectrum.

[0015] In one aspect of the minimally invasive image acquisition system, the designated central wavelength outside the visible spectrum is in the ultraviolet range.

[0016] In one aspect of the minimally invasive image acquisition system, the designated central wavelength outside the visible spectrum is in the infrared range.

[0017] In one aspect of a minimally invasive image acquisition system, a computing system configured to calculate imaging data based on data received from a first light sensing element and data received from a second light sensing element includes a computing system configured to perform a first data analysis on the data received from the first light sensing element and to perform a second data analysis on the data received from the second light sensing element.

[0018] In one aspect of the minimally invasive image acquisition system, the first data analysis is different from the second data analysis.

[0019] In some aspects, a minimally invasive image acquisition system is comprised of a processor and a memory coupled to the processor. The memory may store instructions executable by the processor to: control the operation of a plurality of illumination sources for a tissue sample, wherein each illumination source is configured to emit light having a specified central wavelength; when a first portion of a surgical site is illuminated by at least one of the plurality of illumination sources, receive first data from a first light sensing element related to illumination reflected from a first portion of the surgical site; when a second portion of the surgical site is illuminated by at least one of the plurality of illumination sources, receive second data from a second light sensing element related to illumination reflected from a second portion of the surgical site; calculate imaging data based on the first data received from the first light sensing element and the second data received from the second light sensing element; and transmit the imaging data for receipt by a display system. In some aspects, the second field of view overlaps at least a portion of the first field of view.

[0020] In one aspect of the minimally invasive image acquisition system, the memory coupled to the processor further stores instructions executable by the processor to receive operational data from the surgical instrument related to a function or status of the surgical instrument.

[0021] In one aspect of the minimally invasive image acquisition system, the memory connected to the processor also stores the following instructions, which can be executed by the processor to calculate imaging data based on first data received from the first light sensing element, second data received from the second light sensing element, and operational data related to the function or state of the surgical instrument.

[0022] In some aspects, a minimally invasive image acquisition system may include control circuitry configured to: control operation of a plurality of illumination sources for a tissue sample, wherein each illumination source is configured to emit light having a specified central wavelength; receive first data from a first light sensing element related to illumination reflected from a first portion of a surgical site when a first portion of the surgical site is illuminated by at least one of the plurality of illumination sources; receive second data from a second light sensing element related to illumination reflected from a second portion of the surgical site when a second portion of the surgical site is illuminated by at least one of the plurality of illumination sources; calculate imaging data based on the first data received from the first light sensing element and the second data received from the second light sensing element; and transmit the imaging data for receipt by a display system. In some aspects, the second field of view overlaps at least a portion of the first field of view.

[0023] In some aspects, a non-transitory computer readable medium may store computer readable instructions that, when executed, cause a machine to: control operation of a plurality of illumination sources for a tissue sample, wherein each illumination source is configured to emit light having a specified central wavelength; when a first portion of a surgical site is illuminated by at least one of the plurality of illumination sources, receive first data from a first light sensing element related to illumination reflected from a first portion of the surgical site; when a second portion of the surgical site is illuminated by at least one of the plurality of illumination sources, receive second data from a second light sensing element related to illumination reflected from a second portion of the surgical site; calculate imaging data based on the first data received from the first light sensing element and the second data received from the second light sensing element; and transmit the imaging data for receipt by a display system. In some aspects, the second field of view overlaps at least a portion of the first field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The features of the various aspects are particularly described in the appended claims. However, the various aspects (relating to surgical organizations and methods) and further objects and advantages thereof may best be understood by reference to the following description taken in conjunction with the following drawings.

[0025] Figure 1 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0026] Figure 2 A surgical system for performing surgical procedures in an operating room according to at least one aspect of the present disclosure.

[0027] Figure 3 A surgical hub is paired with a visualization system, a robotic system, and a smart instrument according to at least one aspect of the present disclosure.

[0028] Figure 4A partial perspective view of a surgical hub housing and a combination generator module slidably received in a drawer of the surgical hub housing according to at least one aspect of the present disclosure.

[0029] Figure 5 is a perspective view of a combination generator module having bipolar, ultrasonic and monopolar contacts and smoke evacuation components according to at least one aspect of the present disclosure.

[0030] Figure 6 A single power bus attachment for multiple lateral docking ports of a lateral modular housing configured to accommodate a plurality of modules is shown in accordance with at least one aspect of the present disclosure.

[0031] Figure 7 A vertical modular housing configured to house a plurality of modules according to at least one aspect of the present disclosure is shown.

[0032] Figure 8 A surgical data network including a modular communication hub configured to connect modular devices located in one or more operating rooms of a medical facility or any room in a medical facility dedicated to surgical procedures to a cloud is shown in accordance with at least one aspect of the present disclosure.

[0033] Fig. 9 A computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0034] Fig.10 A surgical hub comprising multiple modules coupled to a modular control tower is shown in accordance with at least one aspect of the present disclosure.

[0035] Fig.11 An aspect of a universal serial bus (USB) hub device in accordance with at least one aspect of the present disclosure is shown.

[0036] Fig.12 A logic diagram of a control system for a surgical instrument or tool according to at least one aspect of the present disclosure is shown.

[0037] Fig.13 Control circuitry configured to control various aspects of a surgical instrument or tool in accordance with at least one aspect of the present disclosure is shown.

[0038] Fig.14 Combinatorial logic circuits configured to control various aspects of a surgical instrument or tool in accordance with at least one aspect of the present disclosure are shown.

[0039] Fig.15 A sequential logic circuit configured to control various aspects of a surgical instrument or tool according to at least one aspect of the present disclosure is shown.

[0040] Fig.16 A surgical instrument or tool including multiple motors that can be activated to perform various functions in accordance with at least one aspect of the present disclosure is shown.

[0041] Fig.17 is a schematic diagram of a robotic surgical instrument configured to operate the surgical tools described herein according to at least one aspect of the present disclosure.

[0042] Fig.18 A block diagram of a surgical instrument programmed to control distal translation of a displacement member in accordance with at least one aspect of the present disclosure is shown.

[0043] Fig.19 is a schematic diagram of a surgical instrument configured to control various functions according to at least one aspect of the present disclosure.

[0044] Fig. 20 A simplified block diagram of a generator configured to provide, among other benefits, inductorless tuning in accordance with at least one aspect of the present disclosure.

[0045] Fig.21 A generator according to at least one aspect of the present disclosure is shown. Fig. 20 An example of a form of a generator).

[0046] Fig.22A A visualization system that may be incorporated into a surgical system in accordance with at least one aspect of the present disclosure is shown.

[0047] Fig. 22B According to at least one aspect of the present disclosure, Fig.22A Top plan view of the handheld unit of the visualization system.

[0048] Fig. 22C According to at least one aspect of the present disclosure, Fig.22A A side plan view of a handheld unit is depicted with an imaging sensor disposed therein.

[0049] Fig.22D According to at least one aspect of the present disclosure, Fig. 22C Multiple imaging sensors are depicted.

[0050] Fig.23A The present invention shows that at least one aspect of the present disclosure can be incorporated into Fig.22A Visualization of multiple laser emitters in the system.

[0051] Fig. 23B Illumination of an image sensor having a Bayer color filter pattern in accordance with at least one aspect of the present disclosure is shown.

[0052] Fig.23C A graphical representation of the operation of a pixel array for multiple frames in accordance with at least one aspect of the present disclosure is shown.

[0053] Fig.23D A diagram illustrating an example of an operational sequence of chroma frames and luma frames according to at least one aspect of the present disclosure.

[0054] Fig.23E Examples of sensor and emitter patterns in accordance with at least one aspect of the present disclosure are shown.

[0055] Fig.23F A graphical representation of the operation of a pixel array in accordance with at least one aspect of the present disclosure is shown.

[0056] Fig.24 A schematic diagram showing one example of an instrument for NIR spectroscopy according to one aspect of the present disclosure is shown.

[0057] Fig.25 An example of an apparatus for determining NIRS based on Fourier transform infrared imaging according to at least one aspect of the present disclosure is schematically illustrated.

[0058] Fig.26A -C shows the change in wavelength of light scattered from moving blood cells according to at least one aspect of the present disclosure.

[0059] Fig. 27 Aspects of an instrument that may be used to detect a Doppler shift of laser light scattered from a portion of tissue in accordance with at least one aspect of the present disclosure are shown.

[0060] Fig.28 Some optical effects on light projected onto tissue having subsurface structures in accordance with at least one aspect of the present disclosure are schematically illustrated.

[0061] Fig.29 An example of the effect of Doppler analysis of light projected onto a tissue sample having subsurface structures in accordance with at least one aspect of the present disclosure is shown.

[0062] Fig. 30A -C schematically illustrates the detection of moving blood cells at tissue depth based on laser Doppler analysis at various laser wavelengths according to at least one aspect of the present disclosure.

[0063] Fig.30D The effect of illuminating a CMOS imaging sensor with multiple wavelengths of light over time in accordance with at least one aspect of the present disclosure is shown.

[0064] Fig.31 An example of using Doppler imaging to detect the presence of subsurface blood vessels in accordance with at least one aspect of the present disclosure is shown.

[0065] Fig.32 A method of identifying subsurface blood vessels based on the Doppler shift of blue light due to blood cells flowing therethrough in accordance with at least one aspect of the present disclosure is shown.

[0066] Fig.33 The positioning of deep subsurface blood vessels according to at least one aspect of the present disclosure is schematically illustrated.

[0067] Fig.34 The positioning of superficial subsurface blood vessels according to at least one aspect of the present disclosure is schematically illustrated.

[0068] Fig.35 A composite image including a surface image and an image of subsurface blood vessels in accordance with at least one aspect of the present disclosure is shown.

[0069] Fig.36 A flow chart of a method for determining a depth of a surface feature in a volume of tissue according to at least one aspect of the present disclosure.

[0070] Fig.37 The effects of the location and properties of non-vascular structures on light projected onto a tissue sample in accordance with at least one aspect of the present disclosure are shown.

[0071] Fig.38 One example of components used in a full-field OCT apparatus according to at least one aspect of the present disclosure is schematically depicted.

[0072] Fig.39 Schematically illustrates the effect of tissue abnormalities on light reflected from a tissue sample in accordance with at least one aspect of the present disclosure.

[0073] Fig.40 Shown is a combined image display derived from tissue visualization modalities in accordance with at least one aspect of the present disclosure.

[0074] Fig.41A -C illustrates several aspects of a display that may be provided to a surgeon for visually identifying a combination of surface and subsurface structures of tissue in a surgical site in accordance with at least one aspect of the present disclosure.

[0075] Fig.42 A flow chart of a method for providing information related to characteristics of tissue to an intelligent surgical instrument according to at least one aspect of the present disclosure.

[0076] Fig.43A and Fig.43BA multi-pixel light sensor according to at least one aspect of the present disclosure is shown, which receives light reflected by tissue sequentially exposed to red light, green light, blue light and infrared light and irradiated by a red laser source, a green laser source, a blue laser source and an ultraviolet laser source.

[0077] Fig.44A and Fig.44B Distal ends of elongated camera probes having a single light sensor and two light sensors, respectively, are shown in accordance with at least one aspect of the present disclosure.

[0078] Fig.44C A perspective view is shown of an example of a monolithic sensor having multiple pixel arrays in accordance with at least one aspect of the present disclosure.

[0079] Fig.45 An example of a pair of fields of view available to two image sensors of an elongated camera probe according to at least one aspect of the present disclosure is shown.

[0080] Fig.46A -D shows an additional example of a pair of fields of view available to two image sensors of an elongated camera probe according to at least one aspect of the present disclosure.

[0081] Fig.47A -C illustrates the use of at least one aspect of the present disclosure incorporating Fig.46D Examples of imaging systems of the disclosed features.

[0082] Fig.48A and Fig.48B Another example of using a dual imaging system in accordance with at least one aspect of the present disclosure is shown.

[0083] Fig.49A -C illustrates an example of a sequence of surgical steps that may benefit from the use of multiple image analysis at a surgical site in accordance with at least one aspect of the present disclosure.

[0084] Fig.50 A timeline depicting situational awareness of a surgical hub according to at least one aspect of the present disclosure. DETAILED DESCRIPTION

[0085] The applicant of the present application owns the following U.S. provisional patent applications filed on March 28, 2018, each of which is incorporated herein by reference in its entirety:

[0086] U.S. Provisional Patent Application Serial No. 62 / 649,302, entitled “INTERACTIVE SURGICAL SYSTEMS WITH ENCRYPTED COMMUNICATION CAPABILITIES”;

[0087] U.S. Provisional Patent Application Serial No. 62 / 649,294, entitled “DATA STRIPPING METHOD TO INTERROGATE PATIENT RECORDS AND CREATE ANONYMIZED RECORD”;

[0088] U.S. Patent Application Serial No. 62 / 649,300, entitled “SURGICAL HUB SITUATIONAL AWARENESS”;

[0089] U.S. Provisional Patent Application Serial No. 62 / 649,309, entitled “SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER”;

[0090] U.S. Provisional Patent Application Serial No. 62 / 649,310, entitled “COMPUTER IMPLEMENTED INTERACTIVE SURGICAL SYSTEMS”;

[0091] U.S. Provisional Patent Application Serial No. 62 / 649,291, entitled “USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT”;

[0092] U.S. Provisional Patent Application Serial No. 62 / 649,296, entitled “ADAPTIVE CONTROL PROGRAM UPDATES FOR SURGICAL DEVICES”;

[0093] U.S. Provisional Patent Application Serial No. 62 / 649,333, entitled “CLOUD-BASED MEDICALANALYTICS FOR CUSTOMIZATION AND RECOMMENDATIONS TO A USER”;

[0094] U.S. Provisional Patent Application Serial No. 62 / 649,327, entitled “CLOUD-BASED MEDICALANALYTICS FOR SECURITY AND AUTHENTICATION TRENDS AND REACTIVE MEASURES”;

[0095] U.S. Provisional Patent Application Serial No. 62 / 649,315, entitled “DATA HANDLING AND PRIORITIZATION IN A CLOUD ANALYTICS NETWORK”;

[0096] U.S. Patent Application Serial No. 62 / 649,313, entitled “CLOUD INTERFACE FOR COUPLED SURGICAL DEVICES”;

[0097] U.S. Provisional Patent Application Serial No. 62 / 649,320, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0098] U.S. Provisional Patent Application Serial No. 62 / 649,307, entitled “AUTOMATIC TOOLADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”; and

[0099] • U.S. Provisional Patent Application Serial No. 62 / 649,323, entitled “SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS.”

[0100] The applicant of the present application owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0101] U.S. Patent Application Serial No. __________, entitled “INTERACTIVE SURGICAL SYSTEMSWITH ENCRYPTED COMMUNICATION CAPABILITIES”; Attorney Docket No. END8499USNP / 170766;

[0102] U.S. Patent Application Serial No. __________, entitled “INTERACTIVE SURGICAL SYSTEMS WITH CONDITION HANDLING OF DEVICES AND DATA CAPABILITIES”; Attorney Docket No. END8499USNP1 / 170766-1;

[0103] U.S. patent application serial number __________, entitled “SURGICAL HUB COORDINATION OF CONTROL AND COMMUNICATION OF OPERATING room DEVICES”; attorney docket number END8499USNP2 / 170766-2;

[0104] U.S. Patent Application Serial No. __________, entitled “SPATIAL AWARENESS OF SURGICAL HUBS IN OPERATING ROOMS”; Attorney Docket No. END8499USNP3 / 170766-3;

[0105] U.S. Patent Application Serial No. __________, entitled “COOPERATIVE UTILIZATION OF DATA DERIVED FROM SECONDARY SOURCES BY INTELLIGENT SURGICAL HUBS”; Attorney Docket No. END8499USNP4 / 170766-4;

[0106] U.S. patent application Ser. No. __________, entitled “SURGICAL HUB CONTROL arrangements”; attorney docket No. END8499USNP5 / 170766-5;

[0107] U.S. Patent Application Serial No. __________, entitled “DATA STRIPPING METHOD TO INTERROGATE PATIENT RECORDS AND CREATE ANONYMIZED RECORD”; Attorney Docket No. END8500USNP / 170767;

[0108] U.S. Patent Application Serial No. __________, entitled “COMMUNICATION HUB AND STORAGE DEVICE FOR STORING PARAMETERS AND STATUS OF A SURGICAL DEVICE TO BE SHARED WITH CLOUD BASED ANALYTICS SYSTEMS”; Attorney Docket No. END8500USNP1 / 170767-1;

[0109] U.S. Patent Application Serial No. __________, entitled “SELF DESCRIBING DATA PACKETSGENERATED AT AN ISSUING INSTRUMENT”; Attorney Docket No. END8500USNP2 / 170767-2;

[0110] U.S. Patent Application Serial No. __________, entitled “DATA PAIRING TO INTERCONNECTA DEVICE MEASURED PARAMETER WITH AN OUTCOME”; Attorney Docket No. END8500USNP3 / 170767-3;

[0111] U.S. Patent Application Serial No. __________, entitled “SURGICAL HUB SITUATIONAL AWARENESS”; Attorney Docket No. END8501USNP / 170768;

[0112] U.S. Patent Application Serial No. __________, entitled “SURGICAL SYSTEM DISTRIBUTED PROCESSING”; Attorney Docket No. END8501USNP1 / 170768-1;

[0113] U.S. Patent Application Serial No. __________, entitled “AGGREGATION AND REPORTING OF SURGICAL HUB DATA”; Attorney Docket No. END8501USNP2 / 170768-2;

[0114] U.S. Patent Application Serial No. __________, entitled “SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER”; Attorney Docket No. END8502USNP / 170769;

[0115] U.S. Patent Application Serial No. __________, entitled “DISPLAY OF ALIGNMENT OF STAPLE CARTRIDGE TO PRIOR LINEAR STAPLE LINE”; Attorney Docket No. END8502USNP1 / 170769-1;

[0116] U.S. Patent Application Serial No. __________, entitled “STERILE FIELD INTERACTIVECONTROL DISPLAYS”; Attorney Docket No. END8502USNP2 / 170769-2;

[0117] U.S. Patent Application Serial No. __________, entitled “COMPUTER IMPLEMENTED INTERACTIVE SURGICAL SYSTEMS”; Attorney Docket No. END8503USNP / 170770;

[0118] U.S. Patent Application Serial No. ___________, entitled “USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT”; Attorney Docket No. END8504USNP / 170771; and

[0119] U.S. Patent Application Serial No. ____________, entitled “CHARACTERIZATION OF TISSUEIRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY”; Attorney Docket No. END8504USNP1 / 170771-1.

[0120] The applicant of the present application owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0121] U.S. Patent Application Serial No. __________, entitled “ADAPTIVE CONTROL PROGRAM UPDATES FOR SURGICAL DEVICES”; Attorney Docket No. END8506USNP / 170773;

[0122] U.S. Patent Application Serial No. ____________, entitled ADAPTIVE CONTROL PROGRAM UPDATES FOR SURGICAL HUBS; Attorney Docket No. END8506USNP1 / 170773-1;

[0123] U.S. Patent Application Serial No. __________, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR CUSTOMIZATION AND RECOMMENDATIONS TO A USER”; Attorney Docket No. END8507USNP / 170774;

[0124] U.S. Patent Application Serial No. __________, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR LINKING OF LOCAL USAGE TRENDS WITH THE RESOURCE ACQUISITION BEHAVIORS OF LARGER DATA SET”; Attorney Docket No. END8507USNP1 / 170774-1;

[0125] U.S. Patent Application Serial No. __________, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR MEDICAL FACILITY SEGMENTED INDIVIDUALIZATION OF INSTRUMENT FUNCTION”; Attorney Docket No. END8507USNP2 / 170774-2;

[0126] U.S. Patent Application Serial No. __________, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR SECURITY AND AUTHENTICATION TRENDS AND REACTIVE MEASURES”; Attorney Docket No. END8508USNP / 170775;

[0127] U.S. Patent Application Serial No. __________, entitled “DATA HANDLING AND PRIORITIZATION IN A CLOUD ANALYTICS NETWORK”; Attorney Docket No. END8509USNP / 170776; and

[0128] U.S. Patent Application Serial No. __________, entitled “CLOUD INTERFACE FOR COUPLED SURGICAL DEVICES”; Attorney Docket No. END8510USNP / 170777.

[0129] The applicant of the present application owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0130] U.S. Patent Application Serial No. __________, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”; Attorney Docket No. END8511USNP / 170778;

[0131] U.S. Patent Application Serial No. __________, entitled “COMMUNICATION ARRANGEMENTSFOR ROBOT-ASSISTED SURGICAL PLATFORMS”; Attorney Docket No. END8511USNP1 / 170778-1;

[0132] U.S. Patent Application Serial No. __________, entitled “CONTROLS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”; Attorney Docket No. END8511USNP2 / 170778-2;

[0133] U.S. patent application Ser. No. __________, entitled “AUTOMATIC TOOL ADJUSTMENTSFOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0134] The attorney’s docket number is END8512USNP / 170779;

[0135] U.S. Patent Application Serial No. __________, entitled “CONTROLLERS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”; Attorney Docket No. END8512USNP1 / 170779-1;

[0136] U.S. Patent Application Serial No. __________, entitled “COOPERATIVE SURGICAL ACTIONSFOR ROBOT-ASSISTED SURGICAL PLATFORMS”; Attorney Docket No. END8512USNP2 / 170779-2;

[0137] U.S. Patent Application Serial No. __________, entitled “DISPLAY ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”; Attorney Docket No. END8512USNP3 / 170779-3; and

[0138] U.S. Patent Application Serial No. __________, entitled “SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”; Attorney Docket No. END8513USNP / 170780.

[0139] Before describing in detail various aspects of the surgical device and generator, it should be noted that the application or use of the illustrative examples is not limited to the details of the configuration and arrangement of the components shown in the drawings and detailed description. The illustrative examples may be implemented alone or in combination with other aspects, variations and modifications, and may be practiced or performed in a variety of ways. In addition, unless otherwise indicated, the terms and expressions used herein are selected for the convenience of the reader to describe the exemplary embodiments and are not for limiting purposes. Moreover, it should be understood that one or more of the aspects, expressions of aspects and / or examples described below may be combined with any one or more of the other aspects, expressions of aspects and / or examples described below.

[0140] See also Figure 1 , a computer-implemented interactive surgical system 100 includes one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 that may include a remote server 113 coupled to a storage device 105). Each surgical system 102 includes at least one surgical hub 106 in communication with the cloud 104 that may include the remote server 113. In one example, as Figure 1 As shown in , the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some aspects, the surgical system 102 may include M number of hubs 106, N number of visualization systems 108, O number of robotic systems 110, and P number of handheld intelligent surgical instruments 112, where M, N, O, and P are integers greater than or equal to one.

[0141] Figure 3An example of a surgical system 102 is shown for performing a surgical procedure on a patient lying flat on an operating table 114 in a surgical operating room 116. A robotic system 110 is used as part of the surgical system 102 during a surgical procedure. The robotic system 110 includes a surgeon's console 118, a patient side cart 120 (surgical robot), and a surgical robotic hub 122. While the surgeon views the surgical site through the surgeon's console 120, the patient side cart 117 can manipulate at least one removably coupled surgical tool 118 through a minimally invasive incision in the patient. Images of the surgical site can be obtained through a medical imaging device 124, which can be manipulated by the patient side cart 120 to orient the imaging device 124. The robotic hub 122 can be used to process images of the surgical site for subsequent display to the surgeon through the surgeon's console 118.

[0142] Other types of robotic systems may be readily adapted for use with the surgical system 102. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Provisional Patent Application Serial No. 62 / 611,339, filed on December 28, 2017, entitled “ROBOT ASSISTED SURGICAL PLATFORM,” the disclosure of which is incorporated herein by reference in its entirety.

[0143] Various examples of cloud-based analytics performed by the cloud 104 and suitable for use in the present disclosure are described in U.S. Provisional Patent Application Serial No. 62 / 611,340, filed on December 28, 2017, entitled “CLOUD-BASED MEDICAL ANALYTICS,” the disclosure of which is incorporated herein by reference in its entirety.

[0144] In various aspects, imaging device 124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0145] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0146] One or more illumination sources may be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescence spectrum) is that portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and may be referred to as visible light or simply light. A typical human eye will respond to wavelengths in the range of about 380 nm to about 750 nm in air.

[0147] The invisible spectrum (i.e., non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0148] In various aspects, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophago-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0149] In one aspect, the imaging device employs multispectral monitoring to discern topography and underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths may be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging may allow for the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title “Advanced Imaging Acquisition Module” in U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed on December 28, 2017, the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring may be a useful tool for repositioning the surgical field after completing the surgical task to perform one or more of the previously described tests on the treated tissue.

[0150] It is self-evident that during any surgical procedure, the operating room and surgical equipment need to be strictly sterilized. The strict sanitary and sterile conditions required in a "surgical room" (i.e., an operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including the imaging device 124 and its attachments and components. It should be understood that the sterile field can be considered to be a designated area that is considered to be free of microorganisms, such as within a tray or sterile towels, or the sterile field can be considered to be an area around a patient that has been prepared for a surgical procedure. The sterile field can include a scrubbed team member that is appropriately dressed, as well as all furniture and fixtures in the area.

[0151] In various aspects, the visualization system 108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically arranged relative to the sterile field, such as Figure 2 . In one aspect, the visualization system 108 includes interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described in U.S. Provisional Patent Application Serial No. 62 / 611,341, filed on December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM,” under the title “Advanced Imaging Acquisition Module,” the disclosure of which is incorporated herein by reference in its entirety.

[0152] like Figure 2 , the main display 119 is positioned in the sterile field to be visible to the operator at the operating table 114. In addition, the visualization tower 111 is positioned outside the sterile field. The visualization tower 111 includes a first non-sterile display 107 and a second non-sterile display 109 that are separated from each other. The visualization system 108 guided by the hub 106 is configured to coordinate the flow of information to the operators inside and outside the sterile field using the displays 107, 109 and 119. For example, the hub 106 can cause the imaging system 108 to display a snapshot of the surgical site recorded by the imaging device 124 on the non-sterile display 107 or 109, while maintaining a real-time feed of the surgical site on the main display 119. The snapshot on the non-sterile display 107 or 109 can allow the non-sterile operator to perform diagnostic steps related to the surgical procedure, for example.

[0153] In one aspect, the hub 106 is also configured to route diagnostic input or feedback entered by a non-sterile operator at the visualization tower 111 to a primary display 119 within the sterile field, where it can be viewed by a sterile operator at the operating table. In one example, the input can be a modified form of a snapshot displayed on a non-sterile display 107 or 109, which can be routed to the primary display 119 via the hub 106.

[0154] See also Figure 2, the surgical instrument 112 is used in a surgical procedure as part of the surgical system 102. The hub 106 is also configured to coordinate the flow of information to the display of the surgical instrument 112. For example, U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed on December 28, 2017, the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 111 can be routed by the hub 106 to the surgical instrument display 115 within the sterile field, where the operator of the surgical instrument 112 can observe the input or feedback. Exemplary surgical instruments suitable for the surgical system 102 are described under the title “Surgical Instrument Hardware” in U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed on December 28, 2017, the disclosure of which is incorporated herein by reference in its entirety.

[0155] See now Figure 3 , the hub 106 is depicted as communicating with the visualization system 108, the robotic system 110, and the handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, and a storage array 134. In certain aspects, as Figure 3 As shown in , the hub 106 also includes a smoke evacuation module 126 and / or a suction / rinsing module 128.

[0156] During surgical procedures, energy application to tissue for sealing and / or cutting is often associated with smoke evacuation, suction of excess fluid, and / or flushing of tissue. Fluid lines, power lines, and / or data lines from different sources often become tangled during surgical procedures. Valuable time can be lost in solving this problem during surgical procedures. Disconnecting a line may require disconnecting the line from its corresponding module, which may require resetting the module. The hub modular housing 136 provides a unified environment for managing power lines, data lines, and fluid lines, which reduces the frequency of tangling between such lines.

[0157] Aspects of the present disclosure provide a surgical hub for surgical procedures involving application of energy to tissue at a surgical site. The surgical hub includes a hub housing and a combination generator module slidably accommodated in a docking base of the hub housing. The docking base includes data contacts and power contacts. The combination generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component seated in a single unit. In one aspect, the combination generator module also includes a smoke exhaust component for connecting the combination generator module to at least one energy delivery cable of a surgical instrument, at least one smoke exhaust component configured to exhaust smoke, fluid, and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke exhaust component.

[0158] In one aspect, the fluid line is a first fluid line and the second fluid line extends from the remote surgical site to an aspiration and irrigation module slidably received in the hub housing. In one aspect, the hub housing includes a fluid interface.

[0159] Certain surgical procedures may require more than one energy type to be applied to tissue. One energy type may be more conducive to cutting tissue, while another different energy type may be more conducive to sealing tissue. For example, a bipolar generator may be used to seal tissue, while an ultrasonic generator may be used to cut sealed tissue. Aspects of the present disclosure provide a solution in which a hub modular housing 136 is configured to accommodate different generators and facilitate interactive communication between them. One of the advantages of the hub modular housing 136 is the ability to quickly remove and / or replace various modules.

[0160] Aspects of the present disclosure provide a modular surgical housing for use in surgical procedures involving application of energy to tissue. The modular surgical housing includes a first energy generator module and a first docking station, the first energy generator module being configured to generate a first energy for application to tissue, the first docking station including a first docking port, the first docking port including first data and power contacts, wherein the first energy generator module is slidably movable into electrical engagement with the power and data contacts, and wherein the first energy generator module is slidably movable out of electrical engagement with the first power and data contacts.

[0161] Further describing the above, the modular surgical shell also includes a second energy generator module, which is configured to generate a second energy different from the first energy for application to tissue, and a second docking base, which includes a second docking port, the second docking port including second data and power contacts, wherein the second energy generator module is slidably movable into electrical engagement with the power and data contacts, and wherein the second energy generator is slidably movable out of electrical contact with the second power and data contacts.

[0162] In addition, the modular surgical shell also includes a communication bus between the first docking port and the second docking port, which is configured to facilitate communication between the first energy generator module and the second energy generator module.

[0163] See also Figure 3-7 , aspects of the present disclosure are presented as a hub modular housing 136 that allows modular integration of the generator module 140, the smoke evacuation module 126, and the suction / irrigation module 128. The hub modular housing 136 also facilitates interactive communication between the modules 140, 126, 128. Figure 5 As shown in , the generator module 140 may be a generator module having integrated monopolar components, bipolar components, and ultrasonic components supported in a single housing unit 139 that is slidably inserted into the hub modular housing 136. Figure 5 As shown in , the generator module 140 may be configured to connect to a monopolar device 146, a bipolar device 147, and an ultrasonic device 148. Alternatively, the generator module 140 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 136. The hub modular housing 136 may be configured to facilitate insertion of multiple generators and interactive communication between the generators docked into the hub modular housing 136 so that the generators will act as a single generator.

[0164] In one aspect, the hub modular housing 136 includes a modular power and communications backplane 149 with external and wireless communications connectors to enable removable attachment of the modules 140, 126, 128 and interactive communications therebetween.

[0165] In one aspect, the hub modular housing 136 includes a docking base or drawer 151 (also referred to herein as a drawer) that is configured to slidably accommodate the modules 140, 126, 128. Figure 4A partial perspective view of a surgical hub housing 136 and a combination generator module 145 slidably received in a docking station 151 of the surgical hub housing 136 is shown. A docking port 152 having power and data contacts on a back side of the combination generator module 145 is configured to engage a corresponding docking port 150 with the power and data contacts of a corresponding docking station 151 of the hub modular housing 136 when the combination generator module 145 is slid into place within the corresponding docking station 151 of the hub modular housing 136. In one aspect, the combination generator module 145 includes bipolar, ultrasonic and monopolar modules and a smoke evacuation module integrated together into a single housing unit 139, such as Figure 5 as shown in .

[0166] In various aspects, the smoke evacuation module 126 includes a fluid line 154 that delivers captured / collected smoke and / or fluid from the surgical site to, for example, the smoke evacuation module 126. Vacuum suction from the smoke evacuation module 126 can draw smoke into an opening of a common conduit at the surgical site. The common conduit coupled to the fluid line can be in the form of a flexible tube that terminates at the smoke evacuation module 126. The common conduit and the fluid line define a fluid path that extends toward the smoke evacuation module 126 housed in the hub housing 136.

[0167] In various aspects, the suction / irrigation module 128 is coupled to a surgical tool including an aspiration fluid line and a suction fluid line. In one example, the aspiration fluid line and the suction fluid line are in the form of flexible tubes extending from the surgical site toward the suction / irrigation module 128. One or more drive systems can be configured to flush fluid to the surgical site and to aspirate fluid from the surgical site.

[0168] In one aspect, the surgical tool comprises a shaft having an end effector at its distal end and at least one energy treatment associated with the end effector, an aspiration tube, and an irrigation tube. The aspiration tube may have an inlet at its distal end, and the aspiration tube extends through the shaft. Similarly, the aspiration tube may extend through the shaft and may have an inlet adjacent to an energy delivery tool. The energy delivery tool is configured to deliver ultrasonic energy and / or RF energy to a surgical site and is coupled to a generator module 140 by a cable initially extending through the shaft.

[0169] The irrigation tube may be in fluid communication with a fluid source, and the aspiration tube may be in fluid communication with a vacuum source. The fluid source and / or vacuum source may be seated in the suction / irrigation module 128. In one example, the fluid source and / or vacuum source may be seated in the hub housing 136 independently of the suction / irrigation module 128. In such examples, a fluid interface is capable of connecting the suction / irrigation module 128 to the fluid source and / or vacuum source.

[0170] In one aspect, the modules 140, 126, 128 and / or their corresponding docking stations on the hub modular housing 136 may include alignment features configured to align the docking ports of the modules to engage with their corresponding ports in the docking stations of the hub modular housing 136. For example, Figure 4 As shown in , the combination generator module 145 includes a side bracket 155, which is configured to slidably engage with a corresponding bracket 156 of a corresponding docking base 151 of the hub modular housing 136. The bracket cooperates to guide the docking port contacts of the combination generator module 145 to electrically engage with the docking port contacts of the hub modular housing 136.

[0171] In some aspects, the drawers 151 of the hub modular housing 136 are the same or substantially the same size, and the size of the module is adjusted to be accommodated in the drawer 151. For example, the side brackets 155 and / or 156 can be larger or smaller depending on the size of the module. In other aspects, the drawers 151 are different sizes and each is designed to accommodate a specific module.

[0172] Additionally, the contacts of a particular module may be keyed to engage the contacts of a particular drawer to avoid inserting a module into a drawer with mismatched contacts.

[0173] like Figure 4 As shown in , the docking port 150 of one drawer 151 can be coupled to the docking port 150 of another drawer 151 via a communication link 157 to facilitate interactive communication between modules seated in the hub modular housing 136. Alternatively or additionally, the docking port 150 of the hub modular housing 136 can facilitate wireless interactive communication between modules seated in the hub modular housing 136. Any suitable wireless communication can be employed, such as, for example, Air Titan-Bluetooth.

[0174] Figure 6 A single power bus attachment is shown for multiple lateral docking ports of a lateral modular housing 160 configured to accommodate multiple modules of a surgical hub 206. The lateral modular housing 160 is configured to laterally accommodate and interconnect modules 161. The modules 161 are slidably inserted into docking bases 162 of the lateral modular housing 160, which includes a bottom plate for interconnecting the modules 161. Figure 6 As shown in FIG, the modules 161 are arranged transversely in the transverse modular housing 160. Alternatively, the modules 161 may be arranged vertically in the transverse modular housing.

[0175] Figure 7A vertical modular housing 164 is shown that is configured to accommodate a plurality of modules 165 of a surgical hub 106. The modules 165 are slidably inserted into a docking station or drawer 167 of the vertical modular housing 164, which includes a bottom plate for interconnecting the modules 165. Although the drawers 167 of the vertical modular housing 164 are arranged vertically, in some cases, the vertical modular housing 164 may include drawers arranged horizontally. In addition, the modules 165 can interact with each other through the docking ports of the vertical modular housing 164. Figure 7 In the example of FIG. 1 , a display 177 is provided for displaying data related to the operation of the module 165. In addition, the vertical modular housing 164 includes a main module 178 that seats a plurality of sub-modules that are slidably received in the main module 178.

[0176] In various aspects, imaging module 138 includes an integrated video processor and a modular light source, and is suitable for use with various imaging devices. In one aspect, the imaging device is composed of a modular housing that can be equipped with a light source module and a camera module. The housing can be a disposable housing. In at least one example, the disposable housing is removably connected to a reusable controller, light source module, and camera module. The light source module and / or camera module can be selectively selected according to the type of surgical procedure. In one aspect, the camera module includes a CCD sensor. On the other hand, the camera module includes a CMOS sensor. On the other hand, the camera module is configured for scanning beam imaging. Similarly, the light source module can be configured to deliver white light or different light, depending on the surgical procedure.

[0177] During a surgical procedure, it may be inefficient to remove a surgical device from the surgical field and replace the surgical device with another surgical device that includes a different camera or a different light source. Temporarily losing sight of the surgical field may lead to undesirable consequences. The modular imaging device of the present disclosure is configured to allow mid-stream replacement of a light source module or a camera module during a surgical procedure without having to remove the imaging device from the surgical field.

[0178] In one aspect, an imaging device includes a tubular housing including a plurality of channels. A first channel is configured to slidably receive a camera module, which can be configured to snap-fit ​​into engagement with the first channel. A second channel is configured to slidably receive a light source module, which can be configured to snap-fit ​​into engagement with the second channel. In another example, the camera module and / or the light source module can be rotated into a final position within their respective channels. A threaded engagement can be used in place of a snap-fit ​​engagement.

[0179] In various examples, multiple imaging devices are placed at different locations in the surgical field to provide multiple views. The imaging module 138 can be configured to switch between imaging devices to provide the best view. In various aspects, the imaging module 138 can be configured to integrate images from different imaging devices.

[0180] Various image processors and imaging devices suitable for use with the present disclosure are described in U.S. Pat. No. 7,995,045, issued on Aug. 9, 2011, entitled “COMBINED SBI AND CONVENTIONAL IMAGE PROCESSOR”, which is incorporated herein by reference in its entirety. Additionally, U.S. Pat. No. 7,982,776, issued on Jul. 19, 2011, entitled “SBI MOTION ARTIFACTREMOVAL APPARATUS AND METHOD”, describes various systems for removing motion artifacts from image data, which is incorporated herein by reference in its entirety. Such systems may be integrated with imaging module 138. In addition, U.S. Patent Application Publication No. 2011 / 0306840, published on December 15, 2011, entitled “CONTROLLABLE MAGNETIC SOURCE TO FIXTUREINTRACORPOREAL APPARATUS,” and U.S. Patent Application Publication No. 2014 / 0243597, published on August 28, 2014, entitled “SYSTEM FOR PERFORMING A MINIMALLY INVASIVE SURGICAL PROCEDURE,” each of which is incorporated herein by reference in its entirety.

[0181] Figure 8A surgical data network 201 is shown that includes a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility or any room in a medical facility that is specifically equipped for surgical operations to a cloud-based system (e.g., a cloud 204 that may include a remote server 213 connected to a storage device 205). In one aspect, the modular communication hub 203 includes a network hub 207 and / or a network switch 209 that communicates with a network router. The modular communication hub 203 may also be connected to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, enabling it to be transferred from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network includes additional features to enable monitoring of traffic through the surgical data network and configuration of each port in the network hub 207 or network switch 209. The intelligent surgical data network may be referred to as a manageable hub or switch. The switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0182] Modular devices 1a-1n located in an operating room may be connected to a modular communication hub 203. A network hub 207 and / or a network switch 209 may be connected to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transmitted to a cloud-based computer via a router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transmitted to a local computer system 210 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be connected to a network switch 209. A network switch 209 may be connected to a network hub 207 and / or a network router 211 to connect devices 2a-2m to a cloud 204. Data associated with devices 2a-2n may be transmitted to a cloud 204 via a network router 211 for data processing and manipulation. Data associated with devices 2a-2m may also be transmitted to a local computer system 210 for local data processing and manipulation.

[0183] It should be understood that the surgical data network 201 can be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. The modular communication hub 203 can be included in a modular control tower that is configured to accommodate multiple devices 1a-1n / 2a-2m. A local computer system 210 can also be included in the modular control tower. The modular communication hub 203 is connected to a display 212 to display images obtained by some of the devices 1a-1n / 2a-2m during a surgical procedure, for example. In various aspects, devices 1a-1n / 2a-2m may include, for example, various modules, such as an imaging module 138 connected to an endoscope, a generator module 140 connected to an energy-based surgical device, a smoke evacuation module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module in other modular devices that can be connected to a modular communication hub 203 of a surgical data network 201.

[0184] In one aspect, the surgical data network 201 may include a combination of a network hub, a network switch, and a network router that connects the device 1a-1n / 2a-2m to the cloud. Any or all of the devices 1a-1n / 2a-2m connected to the network hub or network switch can collect data in real time and transmit the data to a cloud computer for data processing and manipulation. It should be understood that cloud computing relies on shared computing resources rather than using local servers or personal devices to process software applications. The term "cloud" can be used as a metaphor for "Internet", although the term is not limited to this. Therefore, the term "cloud computing" can be used herein to refer to "a type of Internet-based computing", in which different services (such as servers, storage, and applications) are delivered to a modular communication hub 203 and / or a computer system 210 located in a surgical room (e.g., a fixed, mobile, temporary, or on-site operating room or space) and a device connected to the modular communication hub 203 and / or the computer system 210 via the Internet. The cloud infrastructure can be maintained by a cloud service provider. In this case, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services may perform a large number of calculations based on data collected by smart surgical instruments, robots, and other computerized devices located in the operating room. Hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0185] Applying cloud computer data processing technology to the data collected by the devices 1a-1n / 2a-2m, the surgical data network provides improved surgical results, reduced costs and improved patient satisfaction. At least some of the devices 1a-1n / 2a-2m can be used to observe tissue status to evaluate leakage or perfusion of sealed tissue after tissue sealing and cutting procedures. At least some of the devices 1a-1n / 2a-2m can be used to identify pathology, such as the effects of disease, using cloud-based computing to examine data including images of body tissue samples for diagnostic purposes. This includes positioning and edge confirmation of tissue and phenotype. At least some of the devices 1a-1n / 2a-2m can be used to identify the anatomical structure of the body using various sensors integrated with imaging devices and technologies (such as overlapping images captured by multiple imaging devices). The data collected by the devices 1a-1n / 2a-2m (including image data) can be transmitted to the cloud 204 or the local computer system 210 or both for data processing and manipulation, including image processing and manipulation. The data can be analyzed to improve surgical procedure outcomes by determining whether further treatment can be pursued, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and precision robotic applications to tissue-specific sites and conditions. Such data analysis can be further processed using outcome analysis, and using standardized methods can provide helpful feedback to confirm surgical treatments and surgeon behavior, or suggest modifications to surgical treatments and surgeon behavior.

[0186] In one specific implementation, operating room devices 1a-1n may be connected to the modular communication hub 203 via a wired channel or a wireless channel, depending on the configuration of the devices 1a-1n to the network hub. In one aspect, the network hub 207 may be implemented as a local network broadcast device that operates on the physical layer of the open system interconnection (OSI) model. The network hub provides connections to devices 1a-1n located in the same operating room network. The network hub 207 collects data in the form of packets and sends it to a router in half-duplex mode. The network hub 207 does not store any media access control / internet protocol (MAC / IP) for transmitting device data. Only one of the devices 1a-1n can send data through the network hub 207 at a time. The network hub 207 has no knowledge of where to send information and broadcasts all network data on each connection and to the remote server 213 ( Fig. 9 ) Routing tables or intelligence that broadcasts all network data. The network hub 207 can detect basic network errors such as collisions, but broadcasting all information to multiple ports can pose security risks and cause bottlenecks.

[0187] In another specific implementation, the operating room devices 2a-2m can be connected to the network switch 209 through a wired channel or a wireless channel. The network switch 209 operates in the data link layer of the OSI model. The network switch 209 is a multicast device for connecting the devices 2a-2m located in the same operating room to the network. The network switch 209 sends data to the network router 211 in the form of a frame and operates in full-duplex mode. Multiple devices 2a-2m can send data simultaneously through the network switch 209. The network switch 209 stores and uses the MAC addresses of the devices 2a-2m to transmit data.

[0188] The network hub 207 and / or the network switch 209 are connected to a network router 211 to connect to the cloud 204. The network router 211 operates in the network layer of the OSI model. The network router 211 creates a route for transmitting data packets received from the network hub 207 and / or the network switch 211 to cloud-based computer resources to further process and manipulate the data collected by any or all of the devices 1a-1n / 2a-2m. The network router 211 can be used to connect two or more different networks located in different locations, such as, for example, different operating rooms in the same medical facility or different networks in different operating rooms in different medical facilities. The network router 211 sends data to the cloud 204 in the form of packets and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to transmit data.

[0189] In one example, the network hub 207 may be implemented as a USB hub that allows multiple USB devices to be connected to a host computer. The USB hub may expand a single USB port to multiple levels so that more ports are available to connect devices to a host system computer. The network hub 207 may include wired or wireless capabilities for receiving information through a wired channel or a wireless channel. In one aspect, a wireless USB short-range, high-bandwidth wireless radio communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in an operating room.

[0190] In other examples, the operating room devices 1a-1n / 2a-2m can communicate with the modular communication hub 203 via the Bluetooth wireless technology standard for exchanging data from fixed and mobile devices over short distances (using short wavelength UHF radio waves in the ISM band of 2.4 to 2.485 GHz) and building a personal area network (PAN). In other aspects, the operating room devices 1a-1n / 2a-2m can communicate with the modular communication hub 203 via a variety of wireless or wired communication standards or protocols, including but not limited to Wi-Fi (IEEE 802.11 series), WiMAX (IEEE 802.16 series), IEEE 802.20, Long Term Evolution (LTE) and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and Ethernet derivatives thereof, and any other wireless and wired protocols designated as 3G, 4G, 5G and above. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communications such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, etc.

[0191] The modular communication hub 203 can serve as a central connection for one or all of the operating room devices 1a-1n / 2a-2m and handles data types called frames. The frames carry data generated by the devices 1a-1n / 2a-2m. When the modular communication hub 203 receives a frame, it is amplified and transmitted to the network router 211, which transmits the data to the cloud computing resources using multiple wireless or wired communication standards or protocols as described herein.

[0192] Modular communication hub 203 can be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Modular communication hub 203 is generally easy to install, configure and maintain, making it a good option for networking operating room devices 1a-1n / 2a-2m.

[0193] Fig. 9A computer-implemented interactive surgical system 200 is shown. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to the surgical system 102. Each surgical system 202 includes at least one surgical hub 206 in communication with a cloud 204, which may include a remote server 213. In one aspect, the computer-implemented interactive surgical system 200 includes a modular control tower 236 that is connected to a plurality of operating room devices, such as, for example, smart surgical instruments, robots, and other computerized devices located in the operating room. Fig.10 As shown in FIG, modular control tower 236 includes modular communication hub 203 coupled to computer system 210. Fig. 9 As shown in the example of , the modular control tower 236 is connected to an imaging module 238 connected to an endoscope 239, a generator module 240 connected to an energy device 241, a smoke evacuator module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 optionally connected to a display 237, and a non-contact sensor module 242. The operating room devices are connected to cloud computing resources and data storage via the modular control tower 236. The robotic hub 222 can also be connected to the modular control tower 236 and cloud computing resources. The devices / instruments 235, the visualization system 208, etc. can be connected to the modular control tower 236 via wired or wireless communication standards or protocols, as described herein. The modular control tower 236 can be connected to the hub display 215 (e.g., monitor, screen) to display and overlay images received from the imaging module, device / instrument display and / or other visualization system 208. The hub display may also combine graphics and overlay images to display data received from devices connected to the modular control tower.

[0194] Fig.10 A surgical hub 206 is shown comprising a plurality of modules coupled to a modular control tower 236. The modular control tower 236 comprises a modular communication hub 203 (e.g., network connectivity means) and a computer system 210 to provide, for example, local processing, visualization, and imaging. Fig.10 As shown in , the modular communication hub 203 can be hierarchically configured to expand the number of modules (e.g., devices) that can be connected to the modular communication hub 203 and transmit data associated with the modules to the computer system 210, cloud computing resources, or both. Fig.10As shown in , each of the network hubs / switches in the modular communication hub 203 includes three downstream ports and one upstream port. The upstream network hub / switch is connected to the processor to provide communication connections with cloud computing resources and local display 217. Communication with the cloud 204 can be carried out through wired or wireless communication channels.

[0195] The surgical hub 206 employs a non-contact sensor module 242 to measure the size of the operating room and generates a map of the surgical room using an ultrasound or laser-based non-contact measurement device. The ultrasound-based non-contact sensor module scans the operating room by transmitting a burst of ultrasound waves and receiving the echo as it bounces off the walls of the operating room, as described in U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed on December 28, 2017, under the title “Surgical Hub Spatial Awareness Within an Operating Room,” the disclosure of which is incorporated herein by reference in its entirety, wherein the sensor module is configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. The laser-based non-contact sensor module scans the operating room by transmitting laser pulses, receiving laser pulses that bounce off the walls of the operating room, and comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0196] The computer system 210 includes a processor 244 and a network interface 245. The processor 244 is coupled to a communication module 247, a storage device 248, a memory 249, a non-volatile memory 250, and an input / output interface 251 via a system bus. The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures including, but not limited to, a 9-bit bus, Industry Standard Architecture (ISA), Micro Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer System Interface (SCSI), or any other peripheral bus.

[0197] Controller 244 may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be a commercially available LM4F230H5QR ARM Cortex-M4F processor core, such as Texas Instruments, which includes 256KB of single-cycle flash or other non-volatile memory (up to 40 MHz) on-chip memory, a pre-fetch buffer for improved performance above 40 MHz, 32KB of single-cycle sequential random access memory (SRAM), a 32KB MCU loaded with Internal read-only memory (ROM) for software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) simulations, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which can be found in the product data sheet.

[0198] In one aspect, the processor 244 may include a safety controller including two families based on controllers such as TMS570 and RM4x, known under the trade name Hercules ARMCortex R4 also produced by Texas Instruments. The safety controller may be configured specifically for IEC 61508 and ISO 26262 safety critical applications, among others, to provide advanced integrated safety features while delivering scalable performance, connectivity, and memory options.

[0199] System memory includes volatile memory and non-volatile memory. The basic input / output system (BIOS), which contains basic routines such as transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes random access memory (RAM) that acts as external cache memory. In addition, RAM is available in a variety of forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM) enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM) and direct Rambus RAM (DRRAM).

[0200] Computer system 210 also includes removable / non-removable, volatile / non-volatile computer storage media, such as, for example, disk storage. Disk storage includes, but is not limited to, devices such as disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage may include storage media individually or in combination with other storage media, including but not limited to optical disk drives such as optical disk ROM devices (CD-ROM), optical disk recordable drives (CD-R drives), optical disk rewritable drives (CD-RW drives), or digital universal disk ROM drives (DVD-ROM). In order to facilitate the connection of the disk storage device to the system bus, a removable or non-removable interface may be used.

[0201] It should be understood that the computer system 210 includes software that acts as an intermediary between a user and the basic computer resources described in a suitable operating environment. Such software includes an operating system. An operating system that can be stored on a disk storage device is used to control and allocate the resources of the computer system. System applications utilize the operating system to manage resources through program modules and program data stored in system memory or disk storage devices. It should be understood that the various components described herein can be implemented with various operating systems or combinations of operating systems.

[0202] The user inputs commands or information into the computer system 210 through an input device connected to the I / O interface 251. The input device includes, but is not limited to, a pointing device such as a mouse, a touch ball, a stylus, a touch pad, a keyboard, a microphone, a joystick, a game pad, a satellite dish, a scanner, a TV tuner card, a digital camera, a digital video camera, a web camera, etc. These and other input devices are connected to the processor through the system bus via an interface port. Interface ports include, for example, serial ports, parallel ports, game ports, and USB. Output devices use the same type of port as the input device. Thus, for example, a USB port can be used to provide input to the computer system and output information from the computer system to an output device. Output adapters are provided to illustrate that there are some output devices (such as monitors, displays, speakers, and printers) that require special adapters among other output devices. Output adapters include, by way of example, but are not limited to, video and sound cards that provide a connection device between the output device and the system bus. It should be noted that other devices or systems of devices such as remote computers provide both input capabilities and output capabilities.

[0203] Computer system 210 can be operated in a networked environment using a logical connection with one or more remote computers (such as cloud computers) or local computers. A remote cloud computer can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device or other public network node, etc., and generally includes many or all of the elements described relative to the computer system. For simplicity, only a memory storage device with a remote computer is shown. The remote computer is logically connected to the computer system through a network interface, and then physically connected via a communication connection. The network interface covers communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include fiber distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, token ring / IEEE 802.5, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variants thereof, packet switching networks, and digital subscriber lines (DSL).

[0204] In various aspects, Fig.10 computer system 210, imaging module 238 and / or visualization system 208, and / or Figures 9 and 10 The processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) for processing digital images. The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instructions, multiple data (MIMD) techniques to improve speed and efficiency. The digital image processing engine may perform a range of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0205] The communication connection refers to the hardware / software used to connect the network interface to the bus. Although the communication connection is shown for exemplary clarity within the computer system, it can also be located external to the computer system 210. The hardware / software necessary to connect to the network interface includes internal and external technologies such as modems, including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters, and Ethernet cards for exemplary purposes only.

[0206] Fig.11A functional block diagram of one aspect of a USB network hub 300 device according to one aspect of the present disclosure is shown. In the illustrated aspect, the USB network hub device 300 employs a TUSB2036 integrated circuit hub from Texas Instruments. The USB network hub 300 is a CMOS device that provides an upstream USB transceiver port 302 and up to three downstream USB transceiver ports 304, 306, 308 according to the USB 2.0 specification. The upstream USB transceiver port 302 is a differential root data port that includes a differential data negative (DP0) input paired with a differential data positive (DM0) input. The three downstream USB transceiver ports 304, 306, 308 are differential data ports, each of which includes a differential data positive (DP1-DP3) output paired with a differential data negative (DM1-DM3) output.

[0207] The USB network hub 300 device is implemented with a digital state machine rather than a microcontroller, and does not require firmware programming. A fully compatible USB transceiver is integrated into the circuit for the upstream USB transceiver port 302 and all downstream USB transceiver ports 304, 306, 308. The downstream USB transceiver ports 304, 306, 308 support both full-speed devices and low-speed devices by automatically setting the conversion rate according to the speed of the device attached to the port. The USB network hub 300 device can be configured to be in bus-powered mode or self-powered mode, and includes a hub power logic 312 for managing power.

[0208] The USB network hub 300 device includes a serial interface engine 310 (SIE). The SIE 310 is the front end of the USB network hub 300 hardware and handles most of the protocols described in Chapter 8 of the USB specification. The SIE 310 generally includes up to transaction-level signaling. The functions it handles may include: packet identification, transaction ordering, SOP, EOP, RESET and RESUME signal detection / generation, clock / data separation, non-return to zero inversion (NRZI) data encoding / decoding and digit filling, CRC generation and verification (token and data), packet ID (PID) generation and verification / decoding, and / or serial-to-parallel / parallel-to-serial conversion. The 310 receives a clock input 314 and is connected to the pause / resume logic and frame timer 316 circuits and the hub repeater circuit 318 to control the communication between the upstream USB transceiver port 302 and the downstream USB transceiver port 304, 306, 308 through the port logic circuits 320, 322, 324. SIE 310 is coupled to command decoder 326 via interface logic to control commands from the serial EEPROM via serial EEPROM interface 330 .

[0209] In various aspects, the USB network hub 300 can connect 127 functions configured in up to six logical layers (hierarchies) to a single computer. In addition, the USB network hub 300 can be connected to all external devices using a standardized four-wire cable that provides both communication and power distribution. The power is configured as a bus-powered mode and a self-powered mode. The USB network hub 300 can be configured to support four power management modes: a bus-powered hub with individual port power management or a complete set of port power management, and a self-powered hub with individual port power management or a complete set of port power management. In one aspect, the USB network hub 300, the upstream USB transceiver port 302 is inserted into the USB host controller using a USB cable, and the downstream USB transceiver ports 304, 306, 308 are exposed for connecting USB compatible devices, etc.

[0210] Surgical Instrument Hardware

[0211] Fig.12 A logic diagram of a control system 470 of a surgical instrument or tool according to one or more aspects of the present disclosure is shown. The system 470 includes a control circuit. The control circuit includes a microcontroller 461, which includes a processor 462 and a memory 468. For example, one or more of the sensors 472, 474, 476 provide real-time feedback to the processor 462. A motor 482 driven by a motor driver 492 is operably coupled to a longitudinally movable displacement member to drive an I-beam blade element. The tracking system 480 is configured to determine the position of the longitudinally movable displacement member. The position information is provided to the processor 462, which can be programmed or configured to determine the position of the longitudinally movable drive member and the position of the firing member, the firing rod, and the I-beam blade element. Additional motors can be provided at the tool driver interface to control I-beam firing, closure tube travel, shaft rotation, and joint movement. The display 473 displays a variety of operating conditions of the instrument and can include a touch screen function for data input. The information displayed on the display 473 can be superimposed with an image acquired via an endoscopic imaging module.

[0212] In one aspect, microprocessor 461 can be any single-core or multi-core processor, such as those known under the trade name ARM Cortex produced by Texas Instruments. In one aspect, microcontroller 461 can be a LM4F230H5QR ARM Cortex-M4F processor core available from, for example, Texas Instruments, which includes 256KB of single-cycle flash or other non-volatile memory (up to 40 MHz) on-chip memory, a pre-fetch buffer for improved performance above 40 MHz, 32KB of single-cycle SRAM, a CPU loaded with Internal ROM for software, 2KB electrical EEPROM, one or more PWM modules, one or more QEI analog, one or more 12-bit ADC with 12 analog input channels, details of which can be found in the product datasheet.

[0213] In one aspect, the microcontroller 461 may include a safety controller including two families based on controllers such as TMS570 and RM4x, known under the trade name HerculesARM Cortex R4 also produced by Texas Instruments. The safety controller may be configured specifically for IEC 61508 and ISO 26262 safety critical applications, among others, to provide advanced integrated safety features while delivering scalable performance, connectivity, and memory options.

[0214] The microcontroller 461 can be programmed to perform various functions, such as precise control of the speed and position of the knife and articulation system. In one aspect, the microcontroller 461 includes a processor 462 and a memory 468. The electric motor 482 can be a brushed DC (DC) motor having a gearbox and a mechanical link to the articulation or knife system. In one aspect, the motor driver 492 can be an A3941 available from Allegro Microsystems, Inc. Other motor drivers can be easily replaced for use in the tracking system 480 including an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213 entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT" published on October 19, 2017, which is incorporated herein by reference in its entirety.

[0215] The microcontroller 461 can be programmed to provide precise control of the speed and position of the displacement member and articulation system. The microcontroller 461 can be configured to calculate the response in the software of the microcontroller 461. The calculated response is compared to the measured response of the actual system to obtain the "observed" response, which is used for actual feedback decisions. The observed response is a favorable tuning value that balances the smooth continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0216] In one aspect, the motor 482 can be controlled by a motor driver 492 and can be adopted by the firing system of a surgical instrument or tool. In various forms, the motor 482 can be a brushed DC drive motor with a maximum rotation speed of about 25,000 RPM. In other arrangements, the motor 482 can include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 492 can include, for example, an H-bridge driver including a field effect transistor (FET). The motor 482 can be powered by a power assembly that can be releasably mounted to a handle assembly or a tool housing for supplying control power to a surgical instrument or tool. The power assembly can include a battery that can include a plurality of battery cells connected in series that can be used as a power source to provide power to a surgical instrument or tool. In some cases, the battery cells of the power assembly can be replaceable and / or rechargeable. In at least one example, the battery cell can be a lithium-ion battery that can be connected to the power assembly and can be separated from the power assembly.

[0217] Driver 492 may be an A3941 available from Allegro Microsystems, Inc. A3941 492 is a full-bridge controller for use with external N-channel power metal oxide semiconductor field effect transistors (MOSFETs) specifically designed for inductive loads such as brushed DC motors. Driver 492 includes a unique charge pump regulator that provides full (>10V) gate drive for battery voltages as low as 7V and allows the A3941 to operate at reduced gate drive as low as 5.5V. A bootstrap capacitor may be used to provide the above-mentioned battery supply voltage required for the N-channel MOSFET. The internal charge pump of the high-side driver allows DC (100% duty cycle) operation. The full bridge may be driven in fast decay mode or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation may pass through the high-side or low-side FETs. The power FETs are protected from breakdown by resistor-adjustable dead time. Overall diagnostics provide indications of undervoltage, overtemperature, and power bridge faults, and may be configured to protect the power MOSFETs under most short circuit conditions. Other motor drives may be easily substituted for use in tracking system 480 including an absolute positioning system.

[0218] The tracking system 480 includes a controlled motor drive circuit arrangement including a position sensor 472 according to one aspect of the present disclosure. The position sensor 472 for the absolute positioning system provides a unique position signal corresponding to the position of the displacement member. In one aspect, the displacement member represents a longitudinally movable drive member, which includes a rack of drive teeth for meshing engagement with a corresponding drive gear of the gear reducer assembly. In other aspects, the displacement member represents a firing member, which can be adapted and configured to include a rack of drive teeth. In another aspect, the displacement member represents a firing rod or an I-beam, each of which can be adapted and configured to include a rack of drive teeth. Therefore, as used herein, the term displacement member is generally used to refer to any movable member of a surgical instrument or tool such as a drive member, a firing member, a firing rod, an I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member is connected to the firing member, the firing rod, and the I-beam. Therefore, the absolute positioning system can actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various other aspects, the displacement member may be coupled to any position sensor 472 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, firing member, firing rod or I-beam or a combination thereof may be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor. The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system including a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system including a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system including a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system including a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0219] The electric motor 482 may include a rotatable shaft operably interfacing with a gear assembly that is mounted on the displacement member in meshing engagement with a group or rack of drive teeth. The sensor element may be operably coupled to the gear assembly so that a single rotation of the position sensor 472 element corresponds to some linear longitudinal translation of the displacement member. The arrangement of the transmission and the sensor may be connected to a linear actuator via a rack and pinion arrangement, or to a rotary actuator via a spur gear or other connection. The power source supplies power to the absolute positioning system, and the output indicator may display the output of the absolute positioning system. The displacement member represents a longitudinally movable drive member that includes a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member represents a longitudinally movable firing member, a firing rod, an I-beam, or a combination thereof.

[0220] A single rotation of the sensor element associated with the position sensor 472 is equivalent to a longitudinal linear displacement of the displacement member d1, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after a single rotation of the sensor element coupled to the displacement member. The sensor arrangement may be connected via a gear reduction that enables the position sensor 472 to complete only one or more rotations for a full stroke of the displacement member. The position sensor 472 may complete multiple rotations for a full stroke of the displacement member.

[0221] A series of switches (where n is an integer greater than one) may be employed alone or in combination with gear reduction to provide unique position signals for more than one rotation of the position sensor 472. The state of the switches is fed back to the microcontroller 461, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 472 is provided to the microcontroller 461. The position sensor 472 of this sensor arrangement may include a magnetic sensor, an analog rotation sensor (such as a potentiometer), an array of analog Hall effect elements that output a unique combination of position signals or values.

[0222] Position sensor 472 may include any number of magnetic sensing elements, such as, for example, magnetic sensors classified according to whether they measure the total magnetic field or the vector component of the magnetic field. The technologies used to produce the above two types of magnetic sensors cover many aspects of physics and electronics. Technologies for magnetic field sensing include probe coils, fluxgates, optical pumps, nuclear spins, superconducting quantum interference devices (SQUIDs), Hall effects, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetoimpedance, magnetostrictive / piezoelectric composites, magnetosensitive diodes, magnetosensitive transistors, optical fibers, magneto-optics, and magnetic sensors based on micro-electromechanical systems, among others.

[0223] In one aspect, the position sensor 472 for the tracking system 480 including the absolute positioning system includes a magnetic rotation absolute positioning system. The position sensor 472 can be implemented as an AS5055EQFT monolithic magnetic rotation position sensor, which can be purchased from Austria Microsystems, AG. The position sensor 472 interfaces with the microcontroller 461 to provide an absolute positioning system. The position sensor 472 is a low voltage and low power component and includes four Hall effect elements in the area of ​​the position sensor 472 located on the magnet. A high resolution ADC and an intelligent power management controller are also provided on the chip. A coordinate rotation digital computer (CORDIC) processor (also known as the bitwise method and Volder algorithm) is provided to perform a simple and efficient algorithm to calculate hyperbolic and trigonometric functions, which only requires addition, subtraction, digit shifting and table lookup operations. The angular position, alarm bit and magnetic field information are transmitted to the microcontroller 461 via a standard serial communication interface (such as a serial peripheral interface (SPI) interface). The position sensor 472 provides 12 or 14 bit resolution. The position sensor 472 may be an AS5055 chip provided in a small QFN 16-pin 4×4×0.85 mm package.

[0224] The tracking system 480 including the absolute positioning system may include and / or may be programmed to implement a feedback controller such as a PID, state feedback, and an adaptive controller. The power source converts the signal from the feedback controller into a physical input to the system: in this case, a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 472, other sensors may be provided to measure physical parameters of the physical system. In some aspects, other sensors may include sensor arrangements such as those described in U.S. Patent No. 9,345,481, entitled “STAPLE CARTRIDGE TISSUETHICKNESS,” issued May 24, 2016, which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, entitled “STAPLE CARTRIDGE TISSUE THICKNESS,” issued September 18, 2014, which is incorporated herein by reference in its entirety; and U.S. Patent Application Serial No. 15 / 628,175, entitled “TECHNIQUESFOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT,” filed June 20, 2017, which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, wherein the output of the absolute positioning system will have a finite resolution and sampling frequency. The absolute positioning system may include comparison and combination circuits to combine the calculated response with the measured response using algorithms (such as weighted averages and theoretical control loops) that drive the calculated response toward the measured response. The calculated response of the physical system takes into account properties such as mass, inertia, viscous friction, inductance and resistance to predict the state and output of the physical system by knowing the input.

[0225] Thus, the absolute positioning system provides the absolute position of the displacement member when the instrument is powered on, and does not retract or advance the displacement member to a reset (zero or home) position as may be required with conventional rotary encoders, which simply count the number of forward or backward steps taken by the motor 482 to infer the position of the device actuator, drive rod, knife, etc.

[0226] The sensor 474 (such as, for example, a strain gauge or a micro strain gauge) is configured to measure one or more parameters of the end effector, such as, for example, the amplitude of the strain applied to the anvil during the clamping operation, which can indicate the closing force applied to the anvil. The measured strain is converted into a digital signal and provided to the processor 462. Alternatively or in addition to the sensor 474, a sensor 476 (such as, for example, a load sensor) can measure the closing force applied to the anvil by the closed drive system. The sensor 476 such as, for example, a load sensor can measure the firing force applied to the I-beam in the firing stroke of the surgical instrument or tool. The I-beam is configured to engage a wedge-shaped sliding member, which is configured to cam the nail driver upward to push the nail out to contact with the deformation of the anvil. The I-beam also includes a sharp cutting edge, which can be used to cut off tissue when the I-beam is pushed distally by the firing rod. Alternatively, a current sensor 478 can be used to measure the current consumed by the motor 482. The force required to advance the firing member may correspond to, for example, the current consumed by the motor 482. The measured force is converted into a digital signal and provided to the processor 462.

[0227] In one form, a strain gauge sensor 474 can be used to measure the force applied to the tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force on the tissue processed by the end effector. A system for measuring the force applied to the tissue grasped by the end effector includes a strain gauge sensor 474, such as, for example, a micro strain gauge, which is configured to measure, for example, one or more parameters of the end effector. In one aspect, the strain gauge sensor 474 can measure the amplitude or magnitude of the strain applied to the jaw member of the end effector during the clamping operation, which can indicate tissue compression. The measured strain is converted into a digital signal and provided to the processor 462 of the microcontroller 461. The load sensor 476 can measure the force used to operate the knife element, for example, to cut the tissue captured between the anvil and the nail magazine. A magnetic field sensor can be used to measure the thickness of the captured tissue. The measurement value of the magnetic field sensor can also be converted into a digital signal and provided to the processor 462.

[0228] The microcontroller 461 can characterize the corresponding values ​​of the selected position of the firing member and / or the velocity of the firing member using the measurements of tissue compression, tissue thickness, and / or the force required to close the end effector measured by the sensors 474, 476, respectively. In one example, the memory 468 can store techniques, formulas, and / or lookup tables that can be used by the microcontroller 461 in the evaluation.

[0229] The control system 470 of the surgical instrument or tool may also include wired or wireless communication circuitry to communicate with the modular communication hub, such as Figure 8-11 as shown in .

[0230] Fig.13A control circuit 500 is shown, which is configured to control various aspects of a surgical instrument or tool according to one aspect of the present disclosure. The control circuit 500 may be configured to implement various processes described herein. The circuit 500 may include a microcontroller, which includes one or more processors 502 (e.g., microprocessors, microcontrollers) connected to at least one memory circuit 504. The memory circuit 504 stores machine executable instructions that cause the processor 502 to execute machine instructions to implement various processes described herein when executed by the processor 502. The processor 502 may be any of a variety of single-core or multi-core processors known in the art. The memory circuit 504 may include a volatile storage medium and a non-volatile storage medium. The processor 502 may include an instruction processing unit 506 and an operation unit 508. The instruction processing unit may be configured to receive instructions from the memory circuit 504 of the present disclosure.

[0231] Fig.14 Combinatorial logic circuit 510 is shown, which is configured to control various aspects of a surgical instrument or tool according to one aspect of the present disclosure. Combinatorial logic circuit 510 can be configured to implement various processes described herein. Combinatorial logic circuit 510 may include a finite state machine that includes combinatorial logic 512 that is configured to receive data associated with a surgical instrument or tool at input 514, process the data through combinatorial logic 512, and provide output 516.

[0232] Fig.15 A sequential logic circuit 520 configured to control various aspects of a surgical instrument or tool is shown according to one aspect of the present disclosure. The sequential logic circuit 520 or the combinatorial logic 522 may be configured to implement the various processes described herein. The sequential logic circuit 520 may include a finite state machine. The sequential logic circuit 520 may include, for example, the combinatorial logic 522, at least one memory circuit 524, and a clock 529. The at least one memory circuit 524 may store the current state of the finite state machine. In some cases, the sequential logic circuit 520 may be synchronous or asynchronous. The combinatorial logic 522 is configured to receive data associated with a surgical instrument or tool from an input 526, process the data through the combinatorial logic 522, and provide an output 528. In other aspects, the circuit may include a processor (e.g., the processor 502, Fig.13 ) and a finite state machine to implement the various processes herein. In other embodiments, the finite state machine may include a combinational logic circuit (e.g., combinational logic circuit 510, Fig.14 ) and a sequential logic circuit 520.

[0233] Fig.16A surgical instrument or tool is shown that includes multiple motors that can be activated to perform various functions. In some cases, a first motor can be activated to perform a first function, a second motor can be activated to perform a second function, and a third motor can be activated to perform a third function. In some cases, multiple motors of the robotic surgical instrument 600 can be individually activated to cause a firing motion, a closing motion, and / or an articulation motion in an end effector. The firing motion, the closing motion, and / or the articulation motion can be transmitted to the end effector, for example, via a shaft assembly.

[0234] In some cases, the surgical instrument system or tool may include a firing motor 602. The firing motor 602 may be operably coupled to a firing motor drive assembly 604, which may be configured to transmit the firing motion generated by the motor 602 to the end effector, specifically for displacing an I-shaped beam element. In some cases, the firing motion generated by the motor 602 may cause, for example, a staple to be deployed from a staple cartridge into tissue captured by the end effector and / or cause the cutting edge of the I-shaped beam element to be advanced to cut captured tissue. The I-shaped beam element may be retracted by reversing the direction of the motor 602.

[0235] In some cases, the surgical instrument or tool may include a closing motor 603. The closing motor 603 may be operably coupled to a closing motor drive assembly 605, which is configured to transmit the closing motion generated by the motor 603 to the end effector, specifically for displacing the closing tube to close the anvil and compress the tissue between the anvil and the staple cartridge. The closing motion may cause, for example, the end effector to transition from an open configuration to a close configuration to capture tissue. The end effector may be transitioned to an open position by reversing the direction of the motor 603.

[0236] In some cases, the surgical instrument or tool may include, for example, one or more articulation motors 606a, 606b. The motors 606a, 606b may be operably coupled to respective articulation motor drive assemblies 608a, 608b, which may be configured to transmit the articulation generated by the motors 606a, 606b to the end effector. In some cases, the articulation may cause the end effector to articulate relative to an axis, for example.

[0237] As described above, the surgical instrument or tool may include multiple motors that may be configured to perform various independent functions. In some cases, multiple motors of the surgical instrument or tool may be individually or independently activated to perform one or more functions, while other motors remain inactive. For example, articulation motors 606a, 606b may be activated to articulate the end effector, while firing motor 602 remains inactive. Alternatively, firing motor 602 may be activated to fire multiple nails and / or advance the cutting edge, while articulation motor 606 remains inactive. In addition, closing motor 603 may be activated simultaneously with firing motor 602 to advance the closing tube and I-beam element toward the distal side, as described in more detail below.

[0238] In some cases, a surgical instrument or tool may include a common control module 610 that can be used with multiple motors of the surgical instrument or tool. In some cases, the common control module 610 can adapt to one of the multiple motors at a time. For example, the common control module 610 can be individually connected to multiple motors of the surgical instrument and can be separated from the multiple motors of the surgical instrument. In some cases, multiple motors of the surgical instrument or tool can share one or more common control modules such as the common control module 610. In some cases, multiple motors of the surgical instrument or tool can independently and selectively engage the common control module 610. In some cases, the common control module 610 can switch from interfacing with one of the multiple motors of the surgical instrument or tool to interfacing with another of the multiple motors of the surgical instrument or tool.

[0239] In at least one example, the common control module 610 can selectively switch between operably engaging the articulation motors 606a, 606b and operably engaging the firing motor 602 or the closing motor 603. Fig.16 As shown in , the switch 614 can move or change between multiple positions and / or states. In the first position 616, the switch 614 can electrically connect the common control module 610 to the firing motor 602; in the second position 617, the switch 614 can electrically connect the common control module 610 to the closing motor 603; in the third position 618a, the switch 614 can electrically connect the common control module 610 to the first joint motion motor 606a; and in the fourth position 618b, the switch 614 can electrically connect the common control module 610 to, for example, the second joint motion motor 606b. In some cases, a separate common control module 610 can be electrically connected to the firing motor 602, the closing motor 603, and the joint motion motors 606a, 606b at the same time. In some cases, the switch 614 can be a mechanical switch, an electromechanical switch, a solid-state switch, or any suitable switch mechanism.

[0240] Each of the motors 602, 603, 606a, 606b may include a torque sensor to measure the output torque on the shaft of the motor. The force on the end effector may be sensed in any conventional manner, such as by a force sensor on the outside of the jaws or by a torque sensor of the motor used to actuate the jaws.

[0241] In various cases, such as Fig.16 As shown in , the common control module 610 may include a motor driver 626, which may include one or more H-bridge field effect FETs. The motor driver 626 may regulate the power transmitted from a power source 628 to a motor coupled to the common control module 610, for example, based on input from a microcontroller 620 ("controller"). In some cases, when a motor is coupled to the common control module 610, the microcontroller 620 may be used, for example, to determine the current consumed by the motor, as described above.

[0242] In some cases, microcontroller 620 may include a microprocessor 622 ("processor") and one or more non-transitory computer-readable media or storage units 624 ("memory"). In some cases, memory 624 may store various program instructions that, when executed, may cause processor 622 to perform a number of functions and / or calculations described herein. In some cases, one or more of memory units 624 may be coupled to processor 622, for example.

[0243] In some cases, the power source 628 can be used, for example, to power the microcontroller 620. In some cases, the power source 628 can include a battery (or "battery pack" or "power pack"), such as a lithium-ion battery, for example. In some cases, the battery pack can be configured to be releasably mounted to the handle for powering the surgical instrument 600. A plurality of battery cells connected in series can be used as the power source 628. In some cases, the power source 628 can be, for example, replaceable and / or rechargeable.

[0244] In various cases, the processor 622 may control the motor driver 626 to control the position, rotation direction, and / or speed of the motor connected to the common controller 610. In some cases, the processor 622 may signal the motor driver 626 to stop and / or deactivate the motor connected to the common controller 610. It should be understood that the term "processor" as used herein includes any suitable microprocessor, microcontroller, or other basic computing device that combines the functions of the central processing unit (CPU) of a computer on an integrated circuit or at most several integrated circuits. A processor is a multi-purpose programmable device that receives digital data as input, processes the input according to instructions stored in its memory, and then provides a result as output. Because the processor has internal memory, it is an example of sequential digital logic. The operating objects of the processor are numbers and symbols represented in the binary number system.

[0245] In one example, processor 622 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex produced by Texas Instruments. In some cases, microcontroller 620 may be, for example, LM 4F230H5QR available from Texas Instruments. In at least one example, the Texas Instruments LM4F230H5QR is an ARM Cortex-M4F processor core that includes: 256KB of single-cycle flash or other non-volatile memory (up to 40MHZ) on-chip memory, a pre-fetch buffer for improved performance above 40MHz, 32KB of single-cycle SRAM, a Internal ROM for software, 2KB of EEPROM, one or more PWM modules, one or more QEI simulations, one or more 12-bit ADCs with 12 analog input channels, and other features readily available. Other microcontrollers can be easily substituted for use with module 4410. Therefore, the present disclosure should not be limited to this context.

[0246] In some cases, the memory 624 may include program instructions for controlling each of the motors of the surgical instrument 600 that may be coupled to the common controller 610. For example, the memory 624 may include program instructions for controlling the firing motor 602, the closing motor 603, and the articulation motors 606a, 606b. Such program instructions may cause the processor 622 to control the firing, closing, and articulation functions based on input from an algorithm or control program of the surgical instrument or tool.

[0247] In some cases, one or more mechanisms and / or sensors (such as, for example, sensor 630) can be used to alert the processor 622 to program instructions that should be used in a particular setting. For example, the sensor 630 can alert the processor 622 to use program instructions associated with firing, closing, and articulating the end effector. In some cases, the sensor 630 can include, for example, a position sensor that can be used to sense the position of the switch 614. Thus, the processor 622 can use program instructions associated with firing the I-beam of the end effector when, for example, the sensor 630 detects that the switch 614 is in the first position 616; the processor 622 can use program instructions associated with closing the anvil when, for example, the sensor 630 detects that the switch 614 is in the second position 617; and the processor 622 can use program instructions associated with articulating the end effector when, for example, the sensor 630 detects that the switch 614 is in the third position 618a or the fourth position 618b.

[0248] Fig.17 7 is a schematic diagram of a robotic surgical instrument 700 configured to operate the surgical tools described herein according to one aspect of the present disclosure. The robotic surgical instrument 700 can be programmed or configured to control distal / proximal translation of a displacement member, distal / proximal displacement of a closure tube, shaft rotation, and articulation with a single or multiple articulation drive links. In one aspect, the surgical instrument 700 can be programmed or configured to individually control a firing member, a closing member, an axis member, and / or one or more articulation members. The surgical instrument 700 includes a control circuit 710 configured to control a motor-driven firing member, a closing member, an axis member, and / or one or more articulation members.

[0249] In one aspect, the robotic surgical instrument 700 includes a control circuit 710 configured to control the anvil 716 and I-beam 714 (including sharp cutting edges) portions of the end effector 702 via a plurality of motors 704a-704e, a removable staple cartridge 718, an axis 740, and one or more articulation members 742a, 742b. A position sensor 734 may be configured to provide position feedback of the I-beam 714 to the control circuit 710. Other sensors 738 may be configured to provide feedback to the control circuit 710. A timer / counter 731 provides timing and counting information to the control circuit 710. An energy source 712 may be provided to operate the motors 704a-704e, and a current sensor 736 provides motor current feedback to the control circuit 710. The motors 704a-704e may be individually operated by the control circuit 710 in open-loop or closed-loop feedback control.

[0250] In one aspect, the control circuit 710 may include one or more microcontrollers, microprocessors, or other suitable processors for executing instructions that cause the one or more processors to perform one or more tasks. In one aspect, the timer / counter 731 provides an output signal, such as an elapsed time or a digital count, to the control circuit 710 to correlate the position of the I-beam 714 as determined by the position sensor 734 with the output of the timer / counter 731 so that the control circuit 710 can determine the position of the I-beam 714 at a particular time (t) relative to a starting position or a time (t) when the I-beam 714 is at a particular position relative to a starting position. The timer / counter 731 can be configured to measure elapsed time, count external events, or time external events.

[0251] In one aspect, the control circuit 710 can be programmed to control the function of the end effector 702 based on one or more tissue conditions. The control circuit 710 can be programmed to sense tissue conditions, such as thickness, directly or indirectly, as described herein. The control circuit 710 can be programmed to select a firing control program or a closing control program based on the tissue conditions. The firing control program can describe the distal movement of the displacement member. Different firing control programs can be selected to better handle different tissue conditions. For example, when there is thicker tissue, the control circuit 710 can be programmed to translate the displacement member at a lower speed and / or at a lower power. When there is thinner tissue, the control circuit 710 can be programmed to translate the displacement member at a higher speed and / or at a higher power. The closing control program can control the closing force applied to the tissue by the anvil 716. Other control programs control the rotation of the shaft 740 and the articulation members 742a, 742b.

[0252] In one aspect, the control circuit 710 may generate a motor set point signal. The motor set point signal may be provided to various motor controllers 708a-708e. The motor controllers 708a-708e may include one or more circuits configured to provide motor drive signals to the motors 704a-704e to drive the motors 704a-704e as described herein. In some examples, the motors 704a-704e may be brushed DC electric motors. For example, the speed of the motors 704a-704e may be proportional to the corresponding motor drive signals. In some examples, the motors 704a-704e may be brushless DC motors, and the corresponding motor drive signals may include PWM signals provided to one or more stator windings of the motors 704a-704e. Moreover, in some examples, the motor controllers 708a-708e may be omitted, and the control circuit 710 may directly generate the motor drive signals.

[0253] In some examples, the control circuit 710 may initially operate each of the motors 704a-704e in an open-loop configuration for a first open-loop portion of the travel of the displacement member. Based on the response of the robotic surgical instrument 700 during the open-loop portion of the travel, the control circuit 710 may select a firing control program in a closed-loop configuration. The response of the instrument may include the translation distance of the displacement member during the open-loop portion, the time consumed during the open-loop portion, the energy provided to one of the motors 704a-704e during the open-loop portion, the sum of the pulse widths of the motor drive signal, etc. After the open-loop portion, the control circuit 710 may implement the selected firing control program for a second portion of the travel of the displacement member. For example, during the closed-loop portion of the travel, the control circuit 710 may modulate one of the motors 704a-704e based on the translation data describing the position of the displacement member in a closed-loop manner to cause the displacement member to translate at a constant speed.

[0254] In one aspect, the motors 704a-704e may receive power from the energy source 712. The energy source 712 may be a DC power source driven by a main AC power source, a battery, a supercapacitor, or any other suitable energy source. The motors 704a-704e may be mechanically coupled to separate movable mechanical elements, such as the I-beam 714, the anvil 716, the shaft 740, the articulation 742a, and the articulation 742b, via corresponding transmissions 706a-706e. The transmissions 706a-706e may include one or more gears or other linkage components to couple the motors 704a-704e to the movable mechanical elements. The position sensor 734 may sense the position of the I-beam 714. The position sensor 734 may be or include any type of sensor capable of generating position data indicating the position of the I-beam 714. In some examples, the position sensor 734 may include an encoder configured to provide a series of pulses to the control circuit 710 when the I-beam 714 is translated distally and proximally. The control circuit 710 can track the pulses to determine the position of the I-beam 714. Other suitable position sensors can be used, including, for example, proximity sensors. Other types of position sensors can provide other signals indicating the movement of the I-beam 714. Moreover, in some examples, the position sensor 734 can be omitted. In the case where any of the motors 704a-704e is a stepper motor, the control circuit 710 can track the position of the I-beam 714 by aggregating the number and direction of steps that the motor 704 has been instructed to perform. The position sensor 734 can be located in the end effector 702 or at any other part of the instrument. The output of each of the motors 704a-704e includes a torque sensor 744a-744e for sensing force, and has an encoder for sensing the rotation of the drive shaft.

[0255] In one aspect, the control circuit 710 is configured to drive a firing member such as an I-beam 714 portion of the end effector 702. The control circuit 710 provides a motor set point to a motor control 708a, which provides a drive signal to the motor 704a. The output shaft of the motor 704a is coupled to a torque sensor 744a. The torque sensor 744a is coupled to a transmission 706a, which is coupled to the I-beam 714. The transmission 706a includes a movable mechanical element such as a rotating element and a firing member to control the movement of the I-beam 714 along the longitudinal axis of the end effector 702 toward the distal and proximal sides. In one aspect, the motor 704a can be coupled to a knife gear assembly, which includes a knife gear reduction group, and the knife gear reduction group includes a first knife drive gear and a second knife drive gear. The torque sensor 744a provides a firing force feedback signal to the control circuit 710. The firing force signal represents the force required to fire or displace the I-beam 714. The position sensor 734 can be configured to provide the position of the I-beam 714 along the firing stroke or the position of the firing member as a feedback signal to the control circuit 710. The end effector 702 may include an additional sensor 738 configured to provide a feedback signal to the control circuit 710. When ready for use, the control circuit 710 can provide a firing signal to the motor control 708a. In response to the firing signal, the motor 704a can drive the firing member from the proximal stroke start position to the distal end of the stroke start position along the longitudinal axis of the end effector 702. When the firing member is translated toward the distal side, the I-beam 714 having a cutting element positioned at the distal end is advanced toward the distal side to cut the tissue between the nail magazine 718 and the anvil 716.

[0256] In one aspect, the control circuit 710 is configured to drive a closing member, such as the anvil 716 portion of the end effector 702. The control circuit 710 provides a motor set point to the motor control 708b, which provides a drive signal to the motor 704b. The output shaft of the motor 704b is coupled to a torque sensor 744b. The torque sensor 744b is coupled to a transmission 706b coupled to the anvil 716. The transmission 706b includes a movable mechanical element such as a rotating element and a closing member to control the movement of the anvil 716 from an open position and a closed position. In one aspect, the motor 704b is coupled to a closing gear assembly, which includes a closing reduction gear set supported to mesh with a closing spur gear. The torque sensor 744b provides a closing force feedback signal to the control circuit 710. The closing force feedback signal represents the closing force applied to the anvil 716. The position sensor 734 can be configured to provide the position of the closing member as a feedback signal to the control circuit 710. The additional sensor 738 in the end effector 702 can provide a closing force feedback signal to the control circuit 710. The pivotable anvil 716 is positioned opposite the staple cartridge 718. When ready for use, the control circuit 710 can provide a closing signal to the motor control 708b. In response to the closing signal, the motor 704b advances the closing member to grasp the tissue between the anvil 716 and the staple cartridge 718.

[0257] In one aspect, the control circuit 710 is configured to rotate an axis member such as the axis 740 to rotate the end effector 702. The control circuit 710 provides a motor set point to the motor control 708c, which provides a drive signal to the motor 704c. The output shaft of the motor 704c is coupled to a torque sensor 744c. The torque sensor 744c is coupled to a transmission 706c coupled to the axis 740. The transmission 706c includes a movable mechanical element such as a rotating element to control the axis 740 to rotate clockwise or counterclockwise more than 360°. In one aspect, the motor 704c is coupled to a rotation transmission assembly, which includes a tube gear section formed on (or attached to) the proximal end of the proximal closure tube to be operably engaged by a rotation gear assembly operably supported on a tool mounting plate. The torque sensor 744c provides a rotational force feedback signal to the control circuit 710. The rotational force feedback signal represents the rotational force applied to the axis 740. The position sensor 734 may be configured to provide the position of the closing member as a feedback signal to the control circuit 710. An additional sensor 738, such as a shaft encoder, may provide the rotational position of the shaft 740 to the control circuit 710.

[0258] In one aspect, the control circuit 710 is configured to articulate the end effector 702. The control circuit 710 provides a motor set point to a motor control 708d, which provides a drive signal to a motor 704d. The output shaft of the motor 704d is coupled to a torque sensor 744d. The torque sensor 744d is coupled to a transmission 706d coupled to the articulation member 742a. ​​The transmission 706d includes a movable mechanical element such as an articulation element to control the articulation of the end effector 702 ± 65°. In one aspect, the motor 704d is coupled to an articulation nut that is rotatably journaled on a proximal end portion of a distal spine and rotatably driven thereon by an articulation gear assembly. The torque sensor 744d provides an articulation force feedback signal to the control circuit 710. The articulation force feedback signal represents the articulation force applied to the end effector 702. A sensor 738 , such as an articulation encoder, may provide the articulation position of the end effector 702 to the control circuit 710 .

[0259] On the other hand, the joint motion function of the robotic surgical system 700 may include two joint motion components or connecting rods 742a, 742b. These joint motion components 742a, 742b are driven by a separate disk on the robot interface (rack) driven by two motors 708d, 708e. When a separate firing motor 704a is provided, each of the joint motion connecting rods 742a, 742b can be driven in an antagonistic manner relative to another connecting rod so as to provide resistance to the head to maintain motion and load when the head is not moving, and to provide joint motion when the head is articulated. When the head rotates, the joint motion components 742a, 742b are attached to the head with a fixed radius. Therefore, when the head rotates, the mechanical advantage of the push-pull connecting rod changes. This change in mechanical advantage can be more obvious for other joint motion connecting rod drive systems.

[0260] In one aspect, one or more motors 704a-704e may include a brushed DC motor with a gearbox and a mechanical link with a firing member, a closing member, or an articulation member. Another example includes an electric motor 704a-704e that operates a movable mechanical element such as a displacement member, an articulation link, a closing tube, and an axis. External influences are unmeasured, unpredictable effects of things such as tissue, surrounding body, and friction on a physical system. Such external influences may be referred to as drag, which acts relative to one of the electric motors 704a-704e. External influences such as drag may cause the operation of a physical system to deviate from the desired operation of the physical system.

[0261] In one aspect, the position sensor 734 can be implemented as an absolute positioning system. In one aspect, the position sensor 734 can include a magnetic rotation absolute positioning system implemented as an AS5055EQFT monolithic magnetic rotation position sensor, which can be purchased from Austria Microsystems, AG. The position sensor 734 can interface with the control circuit 710 to provide an absolute positioning system. The position can include a Hall effect element located above the magnet and coupled to a CORDIC processor, which is also known as a bitwise method and Volder algorithm, which is provided to implement a simple and efficient algorithm for calculating hyperbolic and trigonometric functions, which only require addition operations, subtraction operations, digit shift operations, and table lookup operations.

[0262] In one aspect, the control circuit 710 may communicate with one or more sensors 738. The sensor 738 may be positioned on the end effector 702 and adapted to operate with the robotic surgical instrument 700 to measure various derived parameters, such as gap distance versus time, tissue compression versus time, and anvil strain versus time. The sensor 738 may include a magnetic sensor, a magnetic field sensor, a strain gauge, a load sensor, a pressure sensor, a force sensor, a torque sensor, an inductive sensor such as an eddy current sensor, a resistive sensor, a capacitive sensor, an optical sensor, and / or any other suitable sensor for measuring one or more parameters of the end effector 702. The sensor 738 may include one or more sensors. The sensor 738 may be located on the nail magazine 718 platform to determine tissue position using segmented electrodes. The torque sensors 744a-744e may be configured to sense forces such as firing forces, closing forces, and / or joint motion forces, etc. Thus, the control circuit 710 can sense (1) the closing load experienced by the distal closure tube and its position, (2) the firing member at the rack and its position, (3) the portion of the staple cartridge 718 with tissue above it, and (4) the loads and positions on the two articulation rods.

[0263] In one aspect, the one or more sensors 738 may include a strain gauge, such as a micro strain gauge, configured to measure the magnitude of the strain in the anvil 716 during the clamping condition. The strain gauge provides an electrical signal whose amplitude varies with the magnitude of the strain. The sensor 738 may include a pressure sensor configured to detect the pressure generated by the compressed tissue present between the anvil 716 and the staple cartridge 718. The sensor 738 may be configured to detect the impedance of the tissue section between the anvil 716 and the staple cartridge 718, which indicates the thickness and / or completeness of the tissue located therebetween.

[0264] In one aspect, the sensor 738 may be implemented as one or more limit switches, electromechanical devices, solid-state switches, Hall effect devices, magnetoresistive (MR) devices, giant magnetoresistive (GMR) devices, magnetometers, and the like. In other implementations, the sensor 738 may be implemented as a solid-state switch that operates under the influence of light, such as an optical sensor, an IR sensor, an ultraviolet sensor, and the like. Likewise, the switch may be a solid-state device, such as a transistor (e.g., a FET, a junction FET, a MOSFET, a bipolar transistor, and the like). In other implementations, the sensor 738 may include a non-electrical conductor switch, an ultrasonic switch, an accelerometer, an inertial sensor, and the like.

[0265] In one aspect, the sensor 738 may be configured to measure the force applied to the anvil 716 by the closure drive system. For example, one or more sensors 738 may be located at the interaction point between the closure tube and the anvil 716 to detect the closing force applied to the anvil 716 by the closure tube. The force applied to the anvil 716 may represent the tissue compression experienced by the tissue segment captured between the anvil 716 and the staple cartridge 718. The one or more sensors 738 may be positioned at various interaction points along the closure drive system to detect the closing force applied to the anvil 716 by the closure drive system. The one or more sensors 738 may be sampled in real time by the processor of the control circuit 710 during the clamping operation. The control circuit 710 receives real-time sample measurements to provide and analyze time-based information and evaluate the closing force applied to the anvil 716 in real time.

[0266] In one aspect, the current sensor 736 can be used to measure the current consumed by each of the motors 704a-704e. The force required to propel any of the movable mechanical elements (such as the I-beam 714) corresponds to the current consumed by one of the motors 704a-704e. The force is converted into a digital signal and provided to the processor 710. The control circuit 710 can be configured to simulate the response of the actual system of the instrument in the software of the controller. The displacement member can be actuated to move the I-beam 714 in the end effector 702 at a target speed or close to the target speed. The robotic surgical system 700 may include a feedback controller, which may be one of any feedback controllers, including but not limited to, for example, PID, state feedback, linear square (LQR) and / or adaptive controllers. The robotic surgical instrument 700 may include a power source to, for example, convert a signal from the feedback controller into a physical input, such as a housing voltage, a PWM voltage, a frequency modulated voltage, a current, a torque and / or a force. Additional details are disclosed in U.S. patent application Ser. No. 15 / 636,829, filed Jun. 29, 2017, entitled “CLOSED LOOP VELOCITY CONTROL TECHNIQUES FOR ROBOTIC SURGICAL INSTRUMENT,” which is incorporated herein by reference in its entirety.

[0267] Fig.18 A block diagram of a surgical instrument 750 programmed to control distal translation of a displacement member is shown according to one aspect of the present disclosure. In one aspect, the surgical instrument 750 is programmed to control distal translation of a displacement member such as an I-beam 764. The surgical instrument 750 includes an end effector 752, which may include an anvil 766, an I-beam 764 (including a sharp cutting edge), and a removable staple cartridge 768.

[0268] The position, movement, displacement and / or translation of a linear displacement member such as the I-beam 764 can be measured by an absolute positioning system, a sensor arrangement and a position sensor 784. Because the I-beam 764 is coupled to a longitudinally movable drive member, the position of the I-beam 764 can be determined by measuring the position of the longitudinally movable drive member using the position sensor 784. Therefore, in the following description, the position, displacement and / or translation of the I-beam 764 can be achieved by the position sensor 784 as described herein. The control circuit 760 can be programmed to control the translation of a displacement member such as the I-beam 764. In some examples, the control circuit 760 may include one or more microcontrollers, microprocessors or other suitable processors for executing instructions that cause one or more processors to control the displacement member (e.g., the I-beam 764) in the manner described. In one aspect, the timer / counter 781 provides an output signal, such as an elapsed time or a digital count, to the control circuit 760 to correlate the position of the I-beam 764 as determined by the position sensor 784 with the output of the timer / counter 781 so that the control circuit 760 can determine the position of the I-beam 764 at a particular time (t) relative to the starting position. The timer / counter 781 can be configured to measure elapsed time, count external events, or time external events.

[0269] The control circuit 760 may generate a motor set point signal 772. The motor set point signal 772 may be provided to the motor controller 758. The motor controller 758 may include one or more circuits configured to provide a motor drive signal 774 to the motor 754 to drive the motor 754, as described herein. In some examples, the motor 754 may be a brushed DC electric motor. For example, the speed of the motor 754 may be proportional to the voltage of the motor drive signal 774. In some examples, the motor 754 may be a brushless DC electric motor, and the motor drive signal 774 may include a PWM signal provided to one or more stator windings of the motor 754. Moreover, in some examples, the motor controller 758 may be omitted, and the control circuit 760 may directly generate the motor drive signal 774.

[0270] The motor 754 may receive power from the energy source 762. The energy source 762 may be or include a battery, a supercapacitor, or any other suitable energy source. The motor 754 may be mechanically coupled to the I-beam 764 via the transmission 756. The transmission 756 may include one or more gears or other linkage components for coupling the motor 754 to the I-beam 764. The position sensor 784 may sense the position of the I-beam 764. The position sensor 784 may be or include any type of sensor capable of generating position data indicating the position of the I-beam 764. In some examples, the position sensor 784 may include an encoder configured to provide a series of pulses to the control circuit 760 when the I-beam 764 is translated toward the distal side and toward the proximal side. The control circuit 760 may track the pulses to determine the position of the I-beam 764. Other suitable position sensors may be used, including, for example, proximity sensors. Other types of position sensors may provide other signals indicating the movement of the I-beam 764. Moreover, in some examples, the position sensor 784 may be omitted. Where motor 754 is a stepper motor, control circuit 760 can track the position of I-beam 764 by aggregating the number and direction of steps motor 754 has been instructed to perform. Position sensor 784 may be located in end effector 752 or at any other portion of the instrument.

[0271] The control circuit 760 may communicate with one or more sensors 788. The sensor 788 may be positioned on the end effector 752 and adapted to operate with the surgical instrument 750 to measure various derived parameters, such as gap distance versus time, tissue compression versus time, and anvil strain versus time. The sensor 788 may include a magnetic sensor, a magnetic field sensor, a strain gauge, a pressure sensor, a force sensor, an inductive sensor such as an eddy current sensor, a resistive sensor, a capacitive sensor, an optical sensor, and / or any other suitable sensor for measuring one or more parameters of the end effector 752. The sensor 788 may include one or more sensors.

[0272] The one or more sensors 788 may include a strain gauge, such as a micro strain gauge, configured to measure the magnitude of the strain in the anvil 766 during the clamping condition. The strain gauge provides an electrical signal whose amplitude varies with the magnitude of the strain. The sensor 788 may include a pressure sensor configured to detect the pressure generated by the compressed tissue present between the anvil 766 and the staple cartridge 768. The sensor 788 may be configured to detect the impedance of the tissue section between the anvil 766 and the staple cartridge 768, which indicates the thickness and / or completeness of the tissue located therebetween.

[0273] The sensor 788 may be configured to measure the force applied to the anvil 766 by the closure drive system. For example, one or more sensors 788 may be located at the interaction point between the closure tube and the anvil 766 to detect the closing force applied to the anvil 766 by the closure tube. The force applied to the anvil 766 may represent the tissue compression experienced by the tissue segment captured between the anvil 766 and the staple cartridge 768. The one or more sensors 788 may be positioned at various interaction points along the closure drive system to detect the closing force applied to the anvil 766 by the closure drive system. The one or more sensors 788 may be sampled in real time by the processor of the control circuit 760 during the clamping operation. The control circuit 760 receives real-time sample measurements to provide and analyze time-based information and evaluate the closing force applied to the anvil 766 in real time.

[0274] A current sensor 786 may be employed to measure the current consumed by the motor 754. The force required to advance the I-beam 764 corresponds to the current consumed by the motor 754. The force is converted into a digital signal and provided to the processor 760.

[0275] The control circuit 760 may be configured to simulate the response of the actual system of the instrument in the software of the controller. The displacement member may be actuated to move the I-beam 764 in the end effector 752 at a target speed or close to the target speed. The surgical instrument 750 may include a feedback controller, which may be one of any feedback controllers, including but not limited to, for example, PID, state feedback, LQR, and / or adaptive controllers. The surgical instrument 750 may include a power source to, for example, convert a signal from the feedback controller into a physical input, such as a housing voltage, a PWM voltage, a frequency modulated voltage, a current, a torque, and / or a force.

[0276] The actual drive system of the surgical instrument 750 is configured to drive the displacement member, cutting member or I-beam 764 through a brushed DC motor with a gear box and a mechanical link to the articulation and / or knife system. Another example is an electric motor 754 that operates the displacement member and articulation drive of, for example, an interchangeable shaft assembly. External influences are the unmeasured, unpredictable effects of things such as tissue, surrounding body, and friction on the physical system. Such external influences may be referred to as drag acting relative to the electric motor 754. External influences such as drag may cause the operation of the physical system to deviate from the desired operation of the physical system.

[0277] Various exemplary aspects relate to a surgical instrument 750 that includes an end effector 752 having a motor-driven surgical stapling and cutting tool. For example, a motor 754 can drive a displacement member distally and proximally along a longitudinal axis of the end effector 752. The end effector 752 can include a pivotable anvil 766, and when configured for use, a staple cartridge 768 is positioned relative to the anvil 766. A clinician can hold tissue between the anvil 766 and the staple cartridge 768, as described herein. When the instrument 750 is ready to be used, the clinician can provide a firing signal, for example, by depressing a trigger of the instrument 750. In response to the firing signal, the motor 754 can drive the displacement member distally from a proximal stroke start position to a stroke end position distal to the stroke start position along the longitudinal axis of the end effector 752. When the displacement member translates distally, an I-beam 764 having a cutting element positioned at a distal end can cut tissue between the staple cartridge 768 and the anvil 766.

[0278] In various examples, the surgical instrument 750 may include a control circuit 760 that is programmed to control the distal translation of a displacement member, such as an I-beam 764, based on one or more tissue conditions, for example. The control circuit 760 may be programmed to sense tissue conditions, such as thickness, directly or indirectly, as described herein. The control circuit 760 may be programmed to select a firing control program based on the tissue conditions. The firing control program may describe the distal movement of the displacement member. Different firing control programs may be selected to better handle different tissue conditions. For example, when thicker tissue is present, the control circuit 760 may be programmed to translate the displacement member at a lower speed and / or at a lower power. When thinner tissue is present, the control circuit 760 may be programmed to translate the displacement member at a higher speed and / or at a higher power.

[0279] In some examples, the control circuit 760 may initially operate the motor 754 in an open-loop configuration for a first open-loop portion of the displacement member's travel. Based on the response of the instrument 750 during the open-loop portion of the travel, the control circuit 760 may select a firing control program. The response of the instrument may include the distance the displacement member is translated during the open-loop portion, the time elapsed during the open-loop portion, the energy provided to the motor 754 during the open-loop portion, the sum of the pulse widths of the motor drive signal, etc. After the open-loop portion, the control circuit 760 may implement the selected firing control program for a second portion of the displacement member's travel. For example, during the closed-loop portion of the travel, the control circuit 760 may modulate the motor 754 based on the translation data describing the position of the displacement member in a closed-loop manner to cause the displacement member to translate at a constant speed. Additional details are disclosed in U.S. patent application serial number 15 / 720,852, filed on September 29, 2017, entitled “SYSTEM AND METHODS FOR CONTROLLING A DISPLAY OF A SURGICAL INSTRUMENT,” which is incorporated herein by reference in its entirety.

[0280] Fig.19 is a schematic diagram of a surgical instrument 790 configured to control various functions according to one aspect of the present disclosure. In one aspect, the surgical instrument 790 is programmed to control distal translation of a displacement member such as an I-beam 764. The surgical instrument 790 includes an end effector 792, which may include an anvil 766, an I-beam 764, and a removable staple cartridge 768 that is interchangeable with an RF cartridge 796 (shown in phantom).

[0281] In one aspect, the sensor 788 may be implemented as a limit switch, an electromechanical device, a solid-state switch, a Hall effect device, an MR device, a GMR device, a magnetometer, and the like. In other implementations, the sensor 638 may be implemented as a solid-state switch that operates under the influence of light, such as an optical sensor, an IR sensor, an ultraviolet sensor, and the like. Likewise, the switch may be a solid-state device, such as a transistor (e.g., a FET, a junction FET, a MOSFET, a bipolar transistor, and the like). In other implementations, the sensor 788 may include a non-electric conductor switch, an ultrasonic switch, an accelerometer, an inertial sensor, and the like.

[0282] In one aspect, the position sensor 784 can be implemented as an absolute positioning system including a magnetic rotary absolute positioning system implemented as an AS5055EQFT monolithic magnetic rotary position sensor (available from Austria Microsystems, AG). The position sensor 784 can interface with the control circuit 760 to provide an absolute positioning system. The position can include a Hall effect element located above a magnet and coupled to a CORDIC processor, also known as a bitwise method and Volder algorithm, which is provided to implement a simple and efficient algorithm for calculating hyperbolic and trigonometric functions that only require addition operations, subtraction operations, bit shift operations, and table lookup operations.

[0283] In one aspect, the I-beam 764 can be implemented as a knife member including a knife body that operably supports the tissue cutting blade thereon and can also include an anvil engagement tab or feature and a channel engagement feature or foot. In one aspect, the staple cartridge 768 can be implemented as a standard (mechanical) surgical fastener cartridge. In one aspect, the RF cartridge 796 can be implemented as an RF cartridge. These and other sensor arrangements are described in U.S. patent application serial number 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUESFOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety.

[0284] The position, movement, displacement and / or translation of a linear displacement member such as the I-beam 764 can be measured by an absolute positioning system, a sensor arrangement, and a position sensor, represented as position sensor 784. Because the I-beam 764 is coupled to a longitudinally movable drive member, the position of the I-beam 764 can be determined by measuring the position of the longitudinally movable drive member using the position sensor 784. Therefore, in the following description, the position, displacement and / or translation of the I-beam 764 can be achieved by the position sensor 784 as described herein. The control circuit 760 can be programmed to control the translation of a displacement member such as the I-beam 764, as described herein. In some examples, the control circuit 760 may include one or more microcontrollers, microprocessors, or other suitable processors for executing instructions that cause one or more processors to control the displacement member (e.g., the I-beam 764) in the manner described. In one aspect, the timer / counter 781 provides an output signal, such as an elapsed time or a digital count, to the control circuit 760 to correlate the position of the I-beam 764 as determined by the position sensor 784 with the output of the timer / counter 781 so that the control circuit 760 can determine the position of the I-beam 764 at a particular time (t) relative to the starting position. The timer / counter 781 can be configured to measure elapsed time, count external events, or time external events.

[0285] The control circuit 760 may generate a motor set point signal 772. The motor set point signal 772 may be provided to the motor controller 758. The motor controller 758 may include one or more circuits configured to provide a motor drive signal 774 to the motor 754 to drive the motor 754, as described herein. In some examples, the motor 754 may be a brushed DC electric motor. For example, the speed of the motor 754 may be proportional to the voltage of the motor drive signal 774. In some examples, the motor 754 may be a brushless DC electric motor, and the motor drive signal 774 may include a PWM signal provided to one or more stator windings of the motor 754. Moreover, in some examples, the motor controller 758 may be omitted, and the control circuit 760 may directly generate the motor drive signal 774.

[0286] The motor 754 may receive power from the energy source 762. The energy source 762 may be or include a battery, a supercapacitor, or any other suitable energy source. The motor 754 may be mechanically coupled to the I-beam 764 via the transmission 756. The transmission 756 may include one or more gears or other linkage components for coupling the motor 754 to the I-beam 764. The position sensor 784 may sense the position of the I-beam 764. The position sensor 784 may be or include any type of sensor capable of generating position data indicating the position of the I-beam 764. In some examples, the position sensor 784 may include an encoder configured to provide a series of pulses to the control circuit 760 when the I-beam 764 is translated toward the distal side and toward the proximal side. The control circuit 760 may track the pulses to determine the position of the I-beam 764. Other suitable position sensors may be used, including, for example, proximity sensors. Other types of position sensors may provide other signals indicating the movement of the I-beam 764. Moreover, in some examples, the position sensor 784 may be omitted. Where motor 754 is a stepper motor, control circuit 760 can track the position of I-beam 764 by aggregating the number and direction of steps the motor has been instructed to perform. Position sensor 784 may be located in end effector 792 or at any other portion of the instrument.

[0287] The control circuit 760 may communicate with one or more sensors 788. The sensor 788 may be positioned on the end effector 792 and adapted to operate with the surgical instrument 790 to measure various derived parameters, such as gap distance versus time, tissue compression versus time, and anvil strain versus time. The sensor 788 may include a magnetic sensor, a magnetic field sensor, a strain gauge, a pressure sensor, a force sensor, an inductive sensor such as an eddy current sensor, a resistive sensor, a capacitive sensor, an optical sensor, and / or any other suitable sensor for measuring one or more parameters of the end effector 792. The sensor 788 may include one or more sensors.

[0288] The one or more sensors 788 may include a strain gauge, such as a micro strain gauge, configured to measure the magnitude of the strain in the anvil 766 during the clamping condition. The strain gauge provides an electrical signal whose amplitude varies with the magnitude of the strain. The sensor 788 may include a pressure sensor configured to detect the pressure generated by the compressed tissue present between the anvil 766 and the staple cartridge 768. The sensor 788 may be configured to detect the impedance of the tissue section between the anvil 766 and the staple cartridge 768, which indicates the thickness and / or completeness of the tissue located therebetween.

[0289] The sensor 788 may be configured to measure the force applied to the anvil 766 by the closure drive system. For example, one or more sensors 788 may be located at the interaction point between the closure tube and the anvil 766 to detect the closing force applied to the anvil 766 by the closure tube. The force applied to the anvil 766 may represent the tissue compression experienced by the tissue segment captured between the anvil 766 and the staple cartridge 768. The one or more sensors 788 may be positioned at various interaction points along the closure drive system to detect the closing force applied to the anvil 766 by the closure drive system. The one or more sensors 788 may be sampled in real time by the processor portion of the control circuit 760 during the clamping operation. The control circuit 760 receives real-time sample measurements to provide and analyze time-based information and evaluate the closing force applied to the anvil 766 in real time.

[0290] A current sensor 786 may be employed to measure the current consumed by the motor 754. The force required to advance the I-beam 764 corresponds to the current consumed by the motor 754. The force is converted into a digital signal and provided to the processor 760.

[0291] When the RF cartridge 796 is loaded in the end effector 792 in place of the staple cartridge 768, the RF energy source 794 is coupled to the end effector 792 and applied to the RF cartridge 796. The control circuit 760 controls the delivery of RF energy to the RF cartridge 796.

[0292] Additional details are disclosed in U.S. patent application serial number 15 / 636,096, filed on June 28, 2017, entitled “SURGICAL SYSTEM COUPLABLE WITH STAPLE CARTRIDGE AND RADIO FREQUENCY CARTRIDGE, AND METHOD OF USING SAME,” which is incorporated herein by reference in its entirety.

[0293] Generator Hardware

[0294] Fig. 208 is a simplified block diagram of a generator 800 configured to provide, among other benefits, inductor-free tuning. Additional details of the generator 800 are described in U.S. Pat. No. 9,060,775, entitled “SURGICAL GENERATOR FOR ULTRASONIC AND ELECTROSURGICAL DEVICES,” issued on June 23, 2015, which is incorporated herein by reference in its entirety. The generator 800 may include a patient-isolated stage 802 that communicates with a non-isolated stage 804 via a power transformer 806. A secondary winding 808 of the power transformer 806 is contained within the isolation stage 802 and may include a tapped configuration (e.g., a center-tapped or non-center-tapped configuration) to define drive signal outputs 810a, 810b, 810c for delivering drive signals to different surgical instruments, such as, for example, ultrasonic surgical instruments, RF electrosurgical instruments, and multi-function surgical instruments including ultrasonic energy modes and RF energy modes that can be delivered separately or simultaneously. Specifically, the drive signal outputs 810a, 810c can output ultrasonic drive signals (e.g., 420V root mean square (RMS) drive signals) to an ultrasonic surgical instrument, and the drive signal outputs 810b, 810c can output RF electrosurgical drive signals (e.g., 100V RMS drive signals) to an RF electrosurgical instrument, where the drive signal output 810b corresponds to the center tap of the power transformer 806.

[0295] In some forms, the ultrasonic drive signal and the electrosurgical drive signal can be provided to different surgical instruments and / or a single surgical instrument such as a multifunctional surgical instrument having the ability to deliver both ultrasonic energy and electrosurgical energy to tissue at the same time. It should be understood that the electrosurgical signal provided to a dedicated electrosurgical instrument and / or provided to a combined multifunctional ultrasonic / electrosurgical instrument can be a therapeutic level signal or a sub-therapeutic level signal, wherein, for example, a sub-therapeutic signal can be used to monitor tissue or instrument conditions and provide feedback to the generator. For example, an ultrasonic signal and an RF signal can be delivered separately or simultaneously from a generator having a single output port to provide a desired output signal to the surgical instrument, as will be discussed in more detail below. Therefore, the generator can combine ultrasonic energy and electrosurgical RF energy and deliver the combined energy to the multifunctional ultrasonic / electrosurgical instrument. A bipolar electrode can be placed on one or both jaws of the end effector. In addition to electrosurgical RF energy, one jaw can be driven simultaneously by ultrasonic energy. Ultrasonic energy can be used to dissect tissue, while electrosurgical RF energy can be used to perform vessel sealing.

[0296] The non-isolated stage 804 may include a power amplifier 812 having an output connected to a primary winding 814 of a power transformer 806. In certain forms, the power amplifier 812 may include a push-pull amplifier. For example, the non-isolated stage 804 may also include a logic device 816 for providing a digital output to a digital / analog converter (DAC) circuit 818, wherein the DAC circuit in turn provides a corresponding analog signal to an input of the power amplifier 812. In certain forms, the logic device 816 may include, for example, a programmable gate array (PGA), an FPGA, a programmable logic device (PLD), in addition to other logic circuits. Therefore, since the input of the power amplifier 812 is controlled via the DAC circuit 818, the logic device 816 may control any of a plurality of parameters (e.g., frequency, waveform shape, waveform amplitude) of the drive signal appearing at the drive signal outputs 810a, 810b, 810c. In certain forms and as discussed below, the logic device 816 in conjunction with a processor (e.g., a DSP discussed below) may implement a plurality of DSP-based algorithms and / or other control algorithms to control parameters of the drive signal output by the generator 800.

[0297] Power may be provided to the power rail of the power amplifier 812 by a switch-mode regulator 820 (e.g., a power converter). In certain forms, the switch-mode regulator 820 may, for example, include an adjustable buck regulator. For example, the non-isolated stage 804 may also include a first processor 822, which in one form may include a DSP processor, such as an Analog Devices ADSP-21469SHARC DSP available from Analog Devices (Norwood, MA), although any suitable processor may be employed in various forms. In certain forms, the DSP processor 822 may control operation of the switch-mode regulator 820 in response to voltage feedback data received by the DSP processor 822 from the power amplifier 812 via an ADC circuit 824. In one form, for example, the DSP processor 822 may receive as input a waveform envelope of a signal (e.g., an RF signal) amplified by the power amplifier 812 via the ADC circuit 824. The DSP processor 822 can then control the switch mode regulator 820 (e.g., via a PWM output) so that the rail voltage provided to the power amplifier 812 tracks the waveform envelope of the amplified signal. By dynamically modulating the rail voltage of the power amplifier 812 based on the waveform envelope, the efficiency of the power amplifier 812 can be significantly improved relative to a fixed rail voltage amplifier solution.

[0298] In certain forms, the logic device 816 in conjunction with the DSP processor 822 may implement a digital synthesis circuit such as a direct digital synthesizer control scheme to control the waveform shape, frequency and / or amplitude of the drive signal output by the generator 800. In one form, for example, the logic device 816 may implement a DDS control algorithm by recalling waveform samples stored in a dynamically updated lookup table (LUT) (such as a RAM LUT), which may be embedded in an FPGA. The control algorithm is particularly useful for ultrasound applications where an ultrasonic transducer such as an ultrasonic transducer may be driven by a pure sinusoidal current at its resonant frequency. Because other frequencies may excite parasitic resonances, minimizing or reducing the overall distortion of the motional branch current may correspondingly minimize or reduce adverse resonance effects. Because the waveform shape of the drive signal output by the generator 800 is affected by various sources of distortion present in the output drive circuit (e.g., power transformer 806, power amplifier 812), voltage and current feedback data based on the drive signal can be input into an algorithm (such as an error control algorithm implemented by the DSP processor 822) that compensates for distortion by appropriately pre-distorting or modifying the waveform samples stored in the LUT in a dynamic and ongoing manner (e.g., in real time). In one form, the amount or degree of pre-distortion applied to the LUT samples can be based on the error between the calculated motional branch current and the desired current waveform shape, which can be determined on a sample-by-sample basis. In this way, the pre-distorted LUT samples, when processed by the drive circuit, can cause the motional branch drive signal to have a desired waveform shape (e.g., a sinusoidal shape) to optimally drive the ultrasonic transducer. Therefore, in such forms, when the distortion effects are taken into account, the LUT waveform samples will not exhibit the desired waveform shape of the drive signal, but rather will exhibit the waveform shape required to ultimately produce the desired waveform shape of the motional branch drive signal.

[0299] The non-isolated stage 804 may also include a first ADC circuit 826 and a second ADC circuit 828 coupled to the output of the power transformer 806 via respective isolation transformers 830, 832 for respectively sampling the voltage and current of the drive signal output by the generator 800. In certain forms, the ADC circuits 826, 828 may be configured to sample at a high speed (e.g., 80 megasamples per second (MSPS)) to enable oversampling of the drive signal. In one form, for example, the sampling speed of the ADC circuits 826, 828 may enable oversampling of the drive signal by approximately 200× (depending on the frequency). In certain forms, the sampling operation of the ADC circuits 826, 828 may be performed by a single ADC circuit receiving input voltage and current signals via a bidirectional multiplexer. By using high speed sampling in forms of the generator 800, among other things, calculation of complex currents flowing through the motional branch (which in certain forms may be used to implement the above-described DDS-based waveform shape control), accurate digital filtering of the sampled signals, and calculation of actual power consumption with high accuracy may be achieved. The voltage and current feedback data output by the ADC circuits 826, 828 may be received and processed by the logic device 816 (e.g., first-in, first-out (FIFO) buffer, multiplexer), and stored in a data memory for subsequent retrieval, for example, by the DSP processor 822. As described above, the voltage and current feedback data may be used as input to an algorithm for pre-distorting or modifying the LUT waveform samples in a dynamic, ongoing manner. In some forms, when the voltage and current feedback data pairs are acquired, it may be desirable to index each stored voltage and current feedback data pair based on or otherwise associated with the corresponding LUT sample output by the logic device 816. Synchronizing the LUT samples and the voltage and current feedback data in this manner facilitates accurate timing and stability of the pre-distortion algorithm.

[0300] In certain forms, the voltage and current feedback data may be used to control the frequency and / or amplitude (e.g., current amplitude) of the drive signal. In one form, for example, the voltage and current feedback data may be used to determine the impedance phase. The frequency of the drive signal may then be controlled to minimize or reduce the difference between the determined impedance phase and the impedance phase set point (e.g., 0°), thereby minimizing or reducing the effects of harmonic distortion and correspondingly improving the impedance phase measurement accuracy. The determination of the phase impedance and frequency control signal may be implemented in the DSP processor 822, for example, where the frequency control signal is provided as an input to a DDS control algorithm implemented by the logic device 816.

[0301] In another form, for example, current feedback data may be monitored to maintain the current amplitude of the drive signal at a current amplitude set point. The current amplitude set point may be specified directly or determined indirectly based on a specific voltage amplitude and power set point. In certain forms, control of the current amplitude may be achieved by a control algorithm in the DSP processor 822, such as, for example, a proportional integral derivative (PID) control algorithm. Variables controlled by the control algorithm to appropriately control the current amplitude of the drive signal may include, for example, the scale of the LUT waveform samples stored in the logic device 816 and / or the full-scale output voltage of the DAC circuit 818 (which provides input to the power amplifier 812) via the DAC circuit 834.

[0302] The non-isolated stage 804 may also include a second processor 836, which is used to provide user interface (UI) functionality, among other things. In one form, the UI processor 836 may include, for example, an Atmel AT91SAM9263 processor with an ARM 926EJ-S core purchased from Atmel Corporation, San Jose, California. Examples of UI functionality supported by the UI processor 836 may include audible and visual user feedback, communication with peripheral devices (e.g., via a USB interface), communication with a foot switch, communication with an input device (e.g., a touch screen display), and communication with an output device (e.g., a speaker). The UI processor 836 may communicate with the DSP processor 822 and the logic device 816 (e.g., via an SPI bus). Although the UI processor 836 may primarily support UI functionality, in some forms, it may also cooperate with the DSP processor 822 to achieve risk mitigation. For example, the UI processor 836 may be programmed to monitor various aspects of user input and / or other input (e.g., touch screen input, foot switch input, temperature sensor input) and disable the drive output of the generator 800 when an error condition is detected.

[0303] In certain forms, both the DSP processor 822 and the UI processor 836 may, for example, determine and monitor the operating state of the generator 800. For the DSP processor 822, the operating state of the generator 800 may, for example, indicate which control and / or diagnostic processes are implemented by the DSP processor 822. For the UI processor 836, the operating state of the generator 800 may, for example, indicate which elements of the UI (e.g., display screen, sound) may be presented to the user. The respective DSP processor 822 and UI processor 836 may independently maintain the current operating state of the generator 800 and identify and evaluate possible transitions of the current operating state. The DSP processor 822 may serve as the principal in this relationship and determine when a transition between operating states will occur. The UI processor 836 may note valid transitions between operating states and may verify whether a particular transition is appropriate. For example, when the DSP processor 822 instructs the UI processor 836 to transition to a particular state, the UI processor 836 may verify that the requested transition is valid. If it is determined by the UI processor 836 that the requested transition between states is invalid, the UI processor 836 may cause the generator 800 to enter a fault mode.

[0304] The non-isolated stage 804 may also include a controller 838 for monitoring input devices (e.g., capacitive touch sensors, capacitive touch screens for turning the generator 800 on and off). In certain forms, the controller 838 may include at least one processor and / or other control device in communication with the UI processor 836. In one form, for example, the controller 838 may include a processor (e.g., a Meg168 8-bit controller available from Atmel) configured to monitor user input provided via one or more capacitive touch sensors. In one form, the controller 838 may include a touch screen controller (e.g., a QT5480 touch screen controller available from Atmel) to control and manage the collection of touch data from the capacitive touch screen.

[0305] In certain forms, when the generator 800 is in the "power off" state, the controller 838 may continue to receive operating power (e.g., via a line from a power source of the generator 800, such as the power source 854 described below). In this manner, the controller 838 may continue to monitor an input device (e.g., a capacitive touch sensor located on a front panel of the generator 800) for turning the generator 800 on and off. When the generator 800 is in the power off state, if activation of the user "on / off" input device is detected, the controller 838 may wake up the power source (e.g., enable operation of one or more DC / DC voltage converters 856 of the power source 854). The controller 838 may thus begin a sequence for transitioning the generator 800 to the "power on" state. Conversely, when the generator 800 is in the power on state, if activation of the "on / off" input device is detected, the controller 838 may begin a sequence for transitioning the generator 800 to the power off state. In certain forms, for example, the controller 838 may report activation of an "on / off" input device to the UI processor 836, which in turn implements the necessary process sequence for transitioning the generator 800 to the power-off state. In such forms, the controller 838 may not have an independent capability for removing power from the generator 800 after the power-on state is established.

[0306] In some forms, the controller 838 may cause the generator 800 to provide audible or other sensory feedback to alert the user that a power-on or power-off sequence has begun. Such an alert may be provided when a power-on or power-off sequence begins and before other processes associated with the sequence begin.

[0307] In certain forms, the isolation stage 802 may include an instrument interface circuit 840 to provide a communication interface, for example, between control circuitry of a surgical instrument (e.g., control circuitry including a handpiece switch) and components of the non-isolated stage 804 (e.g., logic device 816, DSP processor 822, and / or UI processor 836). The instrument interface circuit 840 may exchange information with components of the non-isolated stage 804 via a communication connection (e.g., an IR-based communication connection) that maintains a suitable degree of electrical isolation between the isolation stage 802 and the non-isolated stage 804. For example, the instrument interface circuit 840 may be powered using a low dropout regulator powered by an isolation transformer, which is driven from the non-isolated stage 804.

[0308] In one form, the instrument interface circuit 840 may include a logic circuit 842 (e.g., a logic circuit, a programmable logic circuit, a PGA, an FPGA, a PLD) in communication with a signal conditioning circuit 844. The signal conditioning circuit 844 may be configured to receive a periodic signal (e.g., a 2kHz square wave) from the logic circuit 842 to generate a bipolar interrogation signal having the same frequency. For example, the interrogation signal may be generated using a bipolar current source fed by a differential amplifier. The interrogation signal may be transmitted to a surgical instrument control circuit (e.g., by using a conductor pair in a cable connecting the generator 800 to the surgical instrument) and monitored to determine the state or configuration of the control circuit. The control circuit may include a plurality of switches, resistors, and / or diodes to modify one or more characteristics of the interrogation signal (e.g., amplitude, correction) so that the state or configuration of the control circuit can be uniquely discerned based on the one or more characteristics. In one form, for example, the signal conditioning circuit 844 may include an ADC circuit for generating a sample of a voltage signal appearing across the input of the control circuit as a result of the interrogation signal passing through the control circuit. Subsequently, logic circuit 842 (or a component of non-isolated stage 804) may determine a state or configuration of the control circuit based on the ADC circuit samples.

[0309] In one form, the instrument interface circuit 840 may include a first data circuit interface 846 to enable information exchange between the logic circuit 842 (or other elements of the instrument interface circuit 840) and the first data circuit disposed in the surgical instrument or otherwise associated with the surgical instrument. In some forms, for example, the first data circuit may be disposed in a cable integrally attached to the surgical instrument handpiece, or in an adapter for interfacing a specific surgical instrument type or model with the generator 800. The first data circuit may be implemented in any suitable manner and may communicate with the generator according to any suitable protocol including, for example, those described herein with respect to the first data circuit. In some forms, the first data circuit may include a non-volatile storage device, such as an EEPROM device. In some forms, the first data circuit interface 846 may be implemented separately from the logic circuit 842 and include suitable circuits (e.g., discrete logic devices, processors) to enable communication between the logic circuit 842 and the first data circuit. In other forms, the first data circuit interface 846 may be integral with the logic circuit 842.

[0310] In some forms, the first data circuit may store information related to the associated specific surgical instrument. Such information may include, for example, a model number, a serial number, a number of operations in which the surgical instrument has been used, and / or any other type of information. The information may be read by the instrument interface circuit 840 (e.g., by the logic circuit 842), transmitted to the components of the non-isolated stage 804 (e.g., transmitted to the logic device 816, the DSP processor 822 and / or the UI processor 836) for presentation to the user via an output device and / or for controlling the function or operation of the generator 800. In addition, any type of information may be transmitted to the first data circuit via the first data circuit interface 846 (e.g., using the logic circuit 842) for storage therein. Such information may include, for example, an updated number of operations using the surgical instrument and / or the date and / or time of its use.

[0311] As previously discussed, surgical instruments may be detachable from handpieces (e.g., multifunctional surgical instruments may be detachable from handpieces) to facilitate instrument interchangeability and / or disposability. In such cases, the ability of conventional generators to identify the specific instrument configuration being used and to optimize control and diagnostic processes accordingly may be limited. However, addressing this issue by adding readable data circuits to surgical instruments is problematic from a compatibility perspective. For example, designing a surgical instrument to maintain backward compatibility with a generator that lacks the requisite data reading functionality may be impractical due to, for example, different signaling schemes, design complexity, and cost. The form of the instrument discussed herein addresses these issues by using data circuits that can be economically implemented in existing surgical instruments with minimal design changes to maintain compatibility of the surgical instrument with the current generator platform.

[0312] In addition, the form of the generator 800 can realize communication with the data circuit based on the instrument. For example, the generator 800 can be configured to communicate with the second data circuit contained in the instrument (e.g., a multifunctional surgical instrument). In some forms, the second data circuit can be implemented in a manner similar to the first data circuit described herein. The instrument interface circuit 840 may include a second data circuit interface 848 for realizing the communication. In one form, the second data circuit interface 848 may include a tri-state digital interface, but other interfaces may also be used. In some forms, the second data circuit may generally be any circuit for transmitting and / or receiving data. In one form, for example, the second data circuit may store information related to the associated specific surgical instrument. Such information may include, for example, a model number, a serial number, a plurality of operations in which the surgical instrument has been used, and / or any other type of information.

[0313] In some forms, the second data circuit may store information about the electrical and / or ultrasonic performance of the associated ultrasonic transducer, end effector, or ultrasonic drive system. For example, the first data circuit may indicate an aging frequency slope, as described herein. Additionally or alternatively, any type of information may be transmitted to the second data circuit via the second data circuit interface 848 (e.g., using the logic circuit 842) for storage therein. Such information may, for example, include an updated number of operations in which a surgical instrument is used and / or the date and / or time of its use. In certain forms, the second data circuit may transmit data collected by one or more sensors (e.g., an instrument-based temperature sensor). In certain forms, the second data circuit may receive data from the generator 800 and provide an indication to the user based on the received data (e.g., an LED indication or other visual indication).

[0314] In certain forms, the second data circuit and the second data circuit interface 848 may be configured such that communication between the logic circuit 842 and the second data circuit may be achieved without providing additional conductors for this purpose (e.g., dedicated conductors of a cable connecting the hand piece to the generator 800). In one form, for example, information may be communicated to and from the second data circuit using a single bus communication scheme implemented on an existing cable (such as one of the conductors used to transmit an interrogation signal from the signal conditioning circuit 844 to a control circuit in the hand piece). In this way, design changes or modifications to the surgical instrument that may otherwise be necessary may be minimized or reduced. Furthermore, because the different types of communications implemented on a common physical channel may be frequency-band separated, the presence of the second data circuit may be "invisible" to generators that do not have the requisite data reading functionality, thereby enabling backward compatibility of surgical instruments.

[0315] In some forms, the isolation stage 802 may include at least one blocking capacitor 850-1 connected to the drive signal output 810b to prevent DC current from flowing to the patient. For example, the signal blocking capacitor may be required to comply with medical regulations or standards. Although failures are relatively rare in single capacitor designs, such failures may have adverse consequences. In one form, a second blocking capacitor 850-2 may be provided in series with the blocking capacitor 850-1, wherein current leakage from a point between the blocking capacitors 850-1 and 850-2 is monitored, for example, by an ADC circuit 852, for sampling the voltage induced by the leakage current. These samples may be received, for example, by a logic circuit 842. Based on changes in the leakage current (as indicated by the voltage samples), the generator 800 may determine when at least one of the blocking capacitors 850-1, 850-2 fails, thereby providing benefits over single capacitor designs with a single point of failure.

[0316] In certain forms, the non-isolated stage 804 may include a power source 854 for delivering DC power at an appropriate voltage and current. The power source may include, for example, a 400W power source for outputting a system voltage of 48VDC. The power source 854 may also include one or more DC / DC voltage converters 856 for receiving the output of the power source to generate a DC output at the voltage and current required by the various components of the generator 800. As discussed above in conjunction with the controller 838, one or more of the DC / DC voltage converters 856 may receive input from the controller 838 when the controller 838 detects the activation of a user "on / off" input device to enable operation of the DC / DC voltage converter 856 or wake up the DC / DC voltage converter.

[0317] Fig.21 An example of a generator 900 is shown, which is a generator 800 ( Fig. 20 ) is a form of a surgical instrument. Generator 900 is configured to deliver multiple energy modalities to a surgical instrument. Generator 900 provides RF signals and ultrasonic signals for independently or simultaneously delivering energy to a surgical instrument. The RF signal and the ultrasonic signal may be provided separately or in combination, and may be provided simultaneously. As described above, at least one generator output may deliver multiple energy modalities (e.g., ultrasound, bipolar or monopolar RF, irreversible and / or reversible electroporation and / or microwave energy, etc.) through a single port, and these signals may be delivered separately or simultaneously to an end effector to treat tissue.

[0318] The generator 900 includes a processor 902 coupled to a waveform generator 904. The processor 902 and the waveform generator 904 are configured to generate various signal waveforms based on information stored in a memory coupled to the processor 902, which is not shown for clarity of the present disclosure. Digital information associated with the waveform is provided to the waveform generator 904, which includes one or more DAC circuits to convert the digital input into an analog output. The analog output is fed to the amplifier 1106 for signal conditioning and amplification. The conditioned and amplified output of the amplifier 906 is coupled to the power transformer 908. The signal is coupled to the secondary side in the patient isolation side through the power transformer 908. The first signal of the first energy mode is provided to the surgical instrument between the terminals marked as ENERGY1 and RETURN. The second signal of the second energy mode is coupled across the capacitor 910 and provided to the surgical instrument between the terminals marked as ENERGY2 and RETURN. It should be understood that more than two energy modes may be output, and thus the subscript "n" may be used to specify that up to n ENERGYn terminals may be provided, where n is a positive integer greater than 1. It should also be understood that up to "n" return paths RETURNn may be provided without departing from the scope of the present disclosure.

[0319] The first voltage sensing circuit 912 is connected to the terminals marked as ENERGY1 and the two ends of the RETURN path to measure the output voltage therebetween. The second voltage sensing circuit 924 is connected to the terminals marked as ENERGY2 and the two ends of the RETURN path to measure the output voltage therebetween. As shown in the figure, the current sensing circuit 914 is arranged in series with the RETURN branch on the secondary side of the power transformer 908 to measure the output current of any energy mode. If a different return path is provided for each energy mode, a separate current sensing circuit should be provided in each return branch. The outputs of the first voltage sensing circuit 912 and the second voltage sensing circuit 924 are provided to the corresponding isolation transformers 916, 922, and the output of the current sensing circuit 914 is provided to another isolation transformer 918. The outputs of the isolation transformers 916, 928, 922 on the primary side of the power transformer 908 (non-patient isolation side) are provided to one or more ADC circuits 926. The digitized output of the ADC circuit 926 is provided to the processor 902 for further processing and calculation. The output voltage and output current feedback information can be used to adjust the output voltage and current provided to the surgical instrument and calculate parameters such as output impedance. Input / output communication between the processor 902 and the patient isolation circuit is provided through the interface circuit 920. The sensor can also be in electrical communication with the processor 902 through the interface 920.

[0320] In one aspect, the impedance may be determined by the processor 902 by dividing the output of a first voltage sensing circuit 912 connected across terminals labeled ENERGY1 / RETURN or a second voltage sensing circuit 924 connected across terminals labeled ENERGY2 / RETURN by the output of a current sensing circuit 914 arranged in series with the RETURN branch on the secondary side of the power transformer 908. The outputs of the first voltage sensing circuit 912 and the second voltage sensing circuit 924 are provided to separate isolation transformers 916, 922, and the output of the current sensing circuit 914 is provided to another isolation transformer 916. The digitized voltage and current sense measurements from the ADC circuit 926 are provided to the processor 902 for calculating the impedance. For example, the first energy modality ENERGY1 may be ultrasonic energy, and the second energy modality ENERGY2 may be RF energy. However, in addition to ultrasonic and bipolar or monopolar RF energy modalities, other energy modalities include irreversible and / or reversible electroporation and / or microwave energy, etc. Moreover, although Fig.21The illustrated example shows that a single return path RETURN may be provided for two or more energy modalities, but in other aspects, multiple return paths RETURNn may be provided for each energy modality ENERGYn. Thus, as described herein, the ultrasonic transducer impedance may be measured by dividing the output of the first voltage sensing circuit 912 by the output of the current sensing circuit 914, and the tissue impedance may be measured by dividing the output of the second voltage sensing circuit 924 by the output of the current sensing circuit 914.

[0321] like Fig.21 As shown in , a generator 900 including at least one output port may include a power transformer 908 having a single output and multiple taps to provide power to an end effector in the form of one or more energy modalities (such as ultrasound, bipolar or monopolar RF, irreversible and / or reversible electroporation, and / or microwave energy, etc.), for example, depending on the type of tissue treatment being performed. For example, the generator 900 may deliver energy with a higher voltage and lower current to drive an ultrasonic transducer, deliver energy with a lower voltage and higher current to drive an RF electrode for sealing tissue, or deliver energy with a coagulation waveform for use with monopolar or bipolar RF electrosurgical electrodes. The output waveform from the generator 900 may be manipulated, switched, or filtered to provide a frequency to the end effector of the surgical instrument. The connection of the ultrasonic transducer to the output of the generator 900 will preferably be located between the output terminals labeled ENERGY1 and RETURN, as shown in FIG. Fig.21 In one example, the connection of the RF bipolar electrode to the generator 900 output would preferably be between the outputs labeled ENERGY2 and RETURN. In the case of a monopolar output, the preferred connection would be an active electrode (e.g., a pencil or other probe) at the ENERGY2 output and a suitable return pad connected to the RETURN output.

[0322] Additional details are disclosed in U.S. Patent Application Publication No. 2017 / 0086914, published on March 30, 2017, entitled “TECHNIQUES FOR OPERATING GENERATOR FOR DIGITALLY GENERATING ELECTRICAL SIGNAL WAVEFORMS AND SURGICALINSTRUMENTS,” which is incorporated herein by reference in its entirety.

[0323] As used throughout this specification, the term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that can transmit data through a non-solid medium using modulated electromagnetic radiation. The term does not mean that the associated organization does not contain any wires, although in some aspects they may not. The communication module may implement any of a variety of wireless or wired communication standards or protocols, including but not limited to Wi-Fi (IEEE 802.11 series), WiMAX (IEEE 802.16 series), IEEE 802.20, Long Term Evolution (LTE), Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, Bluetooth, and its Ethernet derivatives, and any other wireless and wired protocols designated as 3G, 4G, 5G and above. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communications such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, etc.

[0324] As used herein, a processor or processing unit is an electronic circuit that performs operations on some external data source (usually a memory or some other data stream). The term is used herein to refer to a central processing unit (Central Processing Unit) in one or more systems (especially System on Chip (SoC)) that combine multiple specialized "processors".

[0325] As used herein, a system on a chip or system on a chip (SoC or SOC) is an integrated circuit (also referred to as an "IC" or "chip") that integrates all the components of a computer or other electronic system. It can contain digital, analog, mixed signal, and often radio frequency functions - all on a single substrate. A SoC integrates a microcontroller (or microprocessor) with advanced peripherals such as a graphics processing unit (GPU), a Wi-Fi module, or a coprocessor. A SoC may or may not include built-in memory.

[0326] As used herein, a microcontroller or controller is a system that integrates a microprocessor with peripheral circuits and memory. A microcontroller (or MCU for a microcontroller unit) can be implemented as a small computer on a single integrated circuit. It can be similar to a SoC; an SoC may include a microcontroller as one of its components. A microcontroller may include one or more core processing units (CPUs) as well as memory and programmable input / output peripherals. Program memory in the form of ferroelectric RAM, NOR flash memory, or OTP ROM, as well as a small amount of RAM, is also often included on the chip. Compared to microprocessors used in personal computers or other general-purpose applications consisting of various discrete chips, microcontrollers can be used for embedded applications.

[0327] As used herein, the term controller or microcontroller may be a stand-alone IC or chip device that interfaces with peripheral devices. This may be a link between the two components of a computer or a controller on an external device that manages the operation of the device (and the connection to the device).

[0328] Any of the processors or microcontrollers described herein may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes: 256KB of single-cycle flash or other non-volatile memory (up to 40 MHz) on-chip memory, a pre-fetch buffer for improving performance beyond 40 MHz, 32KB of single-cycle serial random access memory (SRAM), a 32-bit byte buffer with 100 MHz of RAM, ... Internal read-only memory (ROM) for software, 2KB of electrically erasable programmable read-only memory (EEPROM), one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) simulations, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, and other features that are readily available.

[0329] In one example, the processor may include a safety controller including two families based on controllers such as the TMS570 and RM4x also offered by Texas Instruments under the trade name Hercules ARM Cortex R4. The safety controller may be configured specifically for IEC 61508 and ISO 26262 safety critical applications, among others, to provide advanced integrated safety features while delivering scalable performance, connectivity, and memory options.

[0330] The modular device includes a module that can be accommodated in a surgical hub (such as a combination of Figure 3 and Fig. 9The invention relates to a surgical device or instrument that can be connected to various modules so as to be connected or paired with a corresponding surgical hub. The modular device includes, for example, intelligent surgical instruments, medical imaging devices, suction / irrigation devices, smoke evacuators, energy generators, ventilators, insufflators, and displays. The modular device described herein can be controlled by a control algorithm. The control algorithm can be executed on the modular device itself, on a surgical hub paired with a specific modular device, or on both the modular device and the surgical hub (e.g., via a distributed computing architecture). In some examples, the control algorithm of the modular device controls the device based on data sensed by the modular device itself (i.e., by sensors in, on, or connected to the modular device through the modular device). The data can be related to the patient being operated on (e.g., tissue properties or insufflation pressure) or the modular device itself (e.g., the rate at which the knife is advanced, the motor current, or the energy level). For example, the control algorithm of a surgical suturing and cutting instrument can control the rate at which the motor of the instrument drives its knife through the tissue according to the resistance encountered by the knife as it advances.

[0331] Visualization system

[0332] During a surgical procedure, a surgeon may be required to manipulate tissue to achieve a desired medical outcome. The surgeon's movements are limited by what can be visually observed in the surgical site. As a result, the surgeon may not be aware of, for example, the arrangement of vascular structures underlying the tissue being manipulated during surgery. Because the surgeon cannot visualize the vasculature beneath the surgical site, the surgeon may accidentally sever one or more critical blood vessels during surgery. The solution is a surgical visualization system that can acquire imaging data of the surgical site for presentation to the surgeon, in which the presentation can include information related to the presence and depth of vascular structures beneath the surface of the surgical site.

[0333] In one aspect, the surgical hub 106 incorporates a visualization system 108 to acquire imaging data during a surgical procedure. The visualization system 108 may include one or more illumination sources and one or more light sensors. The one or more illumination sources and the one or more light sensors may be incorporated together into a single device, or may include one or more separate devices. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more light sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments. The following description includes all hardware and software processing techniques disclosed above and those applications presented above that are incorporated herein by reference.

[0334] In some aspects, the visualization system 108 can be integrated into the Figure 1 and Figure 2In addition to the visualization system 108, the surgical system 100 may also include one or more handheld intelligent instruments 112, a multifunctional robotic system 110, one or more visualization systems 108, and a centralized surgical hub system 106, among other components. The centralized surgical hub system 106 may control the above-disclosed and also Figure 3 In one non-limiting example, such functionality may include providing and controlling power to any number of powered surgical devices. In another non-limiting example, such functionality may include controlling fluids provided to and removed from a surgical site. The centralized surgical hub system 106 may also be configured to manage and analyze data received from any of the surgical system components, as well as to communicate data and other information among and between the components of the surgical system. The centralized surgical hub system 106 may also be in data communication with the cloud computing system 104, as disclosed above and for example Figure 1 Depicted.

[0335] In some non-limiting examples, the imaging data generated by the visualization system 108 may be analyzed by the onboard computing components of the visualization system 108, and the analysis results may be transmitted to the centralized surgical hub 106. In alternative non-limiting examples, the imaging data generated by the visualization system 108 may be transmitted directly to the centralized surgical hub 106, where the data may be analyzed by the computing components in the hub system 106. The centralized surgical hub 106 may transmit the image analysis results to any one or more of the other components of the surgical system. In some other non-limiting examples, the centralized surgical hub may transmit the image data and / or the image analysis results to the cloud computing system 104.

[0336] Fig.22A -D and Fig.23A-F depicts various aspects of an example of a visualization system 2108 that can be incorporated into a surgical system. The visualization system 2108 may include an imaging control unit 2002 and a handheld unit 2020. The imaging control unit 2002 may include one or more illumination sources, a power source for the one or more illumination sources, one or more types of data communication interfaces (including USB, Ethernet or wireless interfaces 2004), and one or more video outputs 2006. The imaging control unit 2002 may also include an interface configured to transmit integrated video and image capture data to a USB-enabled device, such as a USB interface 2010. The imaging control unit 2002 may also include one or more computing components, including but not limited to a processor unit, a transient memory unit, a non-transitory memory unit, an image processing unit, a bus structure for forming a data link among the computing components, and any interface (e.g., input and / or output) devices necessary to receive information from components not included in the imaging control unit and transmit information to these components. The non-transitory memory may also contain instructions that, when executed by the processor unit, may perform any number of manipulations of data received from the handheld unit 2020 and / or a computing device not included in the imaging control unit.

[0337] The illumination source may include a white light source 2012 and one or more laser sources. The imaging control unit 2002 may include one or more optical interfaces and / or electrical interfaces for optical and / or electrical communication with the handheld unit 2020. As a non-limiting example, the one or more laser sources may include any one or more of a red laser source, a green laser source, a blue laser source, an infrared laser source, and an ultraviolet laser source. In some non-limiting examples, the red laser source may provide illumination with a peak wavelength in a range between 635nm and 660nm, including the end value. Non-limiting examples of the peak wavelength of the red laser may include about 635nm, about 640nm, about 645nm, about 650nm, about 655nm, about 660nm, or any value or range of values ​​therebetween. In some non-limiting examples, the green laser source may provide illumination with a peak wavelength in a range between 520nm and 532nm, including the end value. Non-limiting examples of peak wavelengths of green lasers may include about 520 nm, about 522 nm, about 524 nm, about 526 nm, about 528 nm, about 530 nm, about 532 nm, or any value or range of values ​​therebetween. In some non-limiting examples, a blue laser source may provide irradiation having a peak wavelength in a range between 405 nm and 445 nm, including the end values. Non-limiting examples of peak wavelengths of blue lasers may include about 405 nm, about 410 nm, about 415 nm, about 420 nm, about 425 nm, about 430 nm, about 435 nm, about 440 nm, about 445 nm, or any value or range of values ​​therebetween. In some non-limiting examples, an infrared laser source may provide irradiation having a peak wavelength in a range between 750 nm and 3000 nm, including the end values. Non-limiting examples of peak wavelengths of infrared lasers may include about 750 nm, about 1000 nm, about 1250 nm, about 1500 nm, about 1750 nm, about 2000 nm, about 2250 nm, about 2500 nm, about 2750 nm, 3000 nm, or any value or range of values ​​therebetween. In some non-limiting examples, an ultraviolet laser source may provide irradiation with a peak wavelength in a range between 200 nm and 360 nm, including the end values. Non-limiting examples of peak wavelengths of ultraviolet lasers may include about 200 nm, about 220 nm, about 240 nm, about 260 nm, about 280 nm, about 300 nm, about 320 nm, about 340 nm, about 360 nm, or any value or range of values ​​therebetween.

[0338] In one non-limiting aspect, the handheld unit 2020 may include a body 2021, a camera mirror cable 2015 attached to the body 2021, and an elongated camera probe 2024. The body 2021 of the handheld unit 2020 may include a handheld unit control button 2022 or other controls to allow a health professional to use the handheld unit 2020 to control the operation of the handheld unit 2020 or other components of the imaging control unit 2002 (including, for example, a light source). The camera mirror cable 2015 may include one or more electrical conductors and one or more optical fibers. The camera mirror cable 2015 may terminate at a proximal end with a camera head connector 2008, where the camera head connector 2008 is configured to mate with the one or more optical interfaces and / or electrical interfaces of the imaging control unit 2002. The electrical conductors may provide power to the handheld unit 2020 (including the body 2021 and the elongated camera probe 2024) and / or any electronic components within the handheld unit 2020 (including the body 2021 and / or the elongated camera probe 2024). The electrical conductors may also be used to provide bidirectional data communication between the handheld unit 2020 and any one or more components of the imaging control unit 2002. The one or more optical fibers may conduct illumination from the one or more illumination sources in the imaging control unit 2002 through the handheld unit body 2021 and to the distal end of the elongated camera probe 2024. In some non-limiting aspects, the one or more optical fibers may also conduct light reflected or refracted from the surgical site to one or more optical sensors disposed in the elongated camera probe 2024, the handheld unit body 2021, and / or the imaging control unit 2002.

[0339] Fig. 22B(Top plan view) Some aspects of the handheld unit 2020 of the visualization system 2108 are depicted in more detail. The handheld unit body 2021 may be constructed of a plastic material. The handheld unit control buttons 2022 or other controls may have a rubber overmold to protect the controls while allowing the surgeon to manipulate them. The camera mirror cable 2015 may have an optical fiber integrated with an electrical conductor, and the camera mirror cable 2015 may have a protective and flexible outer covering, such as PVC. In some non-limiting examples, the camera mirror cable 2015 may be about 10 feet long to allow ease of use during surgical procedures. The length of the camera mirror cable 2015 may be in the range of about 5 feet to about 15 feet. Non-limiting examples of the length of the camera mirror cable 2015 may be about 5 feet, about 6 feet, about 7 feet, about 8 feet, about 9 feet, about 10 feet, about 11 feet, about 12 feet, about 13 feet, about 14 feet, about 15 feet, or any length or range of lengths therebetween. The elongated camera probe 2024 can be made of a rigid material such as stainless steel. In some aspects, the elongated camera probe 2024 can be coupled to the handheld unit body 2021 via a rotatable collar 2026. The rotatable collar 2026 can allow the elongated camera probe 2024 to rotate relative to the handheld unit body 2021. In some aspects, the elongated camera probe 2024 can terminate at a distal end with a plastic window 2028 sealed with epoxy.

[0340] Fig. 22C The depicted side plan view of the handheld unit shows that the light or image sensor 2030 can be disposed at the distal end 2032a of the elongated camera probe or within the handheld unit body 2032b. In some alternative aspects, the light or image sensor 2030 can be disposed with additional optical elements in the imaging control unit 2002. Fig. 22C Also depicted is an example of a light sensor 2030 that includes a CMOS image sensor 2034 disposed within a mount 2036 having a radius of approximately 4 mm. Fig.22D Aspects of a CMOS image sensor 2034 are shown, depicting an active area 2038 of the image sensor. Fig. 22C The CMOS image sensor in is depicted as being disposed within a mount 2036 having a radius of approximately 4 mm, but it will be appreciated that such a sensor and mount combination may have any available size to be disposed within the elongated camera probe 2024, the handheld unit body 2021, or disposed in the image control unit 2002. Some non-limiting examples of such alternative mounts may include a 5.5 mm mount 2136a, a 4 mm mount 2136b, a 2.7 mm mount 2136c, and a 2 mm mount 2136d. It will be appreciated that the image sensor may also include a CCD image sensor. A CMOS or CCD sensor may include an array of individual light sensing elements (pixels).

[0341] FIG. 23A to FIG. 23F Various aspects of some examples of illumination sources and their controls that may be incorporated into visualization system 2108 are depicted.

[0342] Fig.23A Aspects of a laser irradiation system having multiple laser beams emitting multiple wavelengths of electromagnetic energy are shown. As can be seen in the figure, the irradiation system 2700 can include a red laser beam 2720, a green laser beam 2730, and a blue laser beam 2740, all optically coupled together by optical fibers 2755. As can be seen in the figure, each of the laser beams can have a corresponding light sensing element or electromagnetic sensor 2725, 2735, 2745, respectively, for sensing the output of a particular laser beam or wavelength.

[0343] about Fig.23A Additional disclosure of the depicted laser illumination system for the surgical visualization system 2108 may be found in U.S. patent application publication 2014 / 0268860, entitled “CONTROLLING THE INTEGRAL LIGHT ENERGY OF ALASER PULSE,” filed on March 15, 2014, which was published on October 3, 2017 as U.S. Patent 9,777,913, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0344] Fig. 23B An operating cycle of the sensor used in rolling readout mode is shown. It should be understood that the x direction corresponds to time, and the diagonal line 2202 indicates the activity of the internal pointer that reads out each data frame, one row at a time. The same pointer is responsible for resetting each row of pixels for the next exposure cycle. The net accumulation time for each row 2219a-c is equal, but they are staggered in time relative to each other due to the rolling reset and reading process. Therefore, for any scene in which adjacent frames are required to represent different configurations of light, the only option for making each row consistent is to pulse the light between readout cycles 2230a-c. More specifically, the maximum available cycle corresponds to the sum of the blanking time plus any time to service the optical black row or optical blind (OB) row (2218, 2220) at the beginning or end of the frame.

[0345] Fig. 23BAn operating cycle of a sensor used in a rolling readout mode or during a sensor readout 2200 is shown. A frame readout may begin at a vertical line 2210 and may be represented by the vertical line. A readout cycle is represented by a diagonal or oblique line 2202. The sensor may be read out row by row, with the top of the downward sloping edge being the sensor top row 2212 and the bottom of the downward sloping edge being the sensor bottom row 2214. The time between the last row readout and the next readout cycle may be referred to as a blanking time 2216a-d. It should be understood that the blanking times 2216a-d may be the same between successful readout cycles, or they may be different between successful readout cycles. It should be noted that some of the sensor pixel rows may be covered with a light shield (e.g., a metal coating or any other substantially black layer of another material type). These covered pixel rows may be referred to as optical black rows 2218 and 2220. The optical black rows 2218 and 2220 may be used as input to the correction algorithm.

[0346] like Fig. 23B As shown, these optical black rows 2218 and 2220 may be located on the top of the pixel array or at the bottom of the pixel array or at the top and bottom of the pixel array. In some aspects, it may be desirable to control the amount of electromagnetic radiation (e.g., light) exposed to the pixel so that it is integrated or accumulated by the pixel. It should be understood that photons are the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into an electric charge or current. In some aspects, an electronic shutter or rolling shutter may be used to start the accumulation time (2219a-c) by resetting the pixel. The light will then be integrated until the next readout stage. In some aspects, the position of the electronic shutter may be moved between two readout cycles 2202 to control the pixel saturation of a given amount of light. In some alternative aspects lacking an electronic shutter, the accumulation time 2219a-c of the incident light may start during the first readout cycle 2202 and may end at the next readout cycle 2202, which also defines the start of the next accumulation. In some alternative aspects, the amount of light accumulated by each pixel can be controlled by the time 2230a-d that the light is pulsed during the blanking times 2216a-d. This ensures that all rows see the same light emitted from the same light pulse 2230a-c. In other words, each row will begin its accumulation in a first dark environment 2231, which may have a maximum light pulse width at the optically black rear row 2220 of the readout frame (m), and will then receive the light strobe and will end its accumulation in a second dark environment 2232, which may have a maximum light pulse width at the optically black front row 2218 of the next subsequent readout frame (m+1). Therefore, the image generated by the light pulses 2230a-c will only be available during the frame (m+1) readout, without interfering with the frames (m) and (m+2).

[0347] It should be noted that the condition for the light pulses 2230a-c to be read out in only one frame and not interfere with the adjacent frames is that a given light pulse 2230a-c is fired during the blanking time 2216. Because the optical black rows 2218, 2220 are not sensitive to light, the optical black post-row 2220 time of frame (m) and the optical black pre-row 2218 time of frame (m+1) can be added to the blanking time 2216 to determine the maximum range of the firing time of the light pulse 2230.

[0348] In some respects, Fig. 23B Depicted is an example of a timing diagram for sequential frame capture by a conventional CMOS sensor. Such a CMOS sensor may incorporate a Bayer color filter pattern, such as Fig.23C As depicted. It is recognized that the Bayer pattern provides greater detail in luminance than chrominance. It is also recognized that the sensor has reduced spatial resolution because a total of 4 adjacent pixels are required to produce color information for the aggregated spatial portion of the image. In an alternative approach, a color image can be constructed by rapidly strobing the visualization area with various light sources (lasers or light emitting diodes) having different central light wavelengths at high speed.

[0349] The optical gating system may be under the control of the camera system and may include a specially designed CMOS sensor with high speed readout. The main benefit is that the sensor can achieve the same spatial resolution with significantly fewer pixels than a conventional Bayer or 3-sensor camera. Thus, the physical space occupied by the pixel array may be reduced. The actual pulse period (2230a-c) may vary within the repeating pattern, such as Fig. 23B This can be used, for example, to allocate longer time to components that require greater light energy or components with weaker light sources. As long as the average frame rate captured is an integer multiple of the required final system frame rate, the data can simply be buffered in the signal processing chain as needed.

[0350] The facility to reduce the CMOS sensor chip area to a permissible degree by combining all of these approaches is particularly attractive for small diameter (approximately 3 to 10 mm) endoscopes. Specifically, it allows the design of endoscopes in which the sensor is located in the spatially constrained distal end, thereby greatly reducing the complexity and cost of the optical section while providing high definition video. A consequence of this approach is that in order to reconstruct each final full-color image, the data needs to be fused from three separate snapshots in time. Any motion within the scene relative to the optical reference frame of the endoscope will generally reduce the perceived resolution because the edges of objects appear at slightly different locations within each captured component. In the present disclosure, an apparatus is described that reduces this problem, which exploits the fact that spatial resolution is more important for luminance information than for chrominance.

[0351] The basis of this method is that instead of firing a single color of light during each frame, a combination of three wavelengths is used to provide all the luminance information within a single image. Chrominance information is derived from a 3D image with a repeating pattern such as Y-Cb-Y-Cr ( Fig.23D ). While pure luminance data can be provided by careful choice of the pulse ratio, the same is not true for chrominance.

[0352] In one aspect, Fig.23D As shown, an endoscopic system 2300a may include a pixel array 2302a having uniform pixels, and the system 2300a may be operated to receive Y (luminance pulses) 2304a, Cb (ChromaBlue) 2306a, and Cr (ChromaRed) 2308a pulses.

[0353] To complete a full color image, the two chrominance components need to be present as well. However, the same algorithm applied to luma cannot be directly applied to chroma images because it is signed, as reflected in the fact that some of the RGB coefficients are negative. The solution to this is to add luma of sufficient magnitude so that all final pulse energies become positive. As long as the color fusion process in the ISP knows the composition of the chroma frames, they can be decoded by subtracting the appropriate amount of luma from the adjacent frames. The pulse energy ratio is given by:

[0354] Y=0.183·R+0.614·G+0.062·B

[0355] Cb=λ·Y-0.101·R-0.339·G+0.439·B

[0356] Cr=δ·Y+0.439·R-0.399·G-0.040·B

[0357] in

[0358] λ≥0.399 / 0.614=0.552

[0359] δ≥0.399 / 0.614=0.650

[0360] The results show that if the λ factor is equal to 0.552; both the red component and the green component are precisely eliminated, in which case the Cb information can be set with pure blue light. Similarly, setting δ = 0.650 eliminates the blue component and the green component of Cr, and Cr becomes pure red. This particular example is Fig.23E , which also depicts λ and δ as 1 / 2 8 This is a convenient approximation for digital frame reconstruction.

[0361] In the case of the Y-Cb-Y-Cr pulse scheme, the image data is already in the YCbCr space after color fusion. So in this case it makes sense to perform luma and chroma based operations up front before converting back to linear RGB to perform color correction etc.

[0362] The color fusion process is more complicated than the Bayer pattern (see Fig.23C ) is simpler because there is no spatial interpolation. But it does require buffering of frames so that all the necessary information is available for each pixel. In one general aspect, the data for the Y-Cb-Y-Cr pattern can be pipelined to produce one full-color image for every two original captured images. This is achieved by using each chrominance sample twice. Fig.23F A specific example of a 120 Hz frame capture rate providing a 60 Hz final video is depicted in FIG.

[0363] About FIG. 23B to FIG. 23F Additional disclosure of the control of the laser components of the illumination system depicted for the surgical visualization system 108 may be found in U.S. patent application publication 2014 / 0160318, filed on July 26, 2013, entitled “YCBCR PULSED ILLUMINATIONSCHEME IN A LIGHT DEFICIENT ENVIRONMENT,” which was published on December 6, 2016 as U.S. Patent 9,516,239, and U.S. patent application publication 2014 / 0160319, filed on July 26, 2013, entitled “CONTINUOUS VIDEO IN A LIGHT DEFICIENT ENVIRONMENT,” which was published on August 22, 2017 as U.S. Patent 9,743,016, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0364] Subsurface vascular imaging

[0365] During a surgical procedure, a surgeon may be required to manipulate tissue to achieve a desired medical outcome. The surgeon's movements are limited by what can be visually observed in the surgical site. Therefore, the surgeon may not be aware of, for example, the arrangement of vascular structures underlying the tissue being manipulated during surgery.

[0366] Because the surgeon cannot visualize the vasculature beneath the surgical site, the surgeon may accidentally sever one or more critical blood vessels during surgery.

[0367] It would therefore be desirable to have a surgical visualization system that can acquire imaging data of a surgical site for presentation to a surgeon, wherein the presentation can include information related to the presence of vascular structures located beneath the surface of the surgical site.

[0368] Some aspects of the present disclosure also provide control circuitry configured to control irradiation of a surgical site using one or more irradiation sources, such as a laser source, and to receive imaging data from one or more image sensors. In some aspects, the present disclosure provides a non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a device to detect a blood vessel in tissue and determine its depth below the tissue surface.

[0369] In some aspects, a surgical image acquisition system may include: a plurality of illumination sources, wherein each illumination source is configured to emit light having a specified central wavelength; a light sensor, the light sensor configured to receive a portion of the light reflected from the tissue sample when the tissue sample is illuminated by one or more of the plurality of illumination sources; and a computing system. The computing system may be configured to: receive data from the light sensor when the tissue sample is illuminated by each of the plurality of illumination sources; determine a depth position of a structure within the tissue sample based on the data received by the light sensor when the tissue sample is illuminated by each of the plurality of illumination sources, and calculate visualization data about the structure and the depth position of the structure. In some aspects, the visualization data may have a data format usable by a display system, and the structure may include one or more vascular tissues.

[0370] Vascular imaging using NIR spectroscopy

[0371] In one aspect, a surgical image acquisition system may include an independent color cascade of illumination sources including visible light and light outside the visible range to image one or more tissues within a surgical site at different times and at different depths. The surgical image acquisition system may also detect or calculate characteristics of light reflected and / or refracted from the surgical site. The characteristics of the light may be used to provide a composite image of tissue within the surgical site, as well as to provide analysis of underlying tissue that is not directly visible at the surface of the surgical site. The surgical image acquisition system may determine tissue depth location without the need for a separate measurement device.

[0372] In one aspect, the characteristic of light reflected and / or refracted from the surgical site can be the amount of absorbance of light at one or more wavelengths.The various chemical compositions of individual tissues can result in specific light absorption patterns that are wavelength dependent.

[0373] In one aspect, the irradiation source may include a red laser source and a near-infrared laser source, wherein the one or more tissues to be imaged may include vascular tissue, such as a vein or an artery. In some aspects, a red laser source (in the visible range) may be used to image some aspects of the underlying vascular tissue based on spectroscopy in the visible red range. In some non-limiting examples, a red laser source may provide irradiation with a peak wavelength in a range between 635nm and 660nm, including the end value. Non-limiting examples of the peak wavelength of the red laser may include about 635nm, about 640nm, about 645nm, about 650nm, about 655nm, about 660nm, or any value or range of values ​​therebetween. In some other aspects, a near-infrared laser source may be used to image the underlying vascular tissue based on near-infrared spectroscopy. In some non-limiting examples, a near-infrared laser source may emit irradiation with a wavelength in a range between 750 and 3000nm, including the end value. Non-limiting examples of peak wavelengths for infrared lasers may include about 750 nm, about 1000 nm, about 1250 nm, about 1500 nm, about 1750 nm, about 2000 nm, about 2250 nm, about 2500 nm, about 2750 nm, 3000 nm, or any value or range of values ​​therebetween. It can be appreciated that a combination of red and infrared spectra may be used to detect underlying vascular tissue. In some examples, a red laser source having a peak wavelength at about 660 nm and a near IR laser source having a peak wavelength at about 750 nm or at about 850 nm may be used to detect vascular tissue.

[0374] Near infrared spectroscopy (NIRS) is a non-invasive technique that allows tissue oxygenation to be determined based on the spectrophotometric quantification of oxyhemoglobin and deoxyhemoglobin within tissue. In some aspects, NIRS can be used to directly image vascular tissue based on the difference in illumination absorbance between vascular tissue and non-vascular tissue. Alternatively, vascular tissue can be indirectly visualized based on the difference in illumination absorbance of blood flow in the tissue before and after application of physiological interventions such as arterial occlusion methods and venous occlusion methods.

[0375] Instruments for near infrared (NIR) spectroscopy can be similar to those for the UV visible and mid-IR ranges. Such spectroscopy instruments may include an irradiation source, a detector, and a dispersive element to select a specific near IR wavelength for irradiating a tissue sample. In some aspects, the source may include an incandescent light source or a quartz halogen light source. In some aspects, the detector may include a semiconductor (e.g., InGaAs) photodiode or a photoelectric array. In some aspects, the dispersive element may include a prism or, more commonly, a diffraction grating. Fourier transform NIR instruments using interferometers are also common, especially for wavelengths greater than about 1000 nm. Depending on the sample, the spectrum may be measured in reflection or transmission mode.

[0376] Fig.24 An example of an instrument 2400 similar to an instrument for the UV visible and mid-IR range of the NIR spectrum is schematically depicted. A light source 2402 may emit illumination 2404 of a broad spectral range that may be projected onto a dispersive element 2406 (such as a prism or diffraction grating). The dispersive element 2406 may be operable to select a narrow wavelength portion 2408 of the light emitted by the broad spectrum light source 2402, and the selected light portion 2408 may illuminate tissue 2410. Light reflected from the tissue 2412 may be directed to a detector 2416 (e.g., using a dichroic mirror 2414), and the intensity of the reflected light 2412 may be recorded. The wavelength of light illuminating the tissue 2410 may be selected by the dispersive element 2406. In some aspects, the tissue 2410 may be illuminated by only a single narrow wavelength portion 2408 selected by the dispersive element 2406 from the light source 2402. In other aspects, the tissue 2410 may be scanned with various narrow wavelength portions 2408 selected by the dispersive element 2406. In this way, a spectral analysis of tissue 2410 may be obtained in the NIR wavelength range.

[0377] Fig.25 An example of an apparatus 2430 for determining NIRS based on Fourier transform infrared imaging is schematically depicted. Fig.25 , a laser source emitting 2432 light in a near IR range 2434 illuminates a tissue sample 2440. Light reflected 2436 by the tissue 2440 is reflected 2442 by a mirror, such as a dichroic mirror 2444, to a beam splitter 2446. The beam splitter 2446 directs a portion of the light 2448 reflected 2436 by the tissue 2440 to a stationary mirror 2450, and directs a portion of the light 2452 reflected 2436 by the tissue 2440 to a moving mirror 2454. The moving mirror 2454 can oscillate in position based on an attached piezoelectric transducer activated by a sinusoidal voltage having a voltage frequency. The position of the moving mirror 2454 in space corresponds to the frequency of the sinusoidal activation voltage of the piezoelectric transducer. Light reflected from the moving mirror and the stationary mirror can be recombined 2458 at the beam splitter 2446 and directed to a detector 2456. The computing component may receive the signal output of the detector 2456 and perform a Fourier transform (in time) of the received signal. Because the wavelength of light received from the moving reflector 2454 varies in time relative to the wavelength of light received from the stationary reflector 2450, the time-based Fourier transform of the recombined light corresponds to the wavelength-based Fourier transform of the recombined light 2458. In this way, the wavelength-based spectrum of light reflected from the tissue 2440 may be determined, and the spectral characteristics of the light reflected 2436 from the tissue 2440 may be obtained. Thus, changes in the absorbance of illumination from the spectral components of the light reflected from the tissue 2440 may indicate the presence or absence of tissue having a particular light absorption characteristic, such as hemoglobin.

[0378] An alternative to near infrared light to determine hemoglobin oxygenation would be to use monochromatic red light to determine the red light absorbance properties of hemoglobin. The absorbance properties of hemoglobin for red light with a central wavelength of about 660 nm can indicate whether the hemoglobin is oxygenated (arterial blood) or deoxygenated (venous blood).

[0379] In some alternative surgical procedures, contrast agents may be used to improve the data collected regarding oxygenation and tissue oxygen consumption. In one non-limiting example, NIRS technology may be used in conjunction with a bolus injection of a near IR contrast agent with a peak absorbance at about 800 nm, such as indocyanine green (ICG). ICG has been used in some medical procedures to measure cerebral blood flow.

[0380] Vascular imaging using laser Doppler flowmetry

[0381] In one aspect, the characteristic of light reflected and / or refracted from the surgical site can be a Doppler shift in the wavelength of light from its irradiating source.

[0382] Laser Doppler flowmetry can be used to visualize and characterize the flow of particles moving relative to an effectively stationary background. Therefore, laser light scattered by moving particles (such as blood cells) may have a different wavelength than the original illuminating laser source. In contrast, laser light scattered by an effectively stationary background (e.g., vascular tissue) may have the same wavelength as the original illuminating laser source. Changes in the wavelength of scattered light from blood cells can reflect both the flow direction of blood cells relative to the laser source and the blood cell velocity. Fig.26A -C shows the change in wavelength of light scattered from a blood cell, which may be moving away from ( Fig.26A ) or towards ( Fig.26C )The laser source moves.

[0383] exist Fig.26A In each of the figures in -C, the original illumination light 2502 is depicted as having a relative central wavelength of zero. Fig.26A It can be observed in Figure 25 that light scattered from blood cells moving away from the laser source 2504 has a wavelength that is shifted by a certain amount 2506 to a larger wavelength relative to the laser source (and is therefore red-shifted). Fig.26C It can also be observed in Figure 25 that light scattered from blood cells moving toward laser source 2508 has a wavelength shifted by a certain amount 2510 to a shorter wavelength relative to the laser source (and is therefore blue-shifted). The amount of wavelength shift (e.g., 2506 or 2510) can depend on the speed of movement of the blood cells. In some aspects, the amount of red shift (2506) of some blood cells can be approximately the same as the amount of blue shift (2510) of some other blood cells. Alternatively, the amount of red shift (2506) of some blood cells can be different from the amount of blue shift (2510) of some other blood cells. Therefore, based on the relative magnitudes of the wavelength shifts (2506 and 2510), as shown in Figure 25, the amount of blue shift (2510) of some other blood cells can be different. Fig.26A The velocity of the blood cells depicted moving away from the laser source may be less than Fig.26C The velocity of blood cells flowing toward the laser source is depicted. Fig.26B As depicted, light scattered from tissue that does not move relative to the laser source (eg, blood vessel 2512 or non-vascular tissue 2514) may not exhibit any wavelength variation.

[0384] Fig. 27 Aspects of an instrument 2530 that can be used to detect a Doppler shift of laser light scattered from a portion of tissue 2540 are depicted. Light 2534 originating from a laser 2532 can pass through a beam splitter 2544. Some portion of the laser light 2536 can be transmitted by the beam splitter 2544 and can illuminate the tissue 2540. Another portion of the laser light can be reflected 2546 by the beam splitter 2544 to be projected on a detector 2550. Light backscattered 2542 by the tissue 2540 can be directed by the beam splitter 2544 and also projected on the detector 2550. The combination of the light 2534 originating from the laser 2532 and the light 2542 backscattered by the tissue 2540 can result in an interference pattern detected by the detector 2550. The interference pattern received by the detector 2550 can include interference fringes resulting from the combination of the light 2534 originating from the laser 2532 and the Doppler shifted (and therefore wavelength shifted) light backscattered 2452 from the tissue 2540.

[0385] It will be appreciated that the backscattered light 2542 from the tissue 2540 may also include backscattered light from boundary layers within the tissue 2540 and / or wavelength-specific absorption of light by materials within the tissue 2540. Therefore, the interference pattern observed at the detector 2550 may incorporate interference fringe features from these additional optical effects and, therefore, may confound the calculation of the Doppler shift unless properly analyzed.

[0386] Fig.28 Some of these additional optical effects are depicted. As is well known, light that passes through a first optical medium having a first refractive index n1 may be reflected at an interface with a second optical medium having a second refractive index n2. Light that is transmitted through the second optical medium will have a transmission angle relative to the interface that is different from the angle of the incident light based on the difference between the refractive indices n1 and n2 (Snell's law). Fig.28The effect of Snell's law on light projected on a surface of a multi-component tissue 2150 is shown, as may be present in a surgical field. The multi-component tissue 2150 may be comprised of an outer tissue layer 2152 having a refractive index n1 and an embedded tissue, such as a blood vessel having a vessel wall 2156. The vessel wall 2156 may be characterized by a refractive index n2. Blood may flow within the lumen of the vessel 2160. In some aspects, it may be important to determine the location of the vessel 2160 beneath the surface 2154 of the outer tissue layer 2152 during surgery and characterize the blood flow using Doppler shift techniques.

[0387] Incident laser light 2170a may be used to detect blood vessels 2160 and may be directed onto the top surface 2154 of the outer tissue layer 2152. A portion 2172 of the incident laser light 2170a may be reflected at the top surface 2154. Another portion 2170b of the incident laser light 2170a may penetrate the outer tissue layer 2152. The reflected portion 2172 at the top surface 2154 of the outer tissue layer 2152 has the same path length as the incident light 2170a, and therefore has the same wavelength and phase as the incident light 2170a. However, the portion 2170b of the light transmitted into the outer tissue layer 2152 will have a transmission angle different from the incident angle of the light projected onto the tissue surface because the outer tissue layer 2152 has a refractive index n1 that is different from the refractive index of air.

[0388] If the portion of light transmitted through the outer tissue layer 2152 is projected onto a second tissue surface 2158, such as a vessel wall 2156, some portion 2174a, b of the light will be reflected back toward the incident light 2170a. Thus, the light 2174a reflected at the interface between the outer tissue layer 2152 and the vessel wall 2156 will have the same wavelength as the incident light 2170a, but will be phase shifted due to the change in light path length. Projecting the reflected light 2174a, b from the interface between the outer tissue layer 2152 and the vessel wall 2156 together with the incident light onto the sensor will produce an interference pattern based on the phase difference between the two light sources.

[0389] Additionally, a portion of the incident light 2170c may be transmitted through the vessel wall 2156 and penetrate into the vessel lumen 2160. This portion of the incident light 2170c may interact with the blood cells moving in the vessel lumen 2160 and may be reflected back 2176a-c toward a projected light source having a wavelength Doppler shifted according to the velocity of the blood cells, as disclosed above. The Doppler-shifted light 2176a-c reflected from the moving blood cells may be projected onto the sensor along with the incident light, thereby generating an interference pattern having a fringe pattern based on the wavelength difference between the two light sources.

[0390] exist Fig.28, if there is no refractive index change between the emitted light and the light reflected by the moving blood cells, an optical path 2178 of light projected onto the red blood cells in the vascular lumen 2160 is presented. In this example, only the Doppler shift of the wavelength of the reflected light can be detected. However, in addition to the wavelength change due to the Doppler effect, the light reflected by the blood cells (2176a-c) can also incorporate phase changes due to changes in the refractive index of the tissue.

[0391] Therefore, it should be understood that if the light sensor receives incident light, light reflected from one or more tissue interfaces (2172 and 2174a, b), and Doppler shifted light from blood cells (2176a-c), the interference pattern thus produced on the light sensor may include effects due to the Doppler shift (wavelength changes) as well as effects due to refractive index changes within the tissue (phase changes). Therefore, if the effects due to refractive index changes within the sample are not compensated, Doppler analysis of light reflected by the tissue sample may produce erroneous results.

[0392] Fig.29 An example of the effect of Doppler analysis of light projected 2250 on a tissue sample is shown to determine the depth and location of underlying blood vessels. If there is no intervening tissue between the blood vessel and the tissue surface, the interference pattern detected at the sensor may be primarily due to wavelength changes reflected from moving blood cells. Therefore, the spectrum 2252 derived from the interference pattern may generally reflect only the Doppler shift of the blood cells. However, if there is intervening tissue between the blood vessel and the tissue surface, the interference pattern detected at the sensor may be due to a combination of wavelength changes reflected from moving blood cells and phase shifts due to the refractive index of the intervening tissue. The spectrum 2254 derived from such an interference pattern may result in the calculation of a Doppler shift that is confounded by the additional phase changes in the reflected light. In some aspects, if information about the characteristics of the intervening tissue (thickness and refractive index) is known, the resulting spectrum 2256 may be corrected to provide a more accurate calculation of the wavelength change.

[0393] It is recognized that the tissue penetration depth of light depends on the wavelength of the light used. Therefore, the laser source light wavelength can be selected to detect particle motion (such as blood cells) within a specific tissue depth range. Fig. 30A -C schematically depicts a device for detecting moving particles (such as blood cells) at various tissue depths based on laser wavelength. Fig. 30AAs shown, a laser source 2340 can direct an incident laser beam 2342 onto a surface 2344 of a surgical site. A blood vessel 2346 (such as a vein or artery) can be disposed within a tissue 2348 at a depth δ from the tissue surface. The penetration depth 2350 of the laser into the tissue 2348 can depend at least in part on the laser wavelength. Thus, a laser having a wavelength in the red range of about 635nm to about 660nm can penetrate tissue 2351a to a depth of about 1 mm. A laser having a wavelength in the green range of about 520nm to about 532nm can penetrate tissue 2351b to a depth of about 2 to 3 mm. A laser having a wavelength in the blue range of about 405nm to about 445nm can penetrate tissue 2351c to a depth of about 4mm or more. Fig. 30A In the example depicted in FIG. 2346 , blood vessel 2346 may be located at a depth δ of approximately 2 to 3 mm below the tissue surface. Red laser light will not penetrate to this depth, and therefore will not detect blood cells flowing within the blood vessel. However, both green laser light and blue laser light may penetrate to this depth. Therefore, scattered green laser light and blue laser light from blood cells within blood vessel 2346 may exhibit a Doppler shift in wavelength.

[0394] Fig. 30B 2354 . The Doppler shift 2355 of the reflected laser wavelength may appear as shown. The emitted light (or laser source light 2342) projected on the tissue surface 2344 may have a central wavelength 2352. For example, the central wavelength 2352 of the light from a green laser may be in the range of about 520 nm to about 532 nm. If the light is reflected from particles such as red blood cells moving away from the detector, the central wavelength 2354 of the reflected green light may be shifted to a longer wavelength (red shifted). The difference between the central wavelength 2352 of the emitted laser and the central wavelength 2354 of the emitted laser comprises the Doppler shift 2355.

[0395] As above relative to Fig.28 and Fig.29 As disclosed, laser light reflected from structures within tissue 2348 may also show a phase shift in the reflected light due to changes in refractive index caused by changes in tissue structure or composition. The emitted light (or laser source light 2342) projected onto the tissue surface 2344 may have a first phase characteristic 2356. The reflected laser light may have a second phase characteristic 2358. It can be appreciated that a blue laser light that may penetrate tissue to a depth 2351c of about 4 mm or more may encounter a wider variety of tissue structures than a red laser light (about 1 mm 2351a) or a green laser light (about 2 to 3 mm 2351b). Thus, as Fig. 30C As shown, the phase shift 2358 of the reflected blue laser light may be significant due at least to the penetration depth.

[0396] Fig.30D2360a, green laser 2360b, and blue laser 2360c are shown. In some aspects, tissue can be probed in a sequential manner by red laser irradiation 2360a, green laser irradiation 2360b, and blue laser irradiation 2360c. In some alternative examples, one or more combinations of red laser 2360a, green laser 2360b, and blue laser 2360c (e.g., FIG. 23D to FIG. 23F 30D shows the effect of such illumination on CMOS imaging sensors 2362a-d over time. Thus, at a first time t1, CMOS sensor 2362a may be illuminated by red laser 2360a. At a second time t2, CMOS sensor 2362b may be illuminated by green laser 2360b. At a third time t3, CMOS sensor 2362c may be illuminated by blue laser 2360c. The illumination cycle may then be repeated starting at a fourth time t4, at which time CMOS sensor 2362d may again be illuminated by red laser 2360a. It may be appreciated that sequential illumination of tissue by laser illumination of different wavelengths may allow Doppler analysis to be performed at varying tissue depths over time. While red laser source 2360a, green laser source 2360b, and blue laser source 2360c may be used to illuminate a surgical site, it may be appreciated that other wavelengths outside of visible light (such as in the infrared or ultraviolet region) may be used to illuminate a surgical site for Doppler analysis.

[0397] Fig.31 An example of using Doppler imaging to detect the presence of otherwise invisible blood vessels at the surgical site 2600 is shown. Fig.31 In , the surgeon may wish to remove a tumor 2602 present in the right upper posterior lobe 2604 of the lung. Because the lung is highly vascularized, care must be taken to identify only those blood vessels associated with the tumor and to seal only those blood vessels without compromising blood flow to unaffected portions of the lung. Fig.31 2604. The surgeon may then cut an initial anatomical region 2608 in the rim region 2606 and may view exposed blood vessels 2610 for cutting and sealing. A Doppler imaging detector 2620 may be used to locate and identify blood vessels 2612 that are not observable in the anatomical region. The imaging system may receive data from the Doppler imaging detector 2620 for analysis and display of data acquired from the surgical site 2600. In some aspects, the imaging system may include a display to show the surgical site 2600 including a visible image of the surgical site 2600 along with an image overlay of hidden blood vessels 2612 on the image of the surgical site 2600.

[0398] In the above about Fig.31In the disclosed scenario, the surgeon wishes to cut off blood vessels that supply oxygen and nutrients to the tumor while preserving blood vessels associated with non-cancerous tissue. Additionally, blood vessels may be located at different depths in or around the surgical site 2600. Therefore, the surgeon must identify the location (depth) of the blood vessels and determine whether they are suitable for resection. Fig.32 One method of identifying deep blood vessels based on the Doppler shift of light from blood cells flowing through them is shown. As disclosed above, the penetration depth of red laser light is about 1 mm, and the penetration depth of green laser light is about 2 to 3 mm. However, at these wavelengths, blood vessels that are 4 mm or deeper below the surface will be outside the penetration depth. However, a blue laser light can detect such blood vessels based on their blood flow.

[0399] Fig.32 The Doppler shift of laser light reflected from a blood vessel at a specific depth below a surgical site is depicted. The site can be illuminated by a red laser, a green laser, and a blue laser. The central wavelength 2630 of the illuminating light can be normalized to a relative center 3631. If the blood vessel is located at a depth of 4 mm or more below the surface of the surgical site, neither the red laser nor the green laser will be reflected by the blood vessel. Therefore, the central wavelength 2632 of the reflected red light and the central wavelength 2634 of the reflected green light will not differ much from the central wavelength 2630 of the illuminating red or green light, respectively. However, if the site is illuminated by a blue laser, the central wavelength 2638 of the reflected blue light 2636 will be different from the central wavelength 2630 of the illuminating blue light. In some cases, the amplitude of the reflected blue light 2636 can also be significantly reduced at the amplitude of the illuminating blue light. Therefore, the surgeon can determine the presence of deep blood vessels along with their approximate depth, and thereby avoid deep blood vessels during surface tissue dissection.

[0400] Fig.33 and Fig.34 Schematically illustrated is the use of laser sources with different central wavelengths (colors) to determine the approximate depth of a blood vessel below the surface of a surgical site. Fig.33A first surgical site 2650 having a surface 2654 and a blood vessel 2656 disposed below the surface 2654 is depicted. In one approach, the blood vessel 2656 may be identified based on a Doppler shift of light projected onto a flow of blood cells 2658 within the blood vessel 2656. The surgical site 2650 may be illuminated by light from a plurality of lasers 2670, 2676, 2682, each laser being characterized by emitting light at one of several different central wavelengths. As described above, illumination by a red laser 2670 may only penetrate tissue by approximately 1 mm. Thus, if the blood vessel 2656 is located at a depth 2672 less than 1 mm below the surface 2654, the red laser illumination will be reflected 2674, and the Doppler shift of the reflected red illumination 2674 may be determined. Additionally, as described above, illumination by a green laser 2676 may only penetrate tissue by approximately 2 to 3 mm. If the blood vessel 2656 is located at a depth 2678 of about 2 to 3 mm below the surface 2654, the green laser illumination will be reflected 2680 while the red laser illumination 2670 will not be reflected, and the Doppler shift of the reflected green illumination 2680 can be determined. Fig.33 As depicted, blood vessel 2656 is located at a depth 2684 of approximately 4 mm below surface 2654. Thus, neither red laser illumination 2670 nor green laser illumination 2676 will be reflected. Instead, only blue laser illumination will be reflected 2686, and the Doppler shift of reflected blue illumination 2686 can be determined.

[0401] and Fig.33 The depicted blood vessels 2656 are compared to Fig.34 The depicted blood vessel 2656' is located closer to the tissue surface at the surgical site. Blood vessel 2656' may also be distinguished from blood vessel 2656 in that blood vessel 2656' is shown as having a thicker wall 2657. Thus, blood vessel 2656' may be an example of an artery while blood vessel 2656 may be an example of a vein, as arterial walls are known to be thicker than venous walls. In some examples, the thickness of the arterial wall may be about 1.3 mm. As disclosed above, red laser radiation 2670' may penetrate tissue to a depth 2672' of about 1 mm. Thus, even if blood vessel 2656' is exposed to the surgical site (see Fig.312610), due to the thickness of the vessel wall 2657, the red laser light reflected 2674' from the surface of the vessel 2656' may also not be able to visualize the blood flow 2658' within the vessel 2656' under Doppler analysis. However, as disclosed above, the green laser light 2676' projected onto the tissue surface may penetrate to a depth 2678' of about 2 to 3 mm. In addition, the blue laser light projected 2682' onto the tissue surface may penetrate to a depth 2684' of about 4 mm. Therefore, the green laser light may be reflected 2680' from the blood cells flowing 2658' within the vessel 2656', and the blue laser light may be reflected 2686' from the blood cells flowing 2658' within the vessel 2656'. Therefore, Doppler analysis of the reflected green light 2680' and the reflected blue light 2686' may provide information about the blood flow in the near-surface vessels, especially the approximate depth of the vessels.

[0402] As disclosed above, the depth of blood vessels below the surgical site can be detected based on wavelength-dependent Doppler imaging. The blood flow through such blood vessels can also be determined by speckle contrast (interference) analysis. Doppler shifts can indicate moving particles relative to a stationary light source. As disclosed above, Doppler wavelength shifts can be an indication of the speed of particle motion. Individual particles such as blood cells may not be observed individually. However, the speed of each blood cell will produce a proportional Doppler shift. Due to the difference in the Doppler shift of the backscattered light from each blood cell among the blood cells, an interference pattern can be generated by the combination of light backscattered from multiple blood cells. The interference pattern can be an indication of the number density of blood cells within the visualization frame. The interference pattern can be referred to as speckle contrast. Speckle contrast analysis can be calculated using a full-frame 300×300 CMOS imaging array, and the speckle contrast can be directly related to the amount of moving particles (e.g., blood cells) that interact with the laser within a given exposure period.

[0403] The CMOS image sensor can be connected to a digital signal processor (DSP). Each pixel of the sensor can be multiplexed and digitized. The Doppler shift in the light can be analyzed by observing the source laser compared to the Doppler shifted light. Larger Doppler shifts and speckles can be associated with a larger number of blood cells and their velocity in the blood vessel.

[0404] Fig.35 Depicted are aspects of a composite visual display 2800 that may be presented to a surgeon during a surgical procedure. The composite visual display 2800 may be constructed by overlaying a white light image 2830 of the surgical site with a Doppler analysis image 2850.

[0405] In some aspects, the white light image 2830 can depict a surgical site 2832, one or more surgical incisions 2834, and tissue 2836 that is optionally visible within the surgical incisions 2834. The white light image 2830 can be generated by illuminating 2840 the surgical site 2832 with a white light source 2838 and receiving reflected white light 2842 by an optical detector. Although the white light source 2838 can be used to illuminate the surface of the surgical site, in one aspect, as described above with respect to FIG. 23C to FIG. 23F As disclosed, an appropriate combination of red laser light 2854, green laser light 2856, and blue laser light 2858 may be used to visualize the surface of the surgical site.

[0406] In some aspects, the Doppler analysis image 2850 may include vessel depth information along with blood flow information 2852 (from speckle analysis). As disclosed above, vessel depth and blood flow velocity may be obtained by illuminating the surgical site with lasers of multiple wavelengths and determining vessel depth and blood flow based on the known penetration depth of light of a particular wavelength. Generally speaking, the surgical site 2832 may be illuminated by light emitted by one or more lasers, such as a red laser 2854, a green laser 2856, and a blue laser 2858. The CMOS detector 2872 may receive light (2862, 2866, 2870) reflected back from the surgical site 2832 and its surrounding tissue. The Doppler analysis image 2850 may be constructed 2874 based on analysis of multiple pixel data from the CMOS detector 2872.

[0407] In one aspect, a red laser 2854 may emit red laser illumination 2860 at a surgical site 2832, and reflected light 2862 may reveal surface or minimally subsurface structures. In one aspect, a green laser 2856 may emit green laser illumination 2864 at a surgical site 2832, and reflected light 2866 may reveal deeper subsurface features. In another aspect, a blue laser 2858 may emit blue laser illumination 2868 at a surgical site 2832, and reflected light 2870 may reveal blood flow within, for example, deeper vascular structures. Additionally, speckle contrast analysis may present information to the surgeon regarding the flow and velocity of blood through the deeper vascular structures.

[0408] Although not in Fig.35 , but it should be understood that the imaging system can also illuminate the surgical site with light outside the visible range. Such light can include infrared light and ultraviolet light. In some aspects, the source of infrared light or ultraviolet light can include a broadband wavelength source (such as a tungsten source, a tungsten halogen source, or a deuterium source). In some other aspects, the source of infrared light or ultraviolet light can include a narrowband wavelength source (IR diode laser, UV gas laser, or dye laser).

[0409] Fig.36Flowchart 2900 of a method for determining the depth of a surface feature in a piece of tissue. An image acquisition system may illuminate the tissue 2910 with a first light beam having a first center frequency, and receive first reflected light 2912 from the tissue illuminated by the first light beam. The image acquisition system may then calculate a first Doppler shift 2914 based on the first light beam and the first reflected light. The image acquisition system may then illuminate the tissue 2916 with a second light beam having a second center frequency, and receive second reflected light 2918 from the tissue illuminated by the second light beam. The image acquisition system may then calculate a second Doppler shift 2920 based on the second light beam and the second reflected light. The image acquisition system may then calculate a depth 2922 of the tissue feature based at least in part on the first center wavelength, the first Doppler shift, the second center wavelength, and the second Doppler shift. In some aspects, the tissue feature may include the presence of moving particles (such as blood cells moving within a blood vessel), and the flow direction and velocity of the moving particles. It should be understood that the method can be extended to include illuminating the tissue with any one or more additional light beams. Furthermore, the system may calculate an image that includes a combination of an image of the tissue surface and an image of structures disposed within the tissue.

[0410] In some aspects, multiple visual displays may be used. For example, a 3D display may provide a composite image showing a combined white light (or appropriate combination of red laser, green laser, and blue laser) and a laser Doppler image. Additional displays may provide only a white light display or a display showing a composite white light display and a NIRS display to visualize only the blood oxygenation response of the tissue. However, the NIRS display may not be required to allow every cycle of the tissue response.

[0411] Subsurface tissue characterization using multispectral OCT

[0412] During a surgical procedure, a surgeon may employ "smart" surgical devices to manipulate tissue. Such devices may be considered "smart" because they include automated features that direct, control, and / or alter the action of the device based on parameters relevant to its use. These parameters may include the type and / or composition of the tissue being manipulated. If the type and / or composition of the tissue being manipulated is unknown, the action of the smart device may be inappropriate for the tissue being manipulated. As a result, tissue may be damaged or manipulation of the tissue may be inefficient due to inappropriate settings of the smart device.

[0413] The surgeon may manually attempt to change the parameters of the smart device in a trial and error manner, resulting in an inefficient and lengthy surgical procedure.

[0414] It would therefore be desirable to have a surgical visualization system that can probe tissue structures underlying a surgical site to determine their structural and compositional properties and provide such data to intelligent surgical instruments used during the surgical procedure.

[0415] Some aspects of the present disclosure also provide control circuitry configured to control irradiation of a surgical site using one or more irradiation sources, such as a laser source, and to receive imaging data from one or more image sensors. In some aspects, the present disclosure provides a non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause an apparatus to characterize structures below the surface at a surgical site and determine the depth of the structures below the tissue surface.

[0416] In some aspects, a surgical image acquisition system may include: a plurality of illumination sources, wherein each illumination source is configured to emit light having a specified central wavelength; a light sensor, the light sensor configured to receive a portion of the light reflected from the tissue sample when the tissue sample is illuminated by one or more of the plurality of illumination sources; and a computing system. The computing system may be configured to receive data from the light sensor when the tissue sample is illuminated by each of the plurality of illumination sources, calculate structural data related to characteristics of structures within the tissue sample based on the data received by the light sensor when the tissue sample is illuminated by each of the illumination sources, and transmit the structural data related to the characteristics of the structure to be received by the intelligent surgical device. In some aspects, the characteristics of the structure are surface characteristics or structural composition.

[0417] In one aspect, a surgical system may include multiple laser sources and may receive laser light reflected from tissue. The system may use the light reflected from the tissue to calculate surface properties of a component disposed within the tissue. The properties of the component disposed within the tissue may include the composition of the component and / or a metric related to surface unevenness of the component.

[0418] In one aspect, the surgical system may transmit data related to the composition of the component and / or measurements related to the surface unevenness of the component to a second instrument to be used on the tissue to modify control parameters of the second instrument.

[0419] In some aspects, the second device can be an advanced energy device, and the modification of the control parameters can include clamping pressure, operating power level, operating frequency, and transducer signal amplitude.

[0420] As disclosed above, blood vessels may be detected beneath the surface of a surgical site based on the Doppler shift of light reflected by blood cells moving within the blood vessels.

[0421] Laser Doppler flowmetry can be used to visualize and characterize the flow of particles moving relative to an effectively stationary background. Thus, laser light scattered by moving particles (such as blood cells) may have a different wavelength than the original illuminating laser source. In contrast, laser light scattered by an effectively stationary background (e.g., vascular tissue) may have the same wavelength as the original illuminating laser source. Changes in the wavelength of scattered light from blood cells may reflect both the direction of flow of the blood cells relative to the laser source as well as the blood cell velocity. As previously disclosed, Fig.26A -C shows the change in wavelength of light scattered from a blood cell, which may be moving away from ( Fig.26A ) or towards ( Fig.26C )The laser source moves.

[0422] exist Fig.26A In each of the figures in -C, the original illumination light 2502 is depicted as having a relative central wavelength of zero. Fig.26A It can be observed in Figure 25 that light scattered from blood cells moving away from the laser source 2504 has a wavelength that is shifted by a certain amount 2506 to a larger wavelength relative to the laser source (and is therefore red-shifted). Fig.24 C, light scattered from blood cells moving toward laser source 2508 has a wavelength shifted by a certain amount 2510 to a shorter wavelength relative to the laser source (and is therefore blue-shifted). The amount of wavelength shift (e.g., 2506 or 2510) may depend on the speed of movement of the blood cells. In some aspects, the amount of red shift (2506) of some blood cells may be approximately the same as the amount of blue shift (2510) of some other blood cells. Alternatively, the amount of red shift (2506) of some blood cells may be different than the amount of blue shift (2510) of some other blood cells. Thus, based on the relative magnitudes of the wavelength shifts (2506 and 2510), such as Fig.24 The blood cells depicted in A can flow away from the laser source at a speed less than Fig.26C The velocity of blood cells flowing toward the laser source is depicted. Fig.26B As depicted, light scattered from tissue that does not move relative to the laser source (eg, blood vessel 2512 or non-vascular tissue 2514) may not exhibit any wavelength variation.

[0423] previously disclosed, Fig. 27Aspects of an instrument 2530 that can be used to detect a Doppler shift of laser light scattered from a portion of tissue 2540 are depicted. Light 2534 originating from a laser 2532 can pass through a beam splitter 2544. Some portion of the laser light 2536 can be transmitted by the beam splitter 2544 and can illuminate the tissue 2540. Another portion of the laser light can be reflected 2546 by the beam splitter 2544 to be projected on a detector 2550. Light backscattered 2542 by the tissue 2540 can be directed by the beam splitter 2544 and also projected on the detector 2550. The combination of the light 2534 originating from the laser 2532 and the light 2542 backscattered by the tissue 2540 can result in an interference pattern detected by the detector 2550. The interference pattern received by the detector 2550 can include interference fringes resulting from the combination of the light 2534 originating from the laser 2532 and the Doppler shifted (and therefore wavelength shifted) light backscattered 2452 from the tissue 2540.

[0424] It will be appreciated that the backscattered light 2542 from the tissue 2540 may also include backscattered light from boundary layers within the tissue 2540 and / or wavelength-specific absorption of light by materials within the tissue 2540. Therefore, the interference pattern observed at the detector 2550 may incorporate interference fringe features from these additional optical effects and, therefore, may confound the calculation of the Doppler shift unless properly analyzed.

[0425] It will be appreciated that light reflected from tissue may also include backscattered light from boundary layers within the tissue and / or wavelength-specific absorption of light by materials within the tissue. Thus, the interference pattern observed at the detector may incorporate fringe features that, unless properly analyzed, may confound the calculation of the Doppler shift.

[0426] As previously disclosed, Fig.28 Some of these additional optical effects are depicted. As is well known, light passing through a first optical medium having a first refractive index n1 may be reflected at an interface with a second optical medium having a second refractive index n2. Light transmitted through the second optical medium will have a transmission angle relative to the interface that is different from the angle of the incident light based on the difference between the refractive indices n1 and n2 (Snell's law). FIG. 26 illustrates the effect of Snell's law on light projected on the surface of a multi-component tissue 2150, as may be presented in the surgical field. The multi-component tissue 2150 may be composed of an outer tissue layer 2152 having a refractive index n1 and an embedded tissue such as a blood vessel having a vessel wall 2156. The vessel wall 2156 may be characterized by a refractive index n2. Blood may flow within the lumen of the blood vessel 2160. In some aspects, it may be important to determine the location of the blood vessel 2160 below the surface 2154 of the outer tissue layer 2152 during surgery and to characterize the blood flow using Doppler shift techniques.

[0427] Incident laser light 2170a may be used to detect blood vessels 2160 and may be directed onto the top surface 2154 of the outer tissue layer 2152. A portion 2172 of the incident laser light 2170a may be reflected at the top surface 2154. Another portion 2170b of the incident laser light 2170a may penetrate the outer tissue layer 2152. The reflected portion 2172 at the top surface 2154 of the outer tissue layer 2152 has the same path length as the incident light 2170a, and therefore has the same wavelength and phase as the incident light 2170a. However, the portion 2170b of the light transmitted into the outer tissue layer 2152 will have a transmission angle different from the incident angle of the light projected onto the tissue surface because the outer tissue layer 2152 has a refractive index n1 that is different from the refractive index of air.

[0428] If the portion of light transmitted through the outer tissue layer 2152 is projected onto a second tissue surface 2158, such as a vessel wall 2156, some portion 2174a, b of the light will be reflected back toward the incident light 2170a. Thus, the light 2174a reflected at the interface between the outer tissue layer 2152 and the vessel wall 2156 will have the same wavelength as the incident light 2170a, but will be phase shifted due to the change in light path length. Projecting the reflected light 2174a, b from the interface between the outer tissue layer 2152 and the vessel wall 2156 together with the incident light onto the sensor will produce an interference pattern based on the phase difference between the two light sources.

[0429] Additionally, a portion of the incident light 2170c may be transmitted through the vessel wall 2156 and penetrate into the vessel lumen 2160. This portion of the incident light 2170c may interact with the blood cells moving in the vessel lumen 2160 and may be reflected back 2176a-c toward a projected light source having a wavelength Doppler shifted according to the velocity of the blood cells, as disclosed above. The Doppler-shifted light 2176a-c reflected from the moving blood cells may be projected onto the sensor along with the incident light, thereby generating an interference pattern having a fringe pattern based on the wavelength difference between the two light sources.

[0430] exist Fig.28 , if there is no refractive index change between the emitted light and the light reflected by the moving blood cells, an optical path 2178 of light projected onto the red blood cells in the vascular lumen 2160 is presented. In this example, only the Doppler shift of the wavelength of the reflected light can be detected. However, in addition to the wavelength change due to the Doppler effect, the light reflected by the blood cells (2176a-c) can also incorporate phase changes due to changes in the refractive index of the tissue.

[0431] Therefore, it should be understood that if the light sensor receives incident light, light reflected from one or more tissue interfaces (2172 and 2174a, b), and Doppler shifted light from blood cells (2176a-c), the interference pattern thus produced on the light sensor may include effects due to the Doppler shift (wavelength changes) as well as effects due to refractive index changes within the tissue (phase changes). Therefore, if the effects due to refractive index changes within the sample are not compensated, Doppler analysis of light reflected by the tissue sample may produce erroneous results.

[0432] As previously disclosed, Fig.29 An example of the effect of Doppler analysis of light projected 2250 on a tissue sample is shown to determine the depth and location of underlying blood vessels. If there is no intervening tissue between the blood vessel and the tissue surface, the interference pattern detected at the sensor may be primarily due to wavelength changes reflected from moving blood cells. Therefore, the spectrum 2252 derived from the interference pattern may generally reflect only the Doppler shift of the blood cells. However, if there is intervening tissue between the blood vessel and the tissue surface, the interference pattern detected at the sensor may be due to a combination of wavelength changes reflected from moving blood cells and phase shifts due to the refractive index of the intervening tissue. The spectrum 2254 derived from such an interference pattern may result in the calculation of a Doppler shift that is confounded by the additional phase changes in the reflected light. In some aspects, if information about the characteristics of the intervening tissue (thickness and refractive index) is known, the resulting spectrum 2256 may be corrected to provide a more accurate calculation of the wavelength change.

[0433] It will be appreciated that, regardless of the Doppler effect, the phase shift of reflected light from tissue can provide additional information about the underlying tissue structure.

[0434] Fig.37It is shown that the location and characteristics of non-vascular structures can be determined based on the phase difference between the incident light 2372 and the light (2374, 2376, 2378) reflected from the deep tissue structure. As described above, the penetration depth of the light projected onto the tissue depends on the wavelength of the projected illumination. Red laser light (wavelength in the range of about 635nm to about 660nm) can penetrate tissue to a depth of about 1mm. Green laser light (wavelength in the range of about 520nm to about 532nm) can penetrate tissue to a depth of about 2 to 3mm. Blue laser light (wavelength in the range of about 405nm to about 445nm) can penetrate tissue to a depth of about 4mm or more. In one aspect, an interface 2381a with different refractive indices between two tissues located less than or about 1mm below the tissue surface 2380 can reflect 2374 red laser light, green laser light, or blue laser light. The phase of the reflected light 2374 can be compared with the incident light 2372, and thus the difference in the refractive index of the tissue at the interface 2381a can be determined. On the other hand, an interface 2381b with different refractive indices between two tissues located between 2 and 3 mm 2381b below the tissue surface 2380 may reflect 2376 green laser light or blue laser light, but not red laser light. The phase of the reflected light 2376 may be compared with the incident light 2372, and thus the difference in the refractive index of the tissue at the interface 2381b may be determined. On the other hand, an interface 2381c with different refractive indices between two tissues located between 3 and 4 mm 2381c below the tissue surface 2380 may only reflect 2378 blue laser light, but not red or green light. The phase of the reflected light 2378 may be compared with the incident light 2372, and thus the difference in the refractive index of the tissue at the interface 2381c may be determined.

[0435] Therefore, phase interferometry of tissues illuminated by light of different wavelengths can provide information about the relative refractive index of the reflecting tissue and the depth of the tissue. Multiple laser sources and their intensities can be used to evaluate the refractive index of the tissue, so that the relative refractive index of the tissue can be calculated. It has been recognized that different tissues can have different refractive indices. For example, the refractive index can be related to the relative composition of collagen and elastin in the tissue or the amount of hydration of the tissue. Therefore, the technique for measuring the relative tissue refractive index can lead to the identification of the composition of the tissue.

[0436] In some aspects, the intelligent surgical instrument includes an algorithm for determining parameters associated with the function of the instrument. A non-limiting example of such parameters may be the pressure of the anvil against the tissue for the intelligent suturing device. The amount of pressure of the anvil against the tissue may depend on the type and composition of the tissue. For example, less pressure may be required to suturing highly compressed tissue, while a larger amount of pressure may be required to stabilize more non-compressed tissue. Another non-limiting example of parameters associated with the intelligent surgical device may include the rate at which the i-beam knife is fired to cut tissue. For example, rigid tissue may require greater force and slower cutting rate than less rigid tissue. Another non-limiting example of such parameters may be the amount of current provided to the electrodes in the intelligent cauterization or RF sealing device. Tissue composition (such as tissue hydration percentage) may determine the amount of current required for heat sealing tissue. Yet another non-limiting example of such parameters may be the amount of power provided to the ultrasonic transducer of the intelligent ultrasonic cutting device or the driving frequency of the cutting device. Rigid tissue may require more power to cut, and the contact of the ultrasonic cutting tool with the rigid tissue may offset the resonant frequency of the cutter.

[0437] It will be appreciated that a tissue visualization system that can identify tissue type and depth can provide such data to one or more intelligent surgical devices. The intelligent surgical device can then use this identification data and location data to adjust one or more of its operating parameters, allowing it to optimize its manipulation of tissue. It will be appreciated that optical methods of characterizing tissue type can allow for automation of the operating parameters of the intelligent surgical device. Such automation of operation of the intelligent surgical instrument can preferably rely on human estimation to determine the operating parameters of the instrument.

[0438] In one aspect, optical coherence tomography (OCT) is a technique that can visualize subsurface tissue structures based on the phase difference between an illuminating light source and light reflected from structures located within the tissue. Fig.38 An example of an instrument 2470 for optical coherence tomography is schematically depicted. Fig.38, laser source 2472 may emit light 2482 according to any optical wavelength of interest (red, green, blue, infrared, or ultraviolet). Light 2482 may be directed to beam splitter 2486. Beam splitter 2486 directs a portion of light 2488 to tissue sample 2480. Beam splitter 2486 may also direct a portion of light 2492 to stationary reference mirror 2494. Light reflected from tissue sample 2480 and from stationary mirror 2494 may be recombined 2498 at beam splitter 2486 and directed to detector 2496. The phase difference between the light from reference mirror 2494 and from tissue sample 2480 may be detected as an interference pattern at detector 2496. An appropriate computing device may then compute phase information from the interference pattern. Additional computations may then provide information about the structure beneath the surface of the tissue sample. Additional depth information may also be obtained by comparing the interference patterns generated from the sample when illuminated at different wavelengths of laser light.

[0439] As disclosed above, depth information about subsurface tissue structures can be determined based on a combination of laser wavelength and the phase of light reflected from deep tissue structures. Additionally, local tissue surface inhomogeneities can be determined by comparing the phase and amplitude differences of light reflected from different portions of the same subsurface tissue. Measurements of differences in tissue surface characteristics at a defined location compared to tissue surface characteristics at adjacent locations can indicate adhesions in the tissue being probed, tissue degeneration of a tissue layer, infection, or a tumor.

[0440] Fig.39 2544 with substantially the same spread as light 2542 projected onto the surface (specular reflection). Thus, the amount of light received by a light detector having a known fixed aperture may effectively receive the total amount of light 2544 reflected from the smooth surface 2551a. However, increased surface roughness at the tissue surface may result in an increased spread of reflected light relative to incident light (diffuse reflection).

[0441] Due to the increased propagation of reflected light 2546, a certain amount of reflected light 2546 from a tissue surface 2551b having a certain amount of surface unevenness will fall outside the fixed aperture of the light detector. Therefore, the light detector will detect less light (at Fig.39 It will be appreciated that the amount of reflected light propagation will increase as the roughness of the tissue surface increases. Fig.39As depicted, the amplitude of light 2548 reflected from a surface 2551c having significant surface roughness may have a smaller amplitude than light 2544 reflected from a smooth surface 2551a, or light 2546 reflected form a surface 2551b having only a moderate amount of surface roughness. Thus, in some aspects, a single laser source can be used to study the quality of a tissue surface or beneath a surface by comparing the optical properties of reflected light from the tissue with the optical properties of reflected light from an adjacent surface.

[0442] In other aspects, light from multiple laser sources (e.g., lasers emitting light having different central wavelengths) can be used sequentially to detect tissue surface characteristics at various depths below the surface 2550. As disclosed above (ref. Fig.37 ), the absorbance distribution of laser light in tissue depends on the central wavelength of the laser. Laser light with a shorter (bluer) central wavelength can penetrate deeper into tissue than laser light with a longer (redder) central wavelength. Therefore, measurements related to diffuse reflectance of light made at different wavelengths of light can indicate both the amount of surface roughness and the depth of the surface being measured.

[0443] Fig.40 A method of displaying image processing data associated with a combination of tissue visualization modalities is shown. The data used in the display can be derived from image phase data associated with tissue layer composition, image intensity (amplitude) data associated with tissue surface features, and image wavelength data associated with tissue mobility (such as blood cell transport) and tissue depth. As an example, light emitted by a laser in a blue light region 2562 can be projected onto blood flowing at a depth of approximately 4 mm below the tissue surface. The reflected light 2564 can be red-shifted due to the Doppler effect of the blood flow. Therefore, information about the presence of blood vessels and their depth below the surface can be obtained.

[0444] In another example, the tissue layer may be located at a depth of about 2 to 3 mm below the surface of the surgical site. The tissue may include surface unevenness that indicates scarring or other pathology. The emitted red light 2572 may not penetrate to a depth of 2 to 3 mm, and therefore, the reflected red light 2580 may have approximately the same amplitude as the emitted red light 2572 because it cannot detect structures more than 1 mm below the top surface of the surgical site. However, the green light reflected from the tissue 2578 may show that there are surface unevenness at this depth because the amplitude of the reflected green light 2578 may be less than the amplitude of the emitted green light 2570. Similarly, the blue light reflected from the tissue 2574 may show that there are surface unevenness at this depth because the amplitude of the reflected blue light 2574 may be less than the amplitude of the emitted blue light 2562. In one example of an image processing step, a moving window filter 2584 may be used to smooth the image 2582 to reduce inter-pixel noise and reduce small local tissue abnormalities 2586 that may hide more important features 2588.

[0445] Fig.41A -C illustrates several aspects of a display that may be provided to a surgeon for visually identifying surface and subsurface structures of tissue in a surgical site. Fig.41A A surface map of the surgical site may be presented with color coding to indicate structures located at different depths below the surface of the surgical site. Fig.41B Depicted is an example of one of several horizontal slices through tissue at different depths, which may be color coded to indicate depth, and also include data associated with differences in tissue surface abnormalities (eg, as shown in a 3D histogram). Fig.41C Yet another visual display is depicted in which surface roughness and Doppler shift flow cytometry data can indicate subsurface vascular structure and tissue surface properties.

[0446] Fig.42 Flowchart 2950 of a method for providing information related to the characteristics of tissue to an intelligent surgical instrument. An image acquisition system may illuminate tissue 2960 with a first light beam having a first center frequency, and receive first reflected light 2962 from the tissue illuminated by the first light beam. Then, the image acquisition system may calculate first tissue surface characteristics 2964 at a first depth based on the first emitted light beam and the first reflected light from the tissue. Then, the image acquisition system may illuminate tissue 2966 with a second light beam having a second center frequency, and receive second reflected light 2968 from the tissue illuminated by the second light beam. Then, the image acquisition system may calculate second tissue surface characteristics 2970 at a second depth based on the second emitted light beam and the second reflected light from the tissue. Tissue features that may include tissue type, tissue composition, and tissue surface roughness metrics may be determined by the first center light frequency, the second center light frequency, the first reflected light from the tissue, and the second reflected light from the tissue. Tissue characteristics may be used to calculate one or more parameters 2972 ​​related to the function of the intelligent surgical instrument, such as jaw pressure, power to achieve tissue cauterization, or current amplitude and / or frequency to drive a piezoelectric actuator to cut tissue. In some additional examples, the parameters may be transmitted directly or indirectly to the smart surgical instrument 2974, which may modify its operating characteristics in response to the tissue being manipulated.

[0447] Multi-focus minimally invasive camera

[0448] In minimally invasive surgery, such as laparoscopic surgery, a surgeon may use an imaging instrument including a light source and a camera to visualize the surgical site. The imaging instrument may allow the surgeon to visualize the end effector of the surgical device during surgery. However, the surgeon may need to visualize tissue away from the end effector to prevent unintended damage during surgery. Such distant tissue may be outside the field of view of the camera system when focusing on the end effector. The imaging instrument may be moved in order to change the field of view of the camera, but it may be difficult to return the camera system to its initial position after the move.

[0449] The surgeon may attempt to move the imaging system within the surgical site to visualize different portions of the site during surgery. Repositioning of the imaging system is time consuming, and the surgeon cannot guarantee visualization of the same field of view of the surgical site when the imaging system is returned to its original position.

[0450] Therefore, it is desirable to have a medical imaging visualization system that can provide multiple fields of view of a surgical site without repositioning the visualization system. Medical imaging devices include, but are not limited to, laparoscopes, endoscopes, thoracoscopes, etc., as described herein. In some aspects, a single display system can display each of the multiple fields of view of a surgical site substantially simultaneously. The display of each of the multiple fields of view can be independently updated according to a display control system comprised of one or more hardware modules, one or more software modules, one or more firmware modules, or any one or more combinations thereof.

[0451] Some aspects of the present disclosure also provide control circuitry configured to control irradiation of a surgical site using one or more irradiation sources, such as a laser source, and to receive imaging data from one or more image sensors. In some aspects, the control circuitry may be configured to control the operation of one or more light sensor modules to adjust the field of view. In some aspects, the present disclosure provides a non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a device to adjust one or more components of the one or more light sensor modules and process images from each of the one or more light sensor modules.

[0452] One aspect of a minimally invasive image acquisition system may include: a plurality of illumination sources, each of which is configured to emit light having a specified central wavelength; a first light sensing element having a first field of view and configured to receive illumination reflected from a first portion of a surgical site when the first portion of the surgical site is illuminated by at least one of the plurality of illumination sources; a second light sensing element having a second field of view and configured to receive illumination reflected from a second portion of the surgical site when the second portion of the surgical site is illuminated by at least one of the plurality of illumination sources, wherein the second field of view overlaps with at least a portion of the first field of view; and a computing system.

[0453] The computing system may be configured to receive data from the first light sensing element, receive data from the second light sensing element, compute imaging data based on the data received from the first light sensing element and the data received from the second light sensing element, and transmit the imaging data for receipt by a display system.

[0454] A variety of surgical visualization systems have been disclosed above. Such systems provide for visualization of tissue and subtissue structures that may be encountered during one or more surgical procedures. Non-limiting examples of such systems may include: systems for determining the location and depth of subsurface vascular tissue such as veins and arteries; systems for determining the amount of blood flowing through subsurface vascular tissue; systems for determining the depth of non-vascular tissue structures; systems for characterizing the composition of such non-vascular tissue structures; and systems for characterizing one or more surface characteristics of such tissue structures.

[0455] It will be appreciated that a single surgical visualization system may incorporate components of any one or more of these visualization modalities. Fig.22A -D depicts some examples of such surgical visualization systems 2108.

[0456] As disclosed above, in one non-limiting aspect, the surgical visualizatio...

Claims

1. A minimally invasive image acquisition system, comprising: a plurality of illumination sources, wherein each illumination source is configured to emit light having a specified central wavelength; an elongated camera probe having a distal end; a first light sensing element having a first field of view and configured to receive illumination reflected from a first portion of a surgical site when the first portion of the surgical site is illuminated by at least one of the plurality of illumination sources, wherein the first light sensing element is disposed within a distal end of the elongated camera probe; a second light sensing element having a second field of view and configured to receive illumination reflected from a second portion of the surgical site when the second portion of the surgical site is illuminated by at least one of the plurality of illumination sources, wherein the second field of view overlaps at least a portion of the first field of view, wherein the first field of view is wider than the second field of view and the second field of view is disposed entirely within the first field of view, wherein the second light sensing element is disposed within a distal end of the elongated camera probe; as well as A computing system, wherein the computing system is configured to: receiving data from the first light sensing element, receiving data from the second light sensing element, calculating imaging data based on the data received from the first light sensing element and the data received from the second light sensing element, and The imaging data is transmitted for receipt by a display system. 2 . The minimally invasive image acquisition system according to claim 1 , wherein the first light sensing element has an optical component configured to adjust the first field of view. 3 . The minimally invasive image acquisition system according to claim 1 , wherein the second light sensing element has an optical component configured to adjust the second field of view. 4 . The minimally invasive image acquisition system of claim 1 , wherein each of the plurality of illumination sources is configured to emit light having a specified central wavelength, the specified central wavelength being within the visible spectrum. 5 . The minimally invasive image acquisition system of claim 1 , wherein at least one of the plurality of illumination sources is configured to emit light having a specified central wavelength, the specified central wavelength being outside the visible spectrum.

6. The minimally invasive image acquisition system of claim 5, wherein the designated central wavelength outside the visible spectrum is in the ultraviolet range.

7. The minimally invasive image acquisition system of claim 5, wherein the designated central wavelength outside the visible spectrum is in the infrared range.

8. A minimally invasive image acquisition system according to claim 1, wherein the computing system configured to calculate imaging data based on the data received from the first light sensing element and the data received from the second light sensing element includes a computing system configured to perform a first data analysis on the data received from the first light sensing element and to perform a second data analysis on the data received from the second light sensing element.

9. The minimally invasive image acquisition system of claim 8, wherein the first data analysis is different from the second data analysis.

10. The minimally invasive image acquisition system according to claim 1, wherein the computing system comprises: processor; as well as a memory coupled to the processor, the memory storing instructions executable by the processor to: controlling the operation of the plurality of illumination sources, receiving, from the first light sensing element, first data related to illumination reflected from a first portion of the surgical site when the first portion of the surgical site is illuminated by at least one illumination source of the plurality of illumination sources, receiving, from the second light sensing element, second data related to illumination reflected from a second portion of the surgical site when the second portion of the surgical site is illuminated by at least one of the plurality of illumination sources, calculating the imaging data based on the first data received from the first light sensing element and the second data received from the second light sensing element, and The imaging data is transmitted for receipt by the display system.

11. The minimally invasive image acquisition system according to claim 1, wherein the computing system comprises: The control circuit is further configured to: The operation of the plurality of illumination sources is controlled.

12. A non-transitory computer readable medium storing computer readable instructions which, when executed, cause a machine to: controlling operation of a plurality of illumination sources of the tissue sample, wherein each illumination source is configured to emit light having a specified central wavelength, receiving, from a first light sensing element disposed at a distal end of the elongated camera probe, first data related to illumination reflected from a first portion of a surgical site when the first portion of the surgical site is illuminated by at least one of the plurality of illumination sources, receiving second data related to illumination reflected from a second portion of the surgical site from a second light sensing element disposed at a distal end of the elongated camera probe when the second portion of the surgical site is illuminated by at least one of the plurality of illumination sources, wherein the second field of view overlaps at least a portion of the first field of view, wherein The first field of view is wider than the second field of view, and the second field of view is completely disposed within the first field of view, calculating imaging data based on the first data received from the first light sensing element and the second data received from the second light sensing element, and The imaging data is transmitted for receipt by a display system.

Images