Method for operating hierarchical operating modes in surgical systems

Abstract

The surgical system may include tiered access capabilities. The surgical system may be used to analyze at least a portion of a surgical field. Based on control parameters, the system may scale up or down various capabilities, such as visualization processing, endocutter transmission, endocutter algorithm updates, smart cartridge connectivity, smart motor control for the circular stapler, smart energy control, cloud analytics, hub connectivity control, and / or hub visualization and control interaction. The control parameters may include system aspects such as, for example, processing power or bandwidth and / or identification of an appropriate service tier.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application relates to the following, the contents of which are incorporated herein by reference: ● U.S. Patent Application No. 17 / 062,521 (Agent Reference Number END9287USNP2), filed together with this application, entitled "Tiered-Access Surgical Visualization System" ● U.S. Patent Application No. 17 / 062,522 (Agent Reference Number END9287USNP3), filed together with this application, entitled "Surgical Visualization and Particle Trend Analysis System" ● U.S. Patent Application No. 17 / 062,523 (Agent Reference Number END9287USNP4), filed together with this application, entitled "Field Programmable Surgical Visualization System" ● U.S. Patent Application No. 17 / 062,511 (Agent Reference Number END9287USNP5), filed together with this application, entitled "Communication Capability of a Smart Stapler" ● U.S. Patent Application No. 17 / 062,524 (Agent Reference Number END9287USNP6), titled "End Effector Updates," filed together with this application. ● U.S. Patent Application No. 17 / 062,501 (Agent Reference Number END9287USNP7), filed together with this application, entitled "Communication Capability of a Surgical Device with Component" ● U.S. Patent Application No. 17 / 062,502 (Agent Reference Number END9287USNP8), filed together with this application, entitled "Surgical Instrument with Adaptive Function Controls" ● U.S. Patent Application No. 17 / 062,499 (Agent Reference Number END9287USNP9), filed together with this application, entitled "Surgical Instrument with Adaptive Motor Control" ● U.S. Patent Application No. 17 / 062,496 (Agent Reference Number END9287USNP10), filed together with this application, entitled "Surgical Instrument with Adaptive Configuration Control" ● U.S. Patent Application No. 17 / 062,525 (Agent Reference Number END9287USNP11), filed together with this application, entitled "Smart Energy Combo Control Options" ● U.S. Patent Application No. 17 / 062,526 (Agent Reference Number END9287USNP12), filed together with this application, entitled "Cloud Analytics Packages" ● U.S. Patent Application No. 17 / 062,530 (Agent Reference Number END9287USNP13), filed together with this application, entitled "Surgical Hub Having Variable Interconnectivity Capabilities" ● U.S. Patent Application No. 17 / 062,512 (Agent Reference Number END9287USNP14), entitled "Tiered System Display Control Based on Capacity and User Operation" ● U.S. Patent Application No. 17 / 062,508, entitled "Cooperative Surgical Displays" (Agent Reference Number END9287USNP15), ● U.S. Patent Application No. 17 / 062,509, entitled "Interactive Information Overlay on Multiple Surgical Displays" (Agent Reference Number END9287USNP16), ● U.S. Patent Application No. 17 / 062,507, entitled "Communication Control for A Surgeon-Controlled Secondary Display and Primary Display" (Agent Reference Number END9287USNP17), ● U.S. Patent Application No. 17 / 062,513 (Agent Reference Number END9288USNP1), filed together with this application, entitled "Situational Awareness of Instruments Location and Individualization of Users to Control Displays" ● U.S. Patent Application No. 17 / 062,517 (Agent Reference Number END9288USNP2), filed together with this application, entitled "Shared Situational Awareness of The Device Actuator Activity to Prioritize Certain Aspects of Displayed Information," ● U.S. Patent Application No. 17 / 062,520 (Agent Reference Number END9288USNP3), filed together with this application, entitled "Monitoring of User Visual Gaze to Control Which Display System Displays the Primary Information". ● U.S. Patent Application No. 17 / 062,519 (Agent Reference Number END9288USNP4), filed together with this application, entitled "Reconfiguration of Display Sharing," and ● U.S. Patent Application No. 17 / 062,516 (Agent Reference Number END9288USNP5), filed together with this application, entitled "Control A Display Outside the Sterile Field from A Device Within the Sterile Field". [Background technology]

[0002] Surgical systems often incorporate imaging systems that allow clinicians(s) to view the surgical site and / or one or more parts thereof on one or more displays, such as monitors. These displays may be localized to the surgical field and / or remote from the surgical field. The imaging system may include a scope equipped with a camera that views the surgical site and transmits the view to a display accessible to the clinician. Examples of scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, and exoscopy. The imaging system may be limited by the information that can be recognized by and / or communicated to the clinician(s). For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognizable during surgery by certain imaging systems. Additionally, certain imaging systems may be unable to communicate and / or transmit certain information to the clinician(s) during surgery. [Overview of the project] [Means for solving the problem]

[0003] According to one embodiment of the present disclosure, the surgical system may include hierarchical access functionality. The surgical system can be used to analyze at least a portion of the surgical field. Based on control parameters, the system can scale up or down various capabilities such as visualization processing, endocutter transmission, endocutter algorithm updates, smart cartridge connectivity, smart motor control for circular staplers, smart energy control, cloud analysis, hub connectivity control, and / or hub visualization and control interactions. The control parameters may include system aspects such as processing power or bandwidth, and / or identification of appropriate service layers.

[0004] For example, a surgical system may operate in a first operating mode, receive control parameters, and then decide to operate in a second operating mode based on the received control parameters.

[0005] For example, a surgical system may receive control parameters. Based on these control parameters, the surgical system may operate in a first or second operating mode. For example, the control parameters may be, or include, parameters indicating power capacity, memory capacity, bandwidth capacity, and processing suitability. For example, the control parameters may be, or include, parameters indicating processing suitability, where processing suitability indicates a purchased functional layer associated with either the user or the instrument.

[0006] According to various embodiments of the present invention, the following examples are provided. 1. A method for controlling a digital aspect of a surgical system, the method comprising: operating in a first operating mode; receiving control parameters; and determining to operate in a second operating mode based on the received control parameters.

[0007] The above-described embodiment 1 enables adaptive control of the digital aspects of a surgical system, such as visualization and operational functions. Adaptive control may follow control parameters that can be further tailored to specific procedures, tissue types, surgical instruments or their components, stages / states of operation, users, or clinicians. The control parameters may include one or more parameters such as power capacity, memory capacity, bandwidth capacity, processing suitability, functional layers associated with either the user or the instrument, and / or parameters associated with situational awareness. Overall clinical outcomes, surgical efficiency, and / or patient safety can be improved. 2. A method for analyzing at least a portion of a surgical field, the method comprising: receiving control parameters; and operating based on the control parameters in at least one of a first or second operating mode, wherein operating in the first operating mode comprises determining a first criterion representing the current state of moving particles in at least a portion of the surgical field; and operating in the second operating mode comprises determining the first and second criterions, wherein the second criterion represents either the aggregation state of moving particles in at least a portion of the surgical field or the current state of moving particles in at least a portion of the surgical field at a selectable tissue depth. 3. The method according to Example 2, wherein the control parameters include a parameter indicating one of the following: power capacity, memory capacity, bandwidth capacity, and processing suitability. 4. The method according to Example 2, wherein the control parameters include parameters indicating processing suitability, and the processing suitability indicates a functional layer associated with either the user or the device. 5. The method according to any one of Examples 2 to 4, further comprising displaying the first and second measurement criteria as overlays on an image including at least a portion of the surgical field. 6. The method according to any one of Examples 2 to 5, further comprising operating in a second operating mode by receiving situational awareness information and reducing noise based on the situational awareness information.

[0008] The above embodiments 2-6 allow for the dynamic analysis and visualization of at least a portion of the surgical field to provide information indicating the movement of particles in the surgical field that is most relevant or appropriate to a particular procedure, clinical situation, or clinical setting. Based on control parameters, a second operating mode can be implemented, which may be further tailored to a particular procedure, tissue type, or according to the preferred practice of a particular user or clinician. Movement trends may be displayed by graphs and / or metrics that can be overlaid on real-time data. Such trend information may be used to identify occlusion, the vascular sealing / clamping efficiency of instruments, an overview of the vascular tree, and even the magnitude of the amplitude of movement over time. Such real-time aggregation and / or updatable functions of the system may be selectively enabled based on any aspect of the system configuration, e.g., system capacity, power availability, free memory access, transmission capacity, software level, hierarchical function level, etc. This helps ensure that the user or clinician must hand over the most relevant information from the surgical field for the clinical situation, thereby improving patient safety. As demonstrated in Examples 2-6 above, users or clinicians can be presented with the most relevant information and therefore act quickly using relevant information from the surgical field. This can improve clinical outcomes. Functional layers can allow the device or system to be more tailored to specific procedures, tissue types, or surgeon preferences. By associating this layer with users, specific functions appropriate for a particular procedure can be provided only to users who require such functions or who are authorized to perform such procedures. By associating this layer with specific instruments, functions appropriate for a particular procedure or clinical setting can be provided to instruments designed for such procedures or settings. As a result, the safety of procedures performed using the system can be improved. Furthermore, because the system's functions are tailored to specific settings or users, the duration of procedures can be reduced, and the burden on the user to select the appropriate function during a procedure is lessened. 7. A method for controlling transmission in a surgical stapler, comprising: determining that the surgical stapler determines one or more parameters associated with at least one of the surgical stapler or a staple cartridge, wherein the staple cartridge is transmissibly connected to the surgical stapler; and determining a transmission capacity between the surgical stapler and the staple cartridge based on one or more parameters associated with at least one of the surgical stapler or a staple cartridge. 8. The method according to Example 7, wherein determining the transmission capacity between the surgical stapler and the staple cartridge includes determining one or more of the following: whether data is transmitted between the surgical stapler and the staple cartridge, whether the transmission between the surgical stapler and the staple cartridge is unidirectional or bidirectional, or whether power is transmitted from the surgical stapler to the staple cartridge. 9. Determining the transmission capacity between a surgical stapler and a staple cartridge based on one or more parameters associated with at least one of a surgical stapler or a staple cartridge, the method according to Example 7, further comprising determining the transmission capacity between a surgical stapler and a staple cartridge based on one or more of environmental parameters, interference, system capabilities, or system control parameters. 10. The method according to Example 9, wherein the system control parameters include one or more of the following: software level, software revision, software reliability, or purchase level. 11. The method according to Example 10, further comprising a surgical stapler receiving data associated with a staple cartridge from a staple cartridge. 12. Receiving data associated with a staple cartridge includes receiving data associated with one or more of a serial number associated with the staple cartridge, a color associated with the staple cartridge, a length associated with the staple cartridge, or a state associated with the staple cartridge, the method of Example 11. 13. A surgical stapler includes a transmission array, a staple cartridge includes an RFID chip, and receiving, by the surgical stapler, data associated with the staple cartridge from the staple cartridge includes receiving data transmitted by the RFID chip in the transmission array, the method of Example 11 or 12. 14. Determining a transmission capacity between a surgical stapler and a staple cartridge includes determining data regarding discrete data read values that can be transmitted by the staple cartridge to the surgical stapler, the method of any one of Examples 7-13. 15. The method of Example 14, further including the surgical stapler receiving discrete data read values from the staple cartridge.

[0009] According to Examples 7-15, the transmission capacity between a surgical stapler and a staple cartridge connected communicably to each other is determined and used to further control the transmission of data and / or power between the surgical stapler and the staple cartridge to achieve proper use of the staple cartridge. This transmission capacity may be determined from a selective list of transmission control parameters that may further correspond to a selective transmission layer. This enables, for example, an appropriate transmission mode to be established between the stapler and the cartridge according to parameters or data associated with the stapler, parameters or data / read values associated with the staple cartridge, the operator of the stapler, the owner of the stapler, or other relevant elements. Thus, clinical outcomes can be improved. 16. A method performed by a surgical hub operably connected to at least one display, a plurality of augmented reality (AR) devices, and at least one surgical instrument, the method comprising: obtaining AR control parameters; obtaining, from one of the at least one surgical instruments, a first data stream for display on a first display of the at least one displays; determining, based on the AR control parameters, first AR content for overlaying on the first data stream displayed on the first display via a first AR device of the plurality of AR devices; determining, based on the AR control parameters, second AR content for overlaying on the first data stream displayed on the first display via a second AR device of the plurality of AR devices, the second AR content being different from the first AR content; and transmitting the first AR content and the second AR content to the first AR device and the second AR device, respectively. 17. The AR control parameters include the role of the user, and the method further comprises: identifying the role of a first user associated with the first AR device; determining, based on the role of the first user, a first overlay data set for inclusion in the first AR content; identifying the role of a second user associated with the second AR device; and determining, based on the role of the second user, a second overlay data set for inclusion in the second AR content, the second overlay data set being different from the first overlay data set. 18. The method according to embodiment 16 or 17 further comprising instructing the first AR device to detect a user of the first AR device looking at the first display and, in response to the detection, overlay the first overlay data on the content displayed on the first display via the first AR device. 19. The method according to any one of Examples 16 to 18, further comprising receiving instructions for an intended action on a target area, obtaining a prediction result associated with performing the intended action on the target area, and including the prediction result in a first AR content. 20. The method according to Example 19, further comprising acquiring visual data from a surgical scope, acquiring sensor input data from at least one surgical instrument, transmitting the visual data and sensor input data to a remote server, and receiving prediction results from the remote server.

[0010] Through the above embodiments 16-20, the surgical hub enables the same screen to display different content to different members of the surgical team via AR devices. Specifically, the AR content is overlaid on the display for the user of the AR device, and this AR content is not visible to members of the surgical team who are not using that AR device. By using multiple AR devices displaying different AR content, different AR content can be overlaid on the same screen for different users. By using one or more AR devices in this way, it becomes possible to directly transmit information to members of the surgical team who need it, without distracting other members of the team who do not need it. Faster access to this information and less distraction leads to greater efficiency, improved patient safety, and improved surgical outcomes. Directly overlaying this information on the first display using AR devices leads to further improvements, as the first data stream and the AR content are displayed in the same location. This allows both data sources to be viewed together, and therefore more quickly, and enables the AR content to interact with the first data stream, such as labeling items shown in the first data stream or highlighting aspects of its content. This allows coordinated information regarding the progress of a surgical procedure to be transmitted to the surgical team. AR content can assist the surgical team in performing specific steps of a surgical procedure, ensuring correct device setup and use, guaranteeing that the procedure is working as expected, and detecting and correcting any abnormalities or exceptions in the surgical procedure. By displaying predicted outcomes of specific surgical actions as part of the AR content, the surgical team can be supported in the surgical procedure being performed, the surgical action can be performed to higher standards of accuracy and safety, potentially fatal errors can be avoided or minimized, and surgical outcomes can be significantly improved.The use of AR control parameters such as surgical progress, user orientation, and surgical situation allows AR content to be updated in real time, helping to explain events specific to the current surgical procedure, such as any anomalies. Therefore, overall, the displayed AR content further improves efficiency, patient safety, and surgical outcomes. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram of a computer-implemented interactive surgical system. [Figure 2] This shows an exemplary surgical system used to perform surgical procedures in an operating room. [Figure 3] This disclosure illustrates, in at least one aspect, a visualization system, a robotic system, and an exemplary surgical hub paired with an intelligent instrument. [Figure 4] The present disclosure, in at least one aspect, describes a surgical data network having a communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] An exemplary computer-implemented interactive surgical system is shown. [Figure 6] An exemplary surgical hub is shown, including multiple modules connected to a modular control tower. [Figure 7] An example of a surgical instrument or tool is shown. [Figure 8] This exhibits an exemplary surgical instrument or tool having a motor that can be activated to perform various functions. [Figure 9] This is a diagram illustrating an exemplary situational awareness surgical system. [Figure 10] This shows an exemplary timeline of a surgical procedure and reasoning that can be created by a surgical hub from data detected at each step of the surgical procedure. [Figure 11]This is a block diagram of a computer-implemented interactive surgical system. [Figure 12] This illustrates the functional architecture of an exemplary computer-implemented interactive surgical system. [Figure 13] This exhibits an exemplary computer-implemented interactive surgical system configured to adaptively generate control program updates for modular devices. [Figure 14] An exemplary surgical system is shown, including a handle having a controller and motor, an adapter releasably connected to the handle, and a loading unit releasably connected to the adapter. [Figure 15A] This section describes an exemplary flow for determining the operating mode and for operating in the determined mode. [Figure 15B] This shows an exemplary flow for changing the operating mode. [Figure 16A] An illustrative visualization system is shown. [Figure 16B] An illustrative visualization system is shown. [Figure 16C] An illustrative visualization system is shown. [Figure 16D] An illustrative visualization system is shown. [Figure 17A] This document illustrates the illumination of an image sensor with multiple laser emitters and a Bayer pattern of a color filter, which may be incorporated into an exemplary visualization system, a graphical representation of the operation of the pixel array for multiple frames, a schematic diagram of an example of the operation sequence of the chromaticity frame and luminance frame, an example of the sensor pattern and emitter pattern, and a graphical representation of the operation of the pixel array. [Figure 17B] This document illustrates the illumination of an image sensor with multiple laser emitters and a Bayer pattern of a color filter, which may be incorporated into an exemplary visualization system, a graphical representation of the operation of the pixel array for multiple frames, a schematic diagram of an example of the operation sequence of the chromaticity frame and luminance frame, an example of the sensor pattern and emitter pattern, and a graphical representation of the operation of the pixel array. [Figure 17C]This document illustrates the illumination of an image sensor with multiple laser emitters and a Bayer pattern of a color filter, which may be incorporated into an exemplary visualization system, a graphical representation of the operation of the pixel array for multiple frames, a schematic diagram of an example of the operation sequence of the chromaticity frame and luminance frame, an example of the sensor pattern and emitter pattern, and a graphical representation of the operation of the pixel array. [Figure 17D] This document illustrates the illumination of an image sensor with multiple laser emitters and a Bayer pattern of a color filter, which may be incorporated into an exemplary visualization system, a graphical representation of the operation of the pixel array for multiple frames, a schematic diagram of an example of the operation sequence of the chromaticity frame and luminance frame, an example of the sensor pattern and emitter pattern, and a graphical representation of the operation of the pixel array. [Figure 17E] This document illustrates the illumination of an image sensor with multiple laser emitters and a Bayer pattern of a color filter, which may be incorporated into an exemplary visualization system, a graphical representation of the operation of the pixel array for multiple frames, a schematic diagram of an example of the operation sequence of the chromaticity frame and luminance frame, an example of the sensor pattern and emitter pattern, and a graphical representation of the operation of the pixel array. [Figure 17F] This document illustrates the illumination of an image sensor with multiple laser emitters and a Bayer pattern of a color filter, which may be incorporated into an exemplary visualization system, a graphical representation of the operation of the pixel array for multiple frames, a schematic diagram of an example of the operation sequence of the chromaticity frame and luminance frame, an example of the sensor pattern and emitter pattern, and a graphical representation of the operation of the pixel array. [Figure 18] This shows exemplary equipment for NIR spectroscopy. [Figure 19] This document describes exemplary equipment for determining NIRS based on Fourier transform infrared imaging. [Figure 20A] This shows the change in the wavelength of light scattered from migrating blood cells. [Figure 20B] This shows the change in the wavelength of light scattered from migrating blood cells. [Figure 20C] This shows the change in the wavelength of light scattered from migrating blood cells. [Figure 21]Exemplary instruments that may be used to detect the Doppler shift of laser light scattered from a portion of tissue are shown. [Figure 22] This shows an exemplary optical effect on light striking a tissue with a subsurface structure. [Figure 23] This shows an exemplary optical effect on light striking a tissue with a subsurface structure. [Figure 24A] Based on laser Doppler analysis at various laser wavelengths, we demonstrate the detection of migrating blood cells at tissue depth. [Figure 24B] Based on laser Doppler analysis at various laser wavelengths, we demonstrate the detection of migrating blood cells at tissue depth. [Figure 24C] Based on laser Doppler analysis at various laser wavelengths, we demonstrate the detection of migrating blood cells at tissue depth. [Figure 24D] Based on laser Doppler analysis at various laser wavelengths, we demonstrate the detection of migrating blood cells at tissue depth. [Figure 25] This shows an example of using Doppler imaging to detect the presence of subsurface blood vessels. [Figure 26] This shows the Doppler shift of blue light caused by blood cells flowing in subsurface blood vessels. [Figure 27] This shows an example of the location of deep subsurface blood vessels. [Figure 28] This shows an example of the location of superficial subsurface blood vessels. [Figure 29] An exemplary composite image is shown, including images of the surface and subsurface vessels. [Figure 30] An exemplary method for determining the depth of surface features within a tissue sample is provided. [Figure 31A] An exemplary visualization system is shown. [Figure 31B] An exemplary laser light sensor with two sensor modules is shown. [Figure 31C] This is a graphical representation of the exemplary behavior of pixel arrays across multiple frames. [Figure 32] An exemplary method for determining the operating mode is shown. [Figure 33] This document provides an example method for displaying real-time and trend information to users. [Figure 34] An exemplary user interface is shown, displaying real-time and / or trend information. [Figure 35] An example upgrade framework is shown. [Figure 36] This document describes an exemplary method for reconfiguring a field-programmable gate array. [Figure 37] This is a perspective view of a powered surgical stapling system. [Figure 38] Figure 37 is a perspective view of an interchangeable surgical shaft assembly for a powered surgical stapling system. [Figure 39] Figure 37 is an exploded view of a portion of the handle assembly of a powered surgical stapling system. [Figure 40] Figure 37 is an exploded view of the interchangeable surgical shaft assembly. [Figure 41] Figure 37 is another partially exploded view of a portion of the interchangeable surgical shaft assembly. [Figure 42] A perspective view of another powered surgical stapling system. [Figure 43] This illustrates the operating modes of an exemplary surgical instrument. [Figure 44] This illustrates the operating modes of an exemplary surgical instrument. [Figure 45] This illustrates the operating modes of an exemplary surgical instrument. [Figure 46] This is an exemplary analysis system diagram for updating a control program for a surgical instrument, according to at least one aspect of the present disclosure. [Figure 47] This is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 48] This shows an exemplary flow chart for operating a surgical instrument according to its operating mode(s). [Figure 49]This is a perspective view of a surgical instrument having an operably connected interchangeable shaft assembly according to at least one aspect of the present disclosure. [Figure 50] This is an exploded assembly view of a portion of the surgical instrument shown in Figure 49, according to at least one aspect of the present disclosure. [Figure 51] This is an exploded view of a portion of an interchangeable shaft assembly according to at least one aspect of the present disclosure. [Figure 52] Figure 49 is an exploded view of the end effector of a surgical instrument according to at least one aspect of the present disclosure. [Figure 53A] This is a block diagram of the control circuit of the surgical instrument of Figure 49, spanning two drawings, according to at least one aspect of the present disclosure. [Figure 53B] This is a block diagram of the control circuit of the surgical instrument of Figure 49, spanning two drawings, according to at least one aspect of the present disclosure. [Figure 54] Figure 49 is a block diagram of the control circuit of a surgical instrument, showing an interface between a handle assembly and a power supply assembly, and an interface between a handle assembly and a replaceable shaft assembly, according to at least one aspect of the present disclosure. [Figure 55] An exemplary medical device is shown, which may include one or more aspects of this disclosure. [Figure 56] One or more aspects of this disclosure illustrate exemplary end effectors of medical devices surrounding tissue. [Figure 57] The present disclosure illustrates an exemplary end effector for a medical device that compresses tissue, according to one or more aspects of this disclosure. [Figure 58] This disclosure illustrates the exemplary forces exerted by an end-effector of a medical device that compresses tissue, as described in one or more aspects of this disclosure. [Figure 59] Similarly, the present disclosure illustrates the exemplary forces exerted by the end effector of a medical device that compresses tissue, as described in one or more aspects of this disclosure. [Figure 60] An exemplary tissue compression sensor system according to one or more aspects of this disclosure is shown. [Figure 61] Similarly, an exemplary tissue compression sensor system according to one or more aspects of the present disclosure is shown. [Figure 62] Similarly, an exemplary tissue compression sensor system according to one or more aspects of the present disclosure is shown. [Figure 63] These are exemplary circuit diagrams according to one or more aspects of the present disclosure. [Figure 64] Similarly, here is an exemplary circuit diagram according to one or more aspects of the present disclosure. [Figure 65] This is a diagram of a position sensor including a magnetic rotation absolute positioning system according to at least one aspect of the present disclosure. [Figure 66] This is a cross-sectional view of an end effector of a surgical instrument, showing the stroke of a firing member against tissue held by the end effector, according to at least one aspect of the present disclosure. [Figure 67] A block diagram of a surgical system configured to control surgical functions, according to at least one aspect of this disclosure, is shown. [Figure 68] A block diagram of a situation-aware surgical system configured to control surgical functions, according to at least one aspect of this disclosure, is shown. [Figure 69] This is a logical flowchart illustrating a situation-aware, situation-aware algorithm for controlling surgical function, according to at least one aspect of the present disclosure. [Figure 70] A logical flowchart of a process for controlling a surgical instrument according to the physiological type of clamped tissue, according to at least one aspect of this disclosure, is shown. [Figure 71] This is a logic flow diagram illustrating a process for a control program or logic configuration for adjusting a closing speed algorithm, according to at least one aspect of the present disclosure. [Figure 72] A logical flowchart of a process showing a control program for a logical configuration to identify irregularities in the tissue distribution within the end effector of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 73]A logical flow diagram of a process representing a control program or logical configuration for properly positioning pre-stapled tissue within an end effector, according to at least one aspect of this disclosure, is shown. [Figure 74] A logical flow diagram of a process for updating a control program for a modular device, according to at least one aspect of this disclosure, is shown. [Figure 75] A diagram of an analysis system that pushes updates to modular devices through a surgical hub, according to at least one aspect of this disclosure, is shown. [Figure 76] A diagram of a computer-implemented interactive surgical system configured to adaptively generate control program updates for a surgical hub, according to at least one aspect of the present disclosure, is shown. [Figure 77] A logical flow diagram of a process for updating the control program of a surgical hub, according to at least one aspect of this disclosure, is shown. [Figure 78] A logical flow diagram of a process for updating a data analysis algorithm for a control program of a surgical hub, according to at least one aspect of this disclosure, is shown. [Figure 79] This disclosure illustrates at least one aspect of a system for communication between a surgical instrument, a surgical hub, and a cloud computing system. [Figure 80] A logical flowchart of a process for updating the algorithm of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 81] Another logical flowchart of the process for updating the algorithm of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 82] Another logical flowchart of the process for updating the algorithm of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 83] A logical flowchart of a process for updating the algorithms of surgical instruments, according to at least one aspect of this disclosure, is shown. [Figure 84]An example of a surgical instrument's end effector is shown. [Figure 85] An exemplary control system for surgical instruments is shown. [Figure 86] A partial diagram of the end effector of an exemplary surgical instrument is shown. [Figure 87] An exemplary perspective view of a staple cartridge, including active elements and sensors, is shown. [Figure 88] A block diagram of an exemplary active sensor assembly is shown. [Figure 89] An illustrative perspective view of a cartridge containing a pair of data elements is shown. [Figure 90] A block diagram of an exemplary sensor assembly for detecting and / or receiving data from data elements associated with a cartridge is shown. [Figure 91A] An exemplary surgical staple cartridge is shown. [Figure 91B] An exemplary surgical staple cartridge is shown. [Figure 92] This illustrates a typical power and data transmission scenario. [Figure 93] An exemplary block diagram is shown illustrating the components of an exemplary surgical stapler and an exemplary staple cartridge. [Figure 94] An exemplary process for determining the transmission function is shown. [Figure 95] An exemplary process for determining the transmission function is shown. [Figure 96] An exemplary process for determining the transmission function is shown. [Figure 97] An exemplary process for determining the transmission function is shown. [Figure 98] This is a schematic diagram of a surgical instrument configured to operate the surgical tools described herein, according to at least one aspect of this disclosure. [Figure 99] A block diagram of a surgical instrument configured to control various functions, according to at least one aspect of this disclosure, is shown. [Figure 100]A perspective view of a circular stapled surgical instrument according to at least one aspect of the present disclosure is shown. [Figure 101A] Figure 100 shows an enlarged longitudinal section view of the staple head assembly of the device, showing the anvil in the open position, according to at least one aspect of the present disclosure. [Figure 101B] Figure 100 shows an enlarged longitudinal section view of the staple head assembly of the device, showing the anvil in the closed position, according to at least one aspect of the present disclosure. [Figure 101C] Figure 100 shows an enlarged longitudinal section view of the staple fastening head assembly of the device, showing the staple drive unit and blade in the firing position, according to at least one aspect of the present disclosure. [Figure 102] An enlarged partial cross-sectional view of a staple formed on an anvil according to at least one aspect of this disclosure is shown. [Figure 103] This is a partial cutaway of a powered circular stapling device, including a circular stapling head assembly and an anvil, according to at least one aspect of the present disclosure. [Figure 104] This is a partial top view of a circular staple fastening head assembly shown herein, showing a first row of staples (inner staples) and a second row of staples (outer staples), according to at least one aspect of the present disclosure. [Figure 105] A graphical representation of the feasible staple range, as indicated by an available staple height window, based on interstitial gaps, closure force (FTC), or tissue creep stabilization sensed by a device or combination thereof, according to at least one aspect of the present disclosure. [Figure 106] A graphical representation of a first pair of graphs showing an anvil gap and tissue compressive force and time for exemplary firing of a staple fastener, according to at least one aspect of the present disclosure. [Figure 107] A graphical representation of a second pair of graphs showing an anvil gap and tissue compressive force and time for exemplary firing of a staple fastener, according to at least one aspect of the present disclosure. [Figure 108] This is a schematic diagram of a powered circular stapling device showing effective tissue gaps, actual gaps, normal range gaps, and out-of-range gaps, according to at least one aspect of the present disclosure. [Figure 109] This is a logical flow diagram of a process showing a control program or logical configuration for providing arbitrary or forced lockout based on a sensed parameter compared to a threshold, according to at least one aspect of the present disclosure. [Figure 110] This figure shows the extent of the interstitial gap and the resulting form of staples according to at least one aspect of the present disclosure. [Figure 111] A graph representation of three force-to-close (FTC) curves and time, according to at least one aspect of this disclosure. [Figure 112] A detailed graphical representation of the force to close (FTC) curve and time, according to at least one aspect of this disclosure. [Figure 113] A graph and a diagram of an associated powered stapling device showing an anvil closing speed adjustment at a specific key point along the retraction stroke of a trocar, according to at least one aspect of the present disclosure. [Figure 114] This is a logic flow diagram of a process showing a control program or logic configuration for adjusting the closing speed of an anvil portion of a powered stapling device at a specific key point along the retraction stroke of a trocar, according to at least one aspect of the present disclosure. [Figure 115] A graph and a diagram of an associated power stapling device showing the trocar position over time, according to at least one aspect of the present disclosure. [Figure 116] This is a logic flow diagram of a process representing a control program or logic configuration for detecting multidirectional seating motion on a trocar for driving an anvil to proper seating, according to at least one aspect of the present disclosure. [Figure 117]A partial schematic diagram of a circular powered stapling device according to at least one aspect of the present disclosure, showing an anvil closure on the left and an operating knife 201616 on the right. [Figure 118] A graphical representation of the anvil displacement (δ-anvil) along the vertical axis as a function of the force (FTC) for closing the clamp along the horizontal axis, according to at least one aspect of the present disclosure. [Figure 119] In at least one aspect of this disclosure, the left side is a graphical representation of the displacement of knife 201616 (δ knife) along the vertical axis as a function of the velocity (VK mm / sec) of knife 201616 along the horizontal axis, and the right side is a function of the force (FK lbs) of knife 201616 along the horizontal axis. [Figure 120] This is a logical flowchart of a process showing a control program or logical configuration for detecting interstitial gaps and forces for firing in order to adjust knife stroke and velocity, according to at least one aspect of the present disclosure. [Figure 121] This is a logic flow diagram of a process showing a control program or logic configuration for advancing knife 201616 under a high tissue toughness velocity profile with velocity spikes as shown in Figure 119, according to at least one aspect of the present disclosure. [Figure 122] A partial perspective view of a circular stapler is shown, illustrating a circular stapler trocar including a staple cartridge having four predetermined zones, according to at least one aspect of the present disclosure. [Figure 123] A partial perspective view of a circular stapler is shown, illustrating a circular stapler trocar including a staple cartridge having eight predetermined zones, according to at least one aspect of the present disclosure. [Figure 124] The image shows, on the left, two pieces of tissue containing pre-placed staples appropriately arranged on the staple cartridge of Figure 122, and on the right, two pieces of tissue containing pre-placed staples appropriately arranged on the staple cartridge of Figure 122, according to at least one aspect of the present disclosure. [Figure 125]Two structures are shown according to at least one aspect of this disclosure, including pre-placed staples appropriately arranged on the staple cartridge of Figure 123. [Figure 126] Two structures are shown in at least one aspect of this disclosure, including pre-placed staples improperly arranged on the staple cartridge of Figure 123. [Figure 127] A graph showing the tissue impedance signature of a well-placed tissue in Figure 125, according to at least one aspect of this disclosure. [Figure 128] A graph showing the tissue impedance signature of an improperly positioned tissue, according to at least one aspect of this disclosure, as shown in Figure 126. [Figure 129] This is a logical flow diagram of a process showing a control program or logical configuration for selecting an operating mode of a surgical hub, according to at least one aspect of the present disclosure. [Figure 130] This is a logical flow diagram of a process showing a control program or logical configuration for responding to a sensed parameter, according to at least one aspect of the present disclosure. [Figure 131] This is a diagram of a graphical user interface (GUI) for controlling various device parameters, according to at least one aspect of the present disclosure. [Figure 132] A block diagram showing a surgical system according to at least one aspect of the present disclosure. [Figure 133] This disclosure describes a technique for interacting with a patient's Electronic Medical Record (EMR) database, according to at least one aspect of this disclosure. [Figure 134] This disclosure describes a technique for interacting with a patient's electronic medical record (EMR) database, according to at least one aspect of this disclosure. [Figure 135] An exemplary perspective view of a circular stapler according to at least one aspect of the present disclosure is shown. [Figure 136]Figure 135 shows a perspective view of the circular stapler according to at least one aspect of the present disclosure, with the battery pack removed from the housing assembly and the anvil removed from the staple head assembly. [Figure 137] A control system for a surgical stapling device according to at least one aspect of this disclosure is shown. [Figure 138] A flowchart illustrating an exemplary process for adaptive control of surgical instrument function is shown. [Figure 139] An example of an exemplary motorized circular staple fastener according to at least one aspect of this disclosure is shown. [Figure 140] An illustrative representation of the adaptable staple height operating range is shown on an exemplary motor-driven circular staple fastener. [Figure 141] This is an exemplary flowchart of an exemplary motor-driven circular staple fastener operating in stroke control mode. [Figure 142] This is an exemplary flowchart of an exemplary motor-driven circular staple fastener operating in load-controlled mode. [Figure 143] This is an exemplary flowchart of an exemplary motorized circular staple fastener operating in a previous configuration control mode. [Figure 144] This is an exemplary diagram showing various embodiments of an exemplary motorized circular staple fastener operating using adaptive motor control in a load-controlled operating mode. [Figure 145] This is an exemplary flowchart of a motor-driven circular staple fastener operating with adaptive motor control in load control operation mode. [Figure 146] This is another exemplary flowchart of an exemplary motor-driven circular staple fastener operating in load-controlled mode. [Figure 147] This is another exemplary flowchart of an exemplary motor-driven circular staple fastener operating in load-controlled mode. [Figure 148] This is another exemplary flowchart of an exemplary motorized circular staple fastener operating in a previous configuration control operating mode. [Figure 149] This is another exemplary flowchart of an exemplary motorized circular staple fastener operating in a previous configuration control operating mode. [Figure 150] This is another exemplary flowchart of an exemplary motorized circular staple fastener operating in a previous configuration control operating mode. [Figure 151] This is a schematic diagram of a robotic surgical instrument configured to operate the surgical tools described herein, according to at least one aspect of this disclosure. [Figure 152] A block diagram of a surgical instrument programmed to control the distal translation of a displacement member, according to at least one aspect of the present disclosure, is shown. [Figure 153] This is a schematic diagram of a surgical instrument configured to control various functions, according to at least one aspect of the present disclosure. [Figure 154] A system configured to execute an adaptive ultrasonic blade control algorithm within a surgical data network including a modular communication hub, according to at least one aspect of the present disclosure. [Figure 155] An example of a generator according to at least one aspect of this disclosure is shown. [Figure 156] A surgical system comprising a generator and various surgical instruments usable with the generator, according to at least one aspect of the present disclosure. [Figure 157] This is a diagram of a situational awareness surgical system according to at least one aspect of the present disclosure. [Figure 158A] A configuration of a first exemplary modular energy system, according to at least one aspect of the present disclosure, includes a header module and a display screen representing a graphical user interface (GUI) for relaying information about modules connected to the header module. [Figure 158B] A modular energy system, as shown in Figure 159A, mounted on a cart, according to at least one aspect of this disclosure. [Figure 159A]A second exemplary modular energy system configuration, according to at least one aspect of the present disclosure, includes a header module connected together and mounted on a cart, a display screen, an energy module, and an expansion energy module. [Figure 159B] A third exemplary modular energy system configuration, according to at least one aspect of the present disclosure, is similar to the second configuration shown in Figure 160A, except that the header module lacks a display screen. [Figure 160] A fourth exemplary modular energy system configuration, according to at least one aspect of the present disclosure, includes a header module connected together and mounted on a cart, a display screen, an energy module, an expansion energy module, and a technology module. [Figure 161] A fifth exemplary modular energy system configuration, according to at least one aspect of the present disclosure, includes a header module connected together and mounted on a cart, a display screen, an energy module, an expansion energy module, a technology module, and a visualization module. [Figure 162] This is a diagram of a modular energy system including a transmissibly connectable surgical platform, according to at least one aspect of the present disclosure. [Figure 163] This is a perspective view of a header module of a modular energy system including a user interface, according to at least one aspect of the present disclosure. [Figure 164] This is a block diagram of a standalone hub configuration of a modular energy system according to at least one aspect of the present disclosure. [Figure 165] This is a block diagram of a hub configuration of a modular energy system integrated with a surgical control system, according to at least one aspect of the present disclosure. [Figure 166]This is a block diagram of a user interface module connected to a communication module of a modular energy system, according to at least one aspect of the present disclosure. [Figure 167] This is a block diagram of an energy module of a modular energy system according to at least one aspect of the present disclosure. [Figure 168A] A block diagram of an energy module connected to a header module of a modular energy system, according to at least one aspect of this disclosure, is shown. [Figure 168B] A block diagram of an energy module connected to a header module of a modular energy system, according to at least one aspect of this disclosure, is shown. [Figure 169A] A block diagram of a header / user interface (UI) module of a modular energy system for a hub, according to at least one aspect of this disclosure, is shown. [Figure 169B] A block diagram of a header / user interface (UI) module of a modular energy system for a hub, according to at least one aspect of this disclosure, is shown. [Figure 170] This is a block diagram of an energy module of a hub according to at least one aspect of the present disclosure. [Figure 171] This disclosure illustrates at least one aspect of a system for communication between a surgical instrument, a surgical hub, and a cloud computing system. [Figure 172] A logical flowchart of a process for updating the algorithm of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 173] Another logical flowchart of the process for updating the algorithm of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 174] Another logical flowchart of the process for updating the algorithm of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 175]A logical flowchart of a process for updating the algorithms of surgical instruments, according to at least one aspect of this disclosure, is shown. [Figure 176] A logical flowchart of a process for changing or blending the energy modality of a surgical instrument based on detected threshold parameters, according to at least one aspect of this disclosure, is shown. [Figure 177] This disclosure illustrates at least one aspect of a system for communication between a surgical instrument, a surgical hub, and a cloud computing system. [Figure 178] A logical flow diagram of a process for updating the control algorithm of a surgical hub, according to at least one aspect of this disclosure, is shown. [Figure 179] A logical flowchart of a process for updating the algorithm of a surgical instrument, according to at least one aspect of this disclosure, is shown. [Figure 180] A logical flow diagram of a process for updating a surgical system according to at least one aspect of this disclosure is shown. [Figure 180A] An example of a hub connection mode is shown. [Figure 180B] An example of a hub connection mode is shown. [Figure 180C] An example of a hub connection mode is shown. [Figure 181] This shows an example flow for operating under hierarchical hub communication mode. [Figure 182A] This shows an example flow for operating under hierarchical hub communication mode. [Figure 182B] This shows an example flow for operating under hierarchical hub communication mode. [Figure 182C] This shows an example flow for operating under hierarchical hub communication mode. [Figure 183A] This shows an exemplary surgical instrument packaged with an RFID NFC chip. [Figure 183B] This shows the primary displays of the surgical hub, including global and local displays. [Figure 184]An example of a primary display for a surgical hub is shown. [Figure 185] The image shows a surgeon wearing a set of safety glasses, using surgical instruments including a handle assembly housing and wireless circuit board during a surgical procedure. [Figure 186] This is a diagram illustrating an example of an operating room (OR) layout. [Figure 187] This is a block diagram of a gesture recognition system. [Figure 188] This illustrates exemplary role-based interaction and control related to augmented reality and deviceless control systems. [Figure 189] This exhibits exemplary procedure-based interactions and controls related to augmented reality and deviceless control systems. [Figure 190] This is a schematic diagram illustrating an exemplary visualization of anatomical structures via a spectral surgical visualization system. [Figure 191] This is a diagram of an access route for a surgical instrument for a video-assisted thoracoscopic surgery (VATS) procedure, according to at least one aspect of the present disclosure. [Figure 192] This is a diagram of various coordinate systems related to VATS treatment according to at least one aspect of the present disclosure. [Figure 193] This figure illustrates the example changes in the orientation of the display and user controls in response to changes in the orientation of surgical instruments. [Figure 194] This shows an example camera view of a surgical procedure. [Figure 195] An exemplary representation of a surgical visualization system, as shown by at least one aspect of this disclosure, is provided. [Figure 196] This shows an exemplary model of an anatomical structure generated by an exemplary surgical visualization system. [Figure 197] An exemplary representation of an exemplary model is shown according to at least one aspect of this disclosure. [Figure 198]This shows an exemplary display of an exemplary model of an anatomical structure generated by an exemplary surgical visualization system. [Figure 199] This is a diagram of an exemplary fused image generated from a multispectral EMR source. [Figure 200] This illustrates exemplary procedures and their progression, which may be detected by the system's exemplary situational awareness capabilities. [Figure 201A] This example shows a series of surgical procedures involving the analysis of multiple images at the surgical site. [Figure 201B] This example shows a series of surgical procedures involving the analysis of multiple images at the surgical site. [Figure 201C] This example shows a series of surgical procedures involving the analysis of multiple images at the surgical site. [Figure 202] This shows an example of an augmented video image of a preoperative video image, enhanced with data that identifies the displayed elements. [Figure 203] This example shows an augmented reality overlay of a target region with pre-surgical tumor data and real-time Doppler monitoring. [Figure 204] This shows an example flow of a hub operating under hierarchical visualization control mode. [Figure 205] This shows an example flow of a hub operating under hierarchical visualization control mode. [Figure 206] This provides a detailed, illustrative flow for a hub operating under a visualization control mode, where the secondary display is an augmented reality (AR) device. [Figure 207] This shows an exemplary flow for a hub operating under a visualization control mode that supports situational awareness. [Figure 208] This shows an exemplary flow for a hub operating under a visualization control mode that supports situational awareness. [Figure 209] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports adjusting the display based on regulated display events. [Figure 210] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports AR capabilities. [Figure 211] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports AR capabilities. [Figure 212] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports role-based AR capabilities. [Figure 213] This illustrates an exemplary flow of a hub operating under a visualization control mode with AR capabilities that support various display overlays. [Figure 214] This shows an exemplary flow for operation under the multi-display control mode. [Figure 215] This shows an example flow for operating under hierarchical multi-display control mode(s). [Figure 216] This shows an example flow for operating under hierarchical multi-display control mode(s). [Figure 217] This illustrates an exemplary multi-display control mode, such as a one-way transmission mode. [Figure 218] An exemplary multi-display control mode is shown, which may support sterile field display-based control. [Figure 219] This demonstrates an exemplary multi-display control mode that can support remote data aggregation and analysis. [Figure 220] This shows an example flow for operating under hierarchical multi-display control mode(s). [Figure 221] This shows an example flow for operating under hierarchical multi-display control mode(s). [Modes for carrying out the invention]

[0012] The applicant of this application also owns the following U.S. patent applications filed concurrently, each of which is incorporated herein by reference in its entirety. ● U.S. Patent Application No. 15 / 940,656, titled "SURGICAL HUB COORDINATION OF CONTROL AND COMMUNICATION OF OPERATING ROOM DEVICES," filed on March 29, 2018, is currently U.S. Patent Publication No. 2019 / 0201141. ● U.S. Patent Application No. 16 / 361,793, titled "SURGICAL INSTRUMENT COMPRISING AN ADAPTIVE CONTROL SYSTEM," filed on March 22, 2019, is currently U.S. Patent Publication No. 20190314015. ● U.S. Patent Application No. 13 / 803,086, entitled "ARTICULATABLE SURGICAL INSTRUMENT COMPRISING AN ARTICULATION LOCK", now published as U.S. Patent Application Publication No. 2014 / 0263541. ● U.S. Patent Application No. 13 / 800,067, titled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," filed on March 13, 2013, is currently U.S. Patent Publication No. 2014 / 0263552. ● U.S. Patent Application No. 16 / 024,075, titled "SAFETY SYSTEMS FOR SMART POWERED SURGICAL STAPLING," filed on June 29, 2018, is currently U.S. Patent Publication No. 2019 / 0201146. ● U.S. Patent Application No. 15 / 940,679, titled "CLOUD-BASED MEDICAL ANALYTICS FOR LINKING OF LOCAL USAGE TRENDS WITH THE RESOURCE ACQUISITION BEHAVIORS OF LARGER DATA SET," filed on March 29, 2018, is currently published as U.S. Patent Application Publication No. 2019 / 0201144. ● U.S. Patent Application No. 15 / 940,668 (Agent Reference Number END8501USNP2), titled "AGGREGATION AND REPORTING OF SURGICAL HUB DATA," filed on March 29, 2018, is currently U.S. Patent Publication No. 2019 / 0201115. ● U.S. Patent Application Publication No. 2019 / 0200981, filed on July 4, 2019, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS" (U.S. Patent Application No. 16 / 209,423, filed on December 4, 2018), ● U.S. Patent No. 9,072,535, issued on July 5, 2015, entitled "SURGICAL STAPLING INSTRUMENTS WITH ROTATABLE STAPLE DEPLOYMENT ARRANGEMENTS" ● U.S. Patent Application Publication No. 20140263551, titled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM" (U.S. Patent Application No. 13 / 800,025, filed on March 13, 2013), published on September 18, 2014. ● U.S. Patent Application Publication No. 20190000478, titled "SURGICAL SYSTEM COUPLED WITH STAPLE CARTRIDGE AND RADIO FREQUENCY CARTRIDGE, AND METHOD OF USING SAME" (U.S. Patent Application No. 15 / 636,096, filed on June 28, 2017), published on January 3, 2019. ● U.S. Patent Application Publication No. 20160256156, titled "TIME DEPENDENT EVALUATION OF SENSOR DATA TO DETERMIN STABILITY, CREEP, AND VISCOELASTIC ELEMENTS OF MEASURES" (Application No. 14 / 852,982, filed on September 14, 2015), published on September 8, 2016, and ● U.S. Patent No. 10,695,081, issued on June 30, 2020, entitled "CONTROLLING A SURGICAL INSTRUMENT ACCORDING TO SENSED CLOSURE PARAMETERS". ● U.S. Patent Application Publication No. 2019-0200844(A1) titled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY" (U.S. Patent Application No. 16 / 209,385 filed on December 4, 2018), published on July 4, 2019. ● U.S. Patent Application Publication No. 2019-0201137(A1) titled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL" (U.S. Patent Application No. 16 / 209,407 filed on December 4, 2018), published on July 4, 2019. ● U.S. Patent Application Publication No. 2019-0206569(A1) titled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB" (U.S. Patent Application No. 16 / 209,403 filed on December 4, 2018), published on July 4, 2019. ● U.S. Patent Application Publication No. 2017-0296213(A1) titled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT" (U.S. Patent Application No. 15 / 130,590 filed on April 15, 2016), published on October 19, 2017. ● U.S. Patent No. 9,345,481, issued on May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM" ● U.S. Patent Application Publication No. 2018 / 0360452, titled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT" (U.S. Patent Application No. 15 / 628,175, filed on June 20, 2017), published on December 20, 2018. ● U.S. Patent Application No. 16 / 209,416, titled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS," filed on December 4, 2018. ●U.S. Patent Application No. 15 / 940,671, titled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER" (Agent Reference Number END8502USNP), filed on March 29, 2018. ● U.S. Patent Application No. 16 / 182,269, titled "IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE" (Agent Reference Number END9018USNP3), filed on November 6, 2018. ●U.S. Patent Application No. 16 / 729,747, titled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS" (Agent Reference Number END9217USNP1), filed on December 31, 2019. ●U.S. Patent Application No. 16 / 729,778, titled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE" (Agent Reference Number END9219USNP1), filed on December 31, 2019. ●U.S. Patent Application No. 16 / 729,807, titled "METHOD OF USING IMAGING DEVICES IN SURGERY" (Agent Reference Number END9228USNP1), filed on December 31, 2019. ●U.S. Patent Application No. 15 / 940,654, titled "SURGICAL HUB SITUATION AWARENESS," filed on March 29, 2018 (Agent Reference Number END8501USNP), ●U.S. Patent Application No. 15 / 940,704 (Agent Reference Number END8504USNP), filed on March 29, 2018, entitled "USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT" ●U.S. Patent Application No. 16 / 182,290 (Agent Reference Number END9018USNP5), filed on November 6, 2018, entitled "SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION," ● U.S. Patent No. 9,011,427, titled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued on April 21, 2015. ● U.S. Patent No. 9,123,155, issued on September 1, 2015, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES" ●U.S. Patent Application No. 16 / 209,478 (Agent Reference Number END9015USNP1), filed on December 4, 2018, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE," ● U.S. Patent Application Publication No. 20190206563(A1) (U.S. Patent Application No. 16 / 209,465), filed on December 4, 2018, entitled "Method for adaptive control schemes for surgical network control and interaction," ● U.S. Patent Application Publication No. 20190201034(A1) (U.S. Patent Application No. 16 / 182,240), filed on November 6, 2018, entitled "Powered stapling device configured to adjust force, advancement speed, and overall stroke of cutting member based on sensed parameter of firing or clamping," ● U.S. Patent Application Publication No. 20190200996(A1) (U.S. Patent Application No. 16 / 182,229), filed on November 6, 2018, entitled "ADJUSTMENT OF STAPLE HEIGHT OF AT LEAST ONE ROW OF STAPLES BASED ON THE SENSED TISSUE THICKNESS OR FORCE IN CLOSING," ● U.S. Patent Application Publication No. 20190200997(A1) (U.S. Patent Application No. 16 / 182,234), filed on November 6, 2018, entitled "Stapling device with both compulsory and discretionary lockouts based on sensed parameters," ● U.S. Patent Application No. 16 / 458,117, filed on June 30, 2019, entitled "SURGICAL SYSTEM WITH RFID TAGS FOR UPDATING MOTOR ASSEMBLY PARAMETERS" ● U.S. Patent Application Publication No. 2017 / 0296213 (U.S. Patent Application No. 15 / 130,590), titled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," was published on October 19, 2017. ● U.S. Patent Application Publication No. 20190099180(A1) (U.S. Patent Application No. 15 / 720,852), filed on September 29, 2017, entitled "SYSTEM AND METHODS FOR CONTROLLING A DISPLAY OF A SURGICAL INSTRUMENT", ● U.S. Patent Application Publication No. 2014 / 0166728 (U.S. Patent Application No. 13 / 716,318), titled "Motor Driven Rotary Input Circular Stapler with Modular End Effector," was published on June 19, 2014. ● U.S. Patent No. 9,250,172, published on February 2, 2016, entitled "Systems and methods for predicting metabolic and bariatric surgery outcomes" ● U.S. Patent Application Publication No. 20130116218(A1) (U.S. Patent Application No. 13 / 631,095), titled "Methods and compositions of bile acids," was published on May 9, 2013. ● U.S. Patent Application Publication No. 20140087999(A1) (U.S. Patent Application No. 13 / 828,809), titled "Clinical predictors of weight loss," was published on March 27, 2014. ● U.S. Patent No. 8,476,227, published on July 2, 2013, entitled "Methods of activating a melanocortin-4 receptor pathway in obese subjects" ● U.S. Patent Application No. 16 / 574,773, filed on September 18, 2019, entitled "METHOD FOR CALIBRATING MOVEMENTS OF ACTUATED MEMBERS OF POWERED SURGICAL STAPLER" ● U.S. Patent Application No. 16 / 574,797, filed on September 18, 2019, entitled "METHOD FOR CONTROLLING CUTTING MEMBER ACTUATION FOR POWERED SURGICAL STAPLER" ● U.S. Patent Application No. 16 / 574,281, filed on September 18, 2019, entitled "METHOD FOR CONTROLLING END EFFECTOR CLOSURE FOR POWERED SURGICAL STAPLER" ● U.S. Patent Application Publication No. 20190201119(A1) (U.S. Patent Application No. 15 / 940,694), filed on March 29, 2018, entitled "CLOUD-BASED MEDICAL ANALYTICS FOR MEDICAL FACILITY SEGMENTED INDIVIDUALIZATION OF INSTRUMENT FUNCTION" ● U.S. Patent No. 10,492,783, issued on December 3, 2019, entitled "SURGICAL INSTRUMENT WITH IMPROVED STOP / START CONTROL DURING A FIRING MOTION" ● U.S. Patent Application Publication No. 20190200998(A1) (U.S. Patent Application No. 16 / 209,491), filed on December 4, 2018, entitled "METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS", ● U.S. Patent Application No. 15 / 940,636 (Agent Reference Number END8506USNP), titled "ADAPTIVE CONTROL PROGRAM UPDATES FOR SURGICAL DEVICES," filed on March 29, 2018, is currently U.S. Patent Publication No. 2019 / 0206003. ● U.S. Patent Application No. 16 / 562,170 (Agent Reference Number END9070USNP8), titled "MANAGING SIMULTANEOUS MONOPOLAR OUTPUTS USING DUTY CYCLE AND SYNCHRONIZATION," filed on September 5, 2019, is currently U.S. Patent Publication No. 2020 / 0078079. ● U.S. Patent Application No. 16 / 209,490 (Agent Reference Number END9017USNP1) titled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION," filed on December 4, 2018, is currently U.S. Patent Publication No. 2019 / 0206564, and ● U.S. Patent Application No. 16 / 182,246, titled "ADJUSTMENTS BASED ON AIRBORNE PARTICLE PROPERTIES" (Agent Reference Number END9016USNP1), filed on November 6, 2018.

[0013] Referring to Figure 1, the computer-implemented interactive surgical system 100 may include one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 which may include a remote server 113 connected to a storage device 105). Each surgical system 102 may include at least one surgical hub 106 that communicates with the cloud 104 which may include the remote server 113. In one embodiment, as shown in Figure 1, the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some embodiments, the surgical system 102 may include M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P are integers of 1 or more.

[0014] In various embodiments, the visualization system 108 may include one or more imaging sensors strategically positioned relative to a sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 2. In one embodiment, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0015] As shown in Figure 2, the primary display 119 is positioned in the sterile field so that it is visible to the operator on the operating table 114. In addition, a visualization tower 111 is positioned outside the sterile field. The visualization tower 111 may include a first non-sterile display 107 and a second non-sterile display 109 facing opposite directions from each other. The visualization system 108, guided by the hub 106, is configured to utilize displays 107, 109, and 119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 106 may cause the visualization system 108 to display snapshots of the surgical site recorded by the imaging device 124 on the non-sterile displays 107 or 109 while maintaining live video of the surgical site on the primary display 119. The snapshots on the non-sterile displays 107 or 109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0016] In one embodiment, the hub 106 may also be configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 to a primary display 119 in the sterile field, which can then be viewed by a sterile operator on the operating table. In one embodiment, the input may take the form of modifications to a snapshot displayed on a non-sterile display 107 or 109, which can then be sent to the primary display 119 by the hub 106.

[0017] Referring to Figure 2, the surgical instrument 112 is used as part of the surgical system 102 in a surgical procedure. The hub 106 may also be configured to coordinate the flow of information to the display of the surgical instrument 112. For example, such disclosure is incorporated herein by reference in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY". Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 can be sent by the hub 106 to the surgical instrument display 115 in the sterile field, which can then be viewed by the operator of the surgical instrument 112. Examples of surgical instruments suitable for use with surgical system 102 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0018] Figure 2 shows an example of a surgical system 102 used to perform a surgical procedure on a patient lying on an operating table 114 in a surgical operating room 116. A robotic system 110 may be used as part of the surgical system 102 in the surgical procedure. The robotic system 110 may include a surgeon's console 118, a patient-side cart 120 (surgical robot), and a surgical robot hub 122. While the surgeon views the surgical site through the surgeon's console 118, the patient-side cart 120 can manipulate at least one detachably connected surgical tool 117 through a minimally invasive incision in the patient's body. Images of the surgical site are acquired by a medical imaging device 124, which can be manipulated by the patient-side cart 120 to orient the imaging device 124. The robotic hub 122 can be used to process images of the surgical site, which can then be displayed to the surgeon through the surgeon's console 118.

[0019] Other types of robotic systems can be readily adapted for use with surgical system 102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed on 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0020] Various examples of cloud-based analytics performed by Cloud104 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed on 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," which is incorporated herein by reference in its entirety.

[0021] In various embodiments, the imaging device 124 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0022] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0023] One or more illuminators may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0024] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and include invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and include invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0025] In various embodiments, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0026] The imaging device may employ multispectral monitoring to distinguish the topography from the underlying structure. A multispectral image is image data acquired from the entire electromagnetic spectrum within a specific wavelength range. Wavelengths may be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is required in any surgical procedure. The strict hygiene and sterilization conditions required in the “surgical field,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process described above is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient who is ready for surgical treatment. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and restraints within that area.

[0027] Referring here to Figure 3, a hub 106 is shown that communicates with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, a storage array 134, and an operating room mapping module 133. In certain embodiments, as shown in Figure 3, the hub 106 further includes a fume extraction module 126 and / or aspiration / irrigation module 128. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, aspiration of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Valuable time can be lost dealing with this problem during surgical procedures. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The modular enclosure 136 of the hub provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines. An aspect of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within a docking station of the hub enclosure. The docking station includes data contacts and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed in a single unit. In one aspect, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid, and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the fume exhaust component.In one embodiment, the above-mentioned fluid line is a first fluid line, and a second fluid line extends from the remote surgical site to a suction and irrigation module slidably received within the hub enclosure. In one embodiment, the hub enclosure includes a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Embodiments of the present disclosure present a solution in which a modular enclosure 136 of the hub is configured to house various generators and facilitate interactive communication between them. One advantage of the modular enclosure 136 of the hub is that it allows for the rapid removal and / or replacement of various modules. Embodiments of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station including a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station having a second docking port including a second data contact and a second power contact, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is slidably movable to disengage from the second power and data contacts.In addition, the modular surgical enclosure also includes a communication bus between a first docking port and a second docking port, configured to facilitate communication between a first energy generator module and a second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented relating to a modular enclosure 136 of a hub that enables modular integration of a generator module 140, a smoke evacuation module 126, and a suction / irrigation module 128. The modular enclosure 136 of the hub further facilitates interactive communication between modules 140, 126, and 128. The generator module 140 may be a generator module comprising integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the modular enclosure 136 of the hub. The generator module 140 may be configured to connect to a unipolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the modular enclosure 136 of the hub. The modular enclosure 136 of the hub may be configured to facilitate the insertion of multiple generators and interactive communication between generators docked to the modular enclosure 136 of the hub, so that multiple generators function as a single generator.

[0028] Figure 4 shows a surgical data network 201 including a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to a cloud-based system (e.g., a cloud 204 which may include a remote server 213 connected to a storage device 205). In one embodiment, the modular communication hub 203 includes a network hub 207 and / or a network switch 209 that communicate with a network router. The modular communication hub 203 can also be connected to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional mechanisms that constitute each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0029] Modular devices 1a-1n located in the surgical field may be connected to a modular communication hub 203. A network hub 207 and / or a network switch 209 may be connected to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to a local computer system 210 for local data processing and manipulation. Modular devices 2a-2m located in the same surgical field may also be connected to a network switch 209. The network switch 209 may be connected to a network hub 207 and / or a network router 211 to connect devices 2a-2m to a cloud 204. Data associated with devices 2a-2n may be transferred to a cloud 204 via the network router 211 for data processing and manipulation. Data associated with devices 2a-2m may also be transferred to a local computer system 210 for local data processing and manipulation.

[0030] It will be understood that the surgical data network 201 can be extended by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. A modular communication hub 203 may be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. The modular communication hub 203 is connected to a display 212 to display, for example, images acquired by some of the devices 1a-1n / 2a-2m during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 203 of the surgical data network 201, such as an imaging module 138 connected to an endoscope, a generator module 140 connected to an energy-based surgical device, a smoke extraction module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module.

[0031] In one embodiment, the surgical data network 201 may include a combination of a network hub, network switches, and network routers connecting devices 1a-1n / 2a-2m to the cloud. One or all of the devices 1a-1n / 2a-2m connected to the network hub or network switch may collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” may be used as a metaphor for “Internet,” but the term is not limited in that way. Thus, the term “cloud computing” may be used herein to refer to “a type of internet-based computing,” in which various services such as servers, storage devices, and applications are delivered via the Internet to a modular communication hub 203 and / or computer system 210 located in a surgical setting (e.g., a fixed, mobile, temporary, or on-site surgical setting or space), and to devices connected to the modular communication hub 203 and / or computer system 210. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located within one or more surgical sites. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located within the surgical site. The hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage devices.

[0032] By applying cloud computing data processing techniques to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and increased patient satisfaction. At least some of devices 1a-1n / 2a-2m may be used to observe the condition of tissue after tissue sealing and cutting procedures and to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m may be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin identification, as well as phenotype. At least some of devices 1a-1n / 2a-2m may be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud 204 or a local computer system 210, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcomes of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be carried out for tissue-specific sites and conditions. Such data analysis may also involve further prognostic analysis, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0033] The surgical field devices 1a-1n may be connected to the modular communication hub 203 via a wired or wireless channel, depending on the configuration of devices 1a-1n with respect to the network hub. In one embodiment, the network hub 207 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub may provide connectivity to devices 1a-1n located within the same surgical field network. The network hub 207 may collect data in packet form and transmit them to the router in half-duplex mode. The network hub 207 does not need to store any media access control / Internet protocol (MAC / Internet Protocol, IP) for transferring device data. Only one of devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 does not need to have a routing table or intelligence regarding the destination of information and broadcasts all network data to each connection and to the remote server 213 (Figure 4) on the cloud 204. While the Network Hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and could cause bottlenecks.

[0034] Surgical site devices 2a-2m may be connected to network switch 209 via a wired or wireless channel. Network switch 209 functions within the data linkage layer of the OSI model. Network switch 209 may be a multicast device for connecting devices 2a-2m located within the same surgical site to the network. Network switch 209 may transmit data in the form of frames to network router 211 and may operate in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 209. Network switch 209 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0035] The network hub 207 and / or network switch 209 may be connected to the network router 211 to connect to the cloud 204. The network router 211 operates within the network layer of the OSI model. The network router 211 creates a route for sending data packets received from the network hub 207 and / or network switch 211 to cloud-based computing resources for further processing and manipulation of data collected by one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, such as different networks located in different operating rooms of the same medical facility or different operating rooms of different medical facilities. The network router 211 may send data in packet form to the cloud 204 and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to transfer data.

[0036] In one embodiment, the network hub 207 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 207 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located within the surgical field.

[0037] In the embodiment, surgical site devices 1a-1n / 2a-2m may exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with a modular communication hub 203 via the Bluetooth wireless technology standard to build a personal area network (PAN). The surgical site devices 1a-1n / 2a-2m may communicate with the modular communication hub 203 via several wireless or wired communication standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, new radio (NR), long-term evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G, and later. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi and Bluetooth, while the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and Ev-DO.

[0038] The modular communication hub 203 may function as a central connection point for one or all of the surgical field devices 1a-1n / 2a-2m and may handle a data type known as a frame. The frame may carry data generated by the devices 1a-1n / 2a-2m. When a frame is received by the modular communication hub 203, the frame is amplified and transmitted to the network router 211, which then transfers this data to cloud computing resources using some wireless or wired communication standard or protocol as described herein.

[0039] The modular communication hub 203 may be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Because the modular communication hub 203 is generally easy to install, configure, and maintain, it is a good choice for networking surgical site devices 1a-1n / 2a-2m.

[0040] Figure 5 shows a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204 which may include a remote server 213. In one embodiment, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to multiple surgical site devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 236 includes a modular communication hub 203 connected to a computer system 210.

[0041] As shown in the embodiment of Figure 5, the modular control tower 236 may be connected to an imaging module 238 which can be connected to an endoscope 239, a generator module 240 which can be connected to an energy device 241, a fume exhaust module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 optionally connected to a display 237, and a non-contact sensor module 242. Surgical site devices may be connected to cloud computing resources and data storage devices via the modular control tower 236. The robot hub 222 may also be connected to the modular control tower 236 and cloud computing resources. In particular, the device / instrument 235 and the visualization system 208 may be connected to the modular control tower 236 via wired or wireless communication standards or protocols as described herein. The modular control tower 236 may be connected to a hub display 215 (e.g., a monitor, screen) to display and overlay images received from imaging modules, device / instrument displays, and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with the images and overlaid images.

[0042] Figure 6 shows a surgical hub 206 including multiple modules connected to a modular control tower 236. The modular control tower 236 may include a modular communication hub 203, e.g., a network connectivity device, and a computer system 210 for, e.g., local processing, visualization, and imaging. As shown in Figure 6, the modular communication hub 203 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that can be connected to the modular communication hub 203, and data associated with the modules may be transferred to the computer system 210, cloud computing resources, or both. As shown in Figure 6, each network hub / switch within the modular communication hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and local displays 217. Communication to the cloud 204 may be via either a wired communication channel or a wireless communication channel.

[0043] The surgical hub 206 may use a non-contact sensor module 242 to measure the dimensions of the surgical field and generate a map of the surgical field using either an ultrasonic non-contact measuring device or a laser non-contact measuring device. An ultrasonic-based non-contact sensor module is described in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” under the heading “Surgical Hub Spatial Awareness Within an Operating Room,” and the disclosure thereof is incorporated herein by reference in its entirety, and may scan the surgical field by transmitting bursts of ultrasound and receiving echoes as they reflect off the outer walls of the surgical field, the sensor module being configured to determine the size of the surgical field and adjust the Bluetooth pairing distance limits. A laser-based non-contact sensor module can, for example, scan a surgical site by transmitting laser light pulses, receive the laser light pulses reflected from the outer walls of the surgical site, compare the phase of the transmitted pulses with the received pulses to determine the size of the surgical site, and adjust the Bluetooth pairing distance limit.

[0044] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 may be connected via a system bus to a communication module 247, a storage device 248, a memory 249, a non-volatile memory 250, and an input / output interface 251. The system bus may be any of several types of bus structures, including memory buses or memory controllers, peripheral buses or external buses, and / or local buses, using any variety of available bus architectures. Examples of such architectures include, but are not limited to, 9-bit buses, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0045] Processor 244 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which are available in the product datasheet.

[0046] In one embodiment, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0047] System memory can include volatile and non-volatile memory. The basic input / output system (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory can include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0048] The computer system 210 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage devices. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include storage media separately or in combination with other storage media, such as, but are not limited to, optical disk drives including compact disc ROM devices (CD-ROM), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROM). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0049] It should be understood that the computer system 210 may include software that acts as an intermediary between the user and basic computer resources as described in a preferred operating environment. Such software may include an operating system. An operating system, which may be stored on a disk storage device, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on a disk storage device. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0050] The user may input commands or information to the computer system 210 through input devices 251 connected to the I / O interface 251. Input devices may include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards, microphones, joysticks, gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor via the system bus through interface ports 251. Interface ports 251 may include, for example, serial ports, parallel ports, game ports, and USB ports. Output devices 251 may use some of the same types of ports as the input devices 251. Therefore, for example, a USB port may be used to provide input to the computer system and output information from the computer system to the output devices. Output adapters may be provided to indicate that there may be several output devices, such as monitors, displays, speakers, and printers, among others, which may require special adapters. Examples of output adapters include video and sound cards that provide a means of connection between the output device and the system bus, but these are illustrative and not limiting. Note that other devices and / or systems of devices, such as remote computers, may also provide both input and output capabilities.

[0051] Computer system 210 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers can be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described with respect to computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface and subsequently physically connected via a communication interface. The network interface may encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies may include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point connections, integrated services digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL).

[0052] In various embodiments, the computer system 210 in Figure 6, the imaging module 238 in Figures 5-6, and / or the visualization system 208, and / or the processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor may use parallel computing with single-instruction, multiple data (SIMD) or multiple-instruction, multiple data (MIMD) technology to enhance speed and efficiency. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0053] A communication connection(s) may refer to hardware / software used to connect a network interface to a bus. For illustrative purposes only, the communication connection(s) are shown as being inside the computer system, but they may be outside the computer system 210. For illustrative purposes only, hardware / software required for connection to a network interface may include modems including typical telephone-grade modems, cable modems, and DSL modems, ISDN adapters, and internal and external technologies such as Ethernet cards.

[0054] Figure 7 shows a logic diagram of a control system 470 for a surgical instrument or tool according to one or more embodiments of the present disclosure. The system 470 may include a control circuit. The control circuit may include a microcontroller 461 including a processor 462 and memory 468. For example, one or more of sensors 472, 474, and 476 provide real-time feedback to the processor 462. A motor 482 driven by a motor drive unit 492 drives an I-beam knife element by operably connecting a longitudinally movable displacement member. A tracking system 480 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to a processor 462 which can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the firing member, firing bar, and I-beam knife element. Additional motors may be provided to the tool drive unit interface to control the firing of the I-beam, the movement of the occluder, the rotation of the shaft, and joint movement. A display 473 may display various operating conditions of the instrument and may include a touchscreen function for data input. The information displayed on the display 473 may be overlaid with images acquired via the endoscopic imaging module.

[0055] In one embodiment, the microcontroller 461 may be any single coprocessor or multicore processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 461 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes, for example, on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance above 40MHz, 32KB single-cycle SRAM, internal ROM with StellarisWare® software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels.

[0056] In one embodiment, the microcontroller 461 may include a safety controller, which may include two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured specifically for IEC61508 and ISO26262 safety limit applications, in particular, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0057] The microcontroller 461 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 461 may include a processor 462 and memory 468. The electric motor 482 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor drive unit 492 may be the A3941 available from Allegro Microsystems, Inc. Other motor drives may be readily substituted for use in the tracking system 480 with an absolute positioning system. A detailed description of the absolute positioning system is found in U.S. Patent Application Publication No. 2017 / 0296213, “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” published on 19 October 2017, which is incorporated herein by reference in its entirety.

[0058] The microcontroller 461 may be programmed to provide precise control over the speed and position of the displacement member and joint motion system. The microcontroller 461 may be configured to calculate the response within its software. The calculated response may be compared to the measured response of the actual system to obtain an "observed" response, which is used to determine the actual feedback. The observed response is a suitable adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and it can detect external influences on the system.

[0059] In some examples, the motor 482 may be controlled by a motor drive unit 492 and may be used by a surgical instrument or tool launching system. In various forms, the motor 482 may be a brushed DC drive motor having a maximum rotational speed of approximately 25,000 RPM. In some examples, the motor 482 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor drive unit 492 may include, for example, an H-bridge drive unit including a field-effect transistor (FET). The motor 482 may be powered by a power supply assembly removably mounted to a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may include a battery which may include several battery cells connected in series that can be used as a power source for powering a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be a lithium-ion battery which may be connectable to and detachable from the power supply assembly.

[0060] The motor drive unit 492 may be an A3941 available from Allegro Microsystems, Inc. The A3941 492 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The drive unit 492 may include a proprietary charge pump regulator that can provide full (>10V) gate drive to battery voltages down to 7V, and can enable the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to provide the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FET may be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and may be configured to protect the power MOSFETs under most short-circuit conditions. Other motor drive units may be readily substituted for use in the tracking system 480, including the absolute positioning system.

[0061] The tracking system 480 may include a controlled motor drive circuit arrangement including a position sensor 472 according to one aspect of the present disclosure. The position sensor 472 for the absolute positioning system may provide a unique position signal corresponding to the location of the displaced member. In some examples, the displaced member may represent a longitudinally movable drive member including a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displaced member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displaced member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displaced member can generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element that can be displaced. In one aspect, a longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 472 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include a contact-type displacement sensor or a non-contact-type displacement sensor.A linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical detection system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical detection system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0062] The electric motor 482 may include a rotary shaft that operably interfaces with a gear assembly mounted in meshing engagement with a set of drive teeth or rack on the displacement member. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 472 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply may provide power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member including a rack of drive teeth formed thereon for meshing engagement with the corresponding drive gear of the gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0063] One rotation of the sensor element associated with the position sensor 472 may correspond to the longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor mechanism may be connected via a gear reduction that results in the position sensor 472 completing one or more rotations relative to the full stroke of the displacement member. The position sensor 472 may complete multiple rotations relative to the full stroke of the displacement member.

[0064] To provide a unique position signal for two or more rotations of the position sensor 472, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches may be fed back to the microcontroller 461, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 472 is provided to the microcontroller 461. The position sensor 472 of the sensor mechanism may include an array of analog rotation sensors such as a magnetic sensor or potentiometer, or analog Hall effect elements, which output a unique combination of position signals or values.

[0065] The position sensor 472 may comprise any number of magnetic sensing elements, such as magnetic sensors, which are classified according to whether they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors may include numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0066] In one embodiment, the position sensor 472 of the tracking system 480, which includes an absolute positioning system, may include a magnetic rotation absolute positioning system. The position sensor 472 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor, available from Austria Microsystems, AG. The position sensor 472 interfaces with a microcontroller 461 to provide an absolute positioning system. The position sensor 472 may be a low-voltage, low-power component and includes four Hall effect elements in the area of ​​the position sensor 472 that can be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement a simple and efficient algorithm for calculating hyperbolic and trigonometric functions that requires only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 461 via a standard serial communication interface, such as a serial peripheral interface (SPI). The position sensor 472 may provide 12-bit or 14-bit resolution. The position sensor 472 may also be an AS5055 chip, provided in a small QFN 16-pin 4×4×0.85mm package.

[0067] The tracking system 480, including the absolute positioning system, may include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 472, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors(s) may include those described in U.S. Patent No. 9,345,481, “STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM,” issued on 24 May 2016, which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, “STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM,” published on 18 September 2014, which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, “TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT,” filed on 20 June 2017, which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. To predict what the state and output of the physical system will be by knowing the input, the calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, and inductive resistance.

[0068] The absolute positioning system may provide the absolute position of the displacement member when the device is powered on, without requiring the displacement member to be moved back or forward to a reset (zero or home) position, as may be necessary with conventional rotary encoders that simply count the number of forward or backward strokes taken by the motor 482 to estimate the position of the device actuator, drive bar, knife, etc.

[0069] For example, a sensor 474, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of an end effector, such as the amplitude of strain exerted on the anvil during clamping, which can indicate the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to a processor 462. Instead of, or in addition to, sensor 474, a sensor 476, such as a load sensor, can measure the closing force applied to the anvil by the closing drive system. For example, sensor 476, such as a load sensor, can measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward the staple drive unit, pushing the staple out and causing it to deform contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue when the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 478 can be used to measure the current drawn in by the motor 482. The force required to propel the launching member forward may, for example, correspond to the current drawn in by the motor 482. The measured force may be converted into a digital signal and provided to the processor 462.

[0070] In one embodiment, a strain gauge sensor 474 can be used to measure the force applied to tissue by the end effector. A strain gauge can be connected to the end effector to measure the force applied by the end effector to the tissue being treated. The system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 474, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 474 can measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 462 of the microcontroller 461. A load sensor 476 can measure the force used to operate a knife element, for example, to cut tissue trapped between an anvil and a staple cartridge. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor can also be converted into a digital signal and provided to the processor 462.

[0071] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 474 and 476 respectively, can be used by the microcontroller 461 to characterize the selected position of the launching member and / or the corresponding values ​​of the launching member's velocity. In one embodiment, memory 468 can store techniques, equations, and / or lookup tables that can be used by the microcontroller 461 during evaluation.

[0072] The control system 470 for surgical instruments or tools may also include wired or wireless communication circuits for communicating with the modular communication hub 203, as shown in Figures 5 and 6.

[0073] Figure 8 shows a surgical instrument or tool that includes multiple motors that can be activated to perform various functions. In a particular example, the first motor can be activated to perform the first function, the second motor can be activated to perform the second function, the third motor can be activated to perform the third function, the fourth motor can be activated to perform the fourth function, and so on. In a particular example, the multiple motors of the robotic surgical instrument 600 can be activated individually to produce firing, closing, and / or jointing motions in the end effector. The firing, closing, and / or jointing motions can be transmitted to the end effector, for example, via a shaft assembly.

[0074] In certain examples, the surgical instrument system or tool may include a firing motor 602. The firing motor 602 may be operably connected to a firing motor drive assembly 604, which can be configured to transmit the firing motion generated by the motor 602 to an end effector, specifically to displace an I-beam element. In certain examples, the firing motion generated by the motor 602 may, for example, deploy a staple from a staple cartridge into tissue captured by the end effector and / or advance the cutting edge of the I-beam element to cut the captured tissue. The I-beam element can be retracted by reversing the direction of the motor 602.

[0075] In certain examples, the surgical instrument or surgical tool may include a closing motor 603. The closing motor 603 may be operably coupled to a closing motor drive assembly 605, which may be configured to specifically displace a closing tube to close the anvil and transmit the closing motion generated by the motor 603 to the end effector to compress tissue between the anvil and the staple cartridge. The closing motion allows the end effector to transition from an open configuration to an approach configuration, for example, to capture tissue. The end effector may be moved to an open position by reversing the direction of the motor 603.

[0076] In certain examples, a surgical instrument or tool may include, for example, one or more articular motion motors 606a, 606b. The motors 606a, 606b may be operably coupled to corresponding articular motion motor drive assemblies 608a, 608b, which may be configured to transmit the articular motion generated by the motors 606a, 606b to an end effector. In certain examples, the articular motion may cause, for example, the end effector to articulate relative to the shaft.

[0077] As described herein, a surgical instrument or tool may include multiple motors that can be configured to perform various independent functions. In certain examples, multiple motors of a surgical instrument or tool can be activated individually or separately to perform one or more functions while other motors remain stopped. For example, articulation motors 606a and 606b can be activated to articulate an end effector while firing motor 602 remains stopped. Alternatively, firing motor 602 can be activated to fire multiple staples and / or advance a cutting edge while articulation motor 606 remains stopped. Furthermore, a closure motor 603 may be activated simultaneously with firing motor 602 to advance the closure tube and I-beam element distally, as described in more detail below herein.

[0078] In certain examples, a surgical instrument or tool may include a common control module 610 that can be used with multiple motors of the surgical instrument or tool. In certain examples, the common control module 610 may correspond to one of the multiple motors at a time. For example, the common control module 610 may be individually connectable and disconnectable to multiple motors of a robotic surgical instrument. In certain examples, multiple motors of a surgical instrument or tool may share one or more common control modules, such as the common control module 610. In certain examples, multiple motors of a surgical instrument or tool can engage with the common control module 610 individually and selectively. In certain examples, the common control module 610 can selectively switch between interfacing with one of the multiple motors of the surgical instrument or tool and interfacing with another of the multiple motors of the surgical instrument or tool.

[0079] In at least one example, the common control module 610 can be selectively switched between an operable engagement with the articulation motors 606a, 606b and an operable engagement with either the firing motor 602 or the closing motor 603. In at least one example, as shown in Figure 8, the switch 614 can move or transition between multiple positions and / or states. For example, in the first position 616, the switch 614 may electrically connect the common control module 610 to the firing motor 602; in the second position 617, the switch 614 may electrically connect the common control module 610 to the closing motor 603; in the third position 618a, for example, the switch 614 may electrically connect the common control module 610 to the first articulation motor 606a; and in the fourth position 618b, the switch 614 may electrically connect the common control module 610 to the second articulation motor 606b. In certain examples, a separate common control module 610 may also be electrically connected to the launch motor 602, the closing motor 603, and the joint motion motors 606a, 606b. In certain examples, the switch 614 may be a mechanical switch, an electromechanical switch, a solid switch, or any preferred switching mechanism.

[0080] Each of the motors 602, 603, 606a, and 606b may be equipped with a torque sensor for measuring the output torque on the motor shaft. The force on the end effector may be sensed in any conventional manner, such as by force sensors on the outside of the jaws or by torque sensors on the motors that actuate the jaws.

[0081] In various examples, as shown in Figure 8, the common control module 610 may include a motor drive unit 626 which may include one or more H-bridge FETs. The motor drive unit 626 may modulate the power transmitted from the power supply 628 to the motor connected to the common control module 610, for example, based on input from a microcontroller 620 ("controller"). In certain examples, as described herein, the current drawn by the motor can be determined, for example, using the microcontroller 620 while the motor is connected to the common control module 610.

[0082] In certain examples, the microcontroller 620 may include a microprocessor 622 ("processor") and one or more non-temporary computer-readable media or memory units 624 ("memory"). In certain examples, the memory 624 may store various program instructions, which, when executed, cause the processor 622 to perform some of the functions and / or calculations described herein. In certain examples, one or more of the memory units 624 may be linked to the processor 622, for example.

[0083] In certain examples, the power supply 628 can be used to power, for example, a microcontroller 620. In certain examples, the power supply 628 may include, for example, a battery (or "battery pack" or "power pack") such as a lithium-ion battery. In certain examples, the battery pack may be configured to be releasably mounted on a handle to power a surgical instrument 600. Several battery cells connected in series may be used as the power supply 628. In certain examples, the power supply 628 may be, for example, replaceable and / or rechargeable.

[0084] In various examples, the processor 622 can control the motor drive unit 626 to control the position, direction of rotation, and / or speed of a motor connected to a common control module 610. In specific examples, the processor 622 can signal the motor drive unit 626 to stop and / or disable a motor connected to a common control module 610. The term “processor,” as used herein, should be understood to include any suitable microprocessor, microcontroller, or other basic computing device that integrates the functions of a computer’s central processing unit (CPU) on one or up to several integrated circuits. A processor can be a multipurpose programmable device that receives digital data as input, processes that data according to instructions stored in memory, and provides the results as output. It may have internal memory and can therefore be an example of sequential digital logic. A processor may operate with numbers and symbols represented in binary.

[0085] The processor 622 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In a particular example, the microcontroller 620 may be, for example, the LM 4F230H5QR available from Texas Instruments. In at least one example, the Texas Instruments LM4F230H5QR is an ARM Cortex-M4F processor core that includes, among other features readily available in the product datasheet, 256KB of on-chip memory of single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and one or more 12-bit ADCs with 12 analog input channels. Other microcontrollers may be readily substituted for use with module 4410. Therefore, this disclosure should not be limited to this context.

[0086] Memory 624 may include program instructions for controlling each of the motors of the surgical instrument 600, which can be connected to a common control module 610. For example, memory 624 may include program instructions for controlling the firing motor 602, the closing motor 603, and the joint movement motors 606a, 606b. Such program instructions can cause the processor 622 to control the firing function, closing function, and joint movement function according to input from an algorithm or control program of the surgical instrument or tool.

[0087] For example, one or more mechanisms and / or sensors, such as sensor 630, can be used to alert processor 622 of program instructions to be used in a particular setting. For example, sensor 630 may alert processor 622 to use program instructions related to the firing, closing, and articulation of the end effector. In a particular example, sensor 630 may include a position sensor that can be used, for example, to sense the position of switch 614. Thus, processor 622 may use program instructions associated with the firing of the I-beam of the end effector when, for example, it detects through sensor 630 that switch 614 is in the first position 616, and processor 622 may use program instructions associated with the closing of the anvil when, for example, it detects through sensor 630 that switch 614 is in the second position 617, and processor 622 may use program instructions associated with the articulation of the end effector when, for example, it detects through sensor 630 that switch 614 is in the third position 618a or the fourth position 618b.

[0088] Figure 9 shows a diagram of a situation-aware surgical system 5100 according to at least one aspect of the present disclosure. In some examples, the data source 5126 may include, for example, a modular device 5102 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 5122 (e.g., an EMR database containing patient records), and a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor). The surgical hub 5104 may be configured to derive contextual information about a surgical procedure from the data, for example, based on a specific combination(s) of received data or a specific order in which the data is received from the data source 5126. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, a specific step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This function of the surgical hub 5104 for deriving or inferring information related to surgical procedures from received data may be referred to as “situational awareness.” In one example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104, for deriving contextual information related to surgical procedures from received data.

[0089] The situational awareness system of the surgical hub 5104 can be configured to derive contextual information from data received from the data source 5126 in various different ways. For example, the situational awareness system may include a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 5122, patient monitoring device 5124, and / or modular device 5102) with corresponding contextual information about surgical procedures. In other words, the machine learning system can be trained to accurately derive contextual information about surgical procedures from provided inputs. In an embodiment, the situational awareness system may include a lookup table that stores pre-characterized contextual information about surgical procedures, associated with one or more inputs (or ranges of inputs) corresponding to that contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situational awareness system to control the modular device 5102. In the embodiment, context information received by the situation awareness system of the surgical hub 5104 can be associated with a specific control adjustment of one or more modular devices 5102, or a set of control adjustments. In the embodiment, the situation awareness system may include a further machine learning system, a lookup table, or other such system that generates or retrieves one or more control adjustments of one or more modular devices 5102 when context information is provided as input.

[0090] A surgical hub 5104 incorporating a situation awareness system can provide several advantages to a surgical system 5100. One advantage may include improving the interpretation of sensed and collected data, which in turn can improve the processing accuracy during a surgical procedure and / or the use of the data. Returning to a previous example, the situation awareness surgical hub 5104 can determine what type of tissue is being operated on, and thus, when an unexpectedly high force for closing the end effector of a surgical instrument is detected, the situation awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the type of tissue.

[0091] The type of tissue being operated on can affect the adjustments made to the compression speed and load thresholds of surgical stapling and cutting instruments for specific tissue gap measurements. The situation awareness surgical hub 5104 can estimate whether the surgical procedure being performed is a thoracic procedure or an abdominal procedure, whereby the surgical hub 5104 can determine whether the tissue clamped by the end effectors of the surgical stapling and cutting instruments is a lung (in the case of a thoracic procedure) or a stomach (in the case of an abdominal procedure). Then, the surgical hub 5104 can appropriately adjust the compression speed and load thresholds of the surgical stapling and cutting instruments according to the type of tissue.

[0092] The type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situation awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure utilizes insufflation), and can determine the type of procedure. Generally, since a certain type of procedure can be performed within a specific body cavity, the surgical hub 5104 can appropriately control the motor speed of the smoke evacuator according to the body cavity being operated on. Thus, the situation awareness surgical hub 5104 can provide a consistent amount of smoke evacuation for both thoracic and abdominal surgeries.

[0093] The type of procedure being performed may affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, in arthroscopy, the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid, and therefore a higher energy level may be required. The situational awareness surgical hub 5104 can determine whether the surgical procedure is an arthroscopy. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on may affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situational awareness surgical hub 5104 can determine what type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue shape for the surgical procedure. Furthermore, the situation-aware surgical hub 5104 can be configured to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument not simply for each procedure, but throughout the course of the surgical procedure. The situation-aware surgical hub 5104 can determine which steps of the surgical procedure are being performed or will be performed, and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type according to the steps of the surgical procedure.

[0094] In the embodiment, the surgical hub 5104 can derive data from an additional data source 5126 to improve conclusions drawn from one data source 5126. The context-aware surgical hub 5104 can enhance data received from the modular device 5102 with contextual information constructed from other data sources 5126 regarding the surgical procedure. For example, the context-aware surgical hub 5104 can be configured to determine whether hemostasis has been achieved (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, video or image data may not be conclusive. Therefore, in one example, the surgical hub 5104 can be further configured to make a determination regarding the integrity of staple lines or tissue welds by comparing physiological measurements (e.g., blood pressure sensed by a BP monitor communicably connected to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (Figure 2) communicably connected to the surgical hub 5104). In other words, the context-aware system of the surgical hub 5104 can take physiological measurement data into consideration to provide additional context when analyzing visualization data. This additional context can be useful when visualization data may not be conclusive or may be incomplete on its own.

[0095] For example, the situational awareness surgical hub 5104 can proactively activate the generator to which the RF electrosurgical instrument is connected if it is determined that the instrument will be needed in a subsequent step of the procedure. By proactively activating the energy source, the instrument can be ready for use as soon as the preceding step of the procedure is completed.

[0096] The situational awareness surgical hub 5104 can determine, according to the shape(s) of the surgical site that the surgeon is expected to need to see, whether the current or subsequent steps of the surgical procedure require different views or magnifications on the display. The surgical hub 5104 can then appropriately and proactively change the displayed view (e.g., supplied from a medical imaging device for the visualization system 108), so that the display automatically adjusts throughout the surgical procedure.

[0097] The situational awareness surgical hub 5104 can determine which steps of a surgical procedure are being performed or will be performed next, and whether specific data or comparisons of data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0098] Errors may be detected during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situation-aware surgical hub 5104 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read the corresponding checklist, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 5104 has determined to be performed. In some examples, the surgical hub 5104 can be configured to compare a list of items for the procedure and / or a list of devices paired with the surgical hub 5104 to a recommended or expected manifest of items and / or devices for a given surgical procedure. If any discontinuities exist between lists, the surgical hub 5104 can be configured to provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, and / or other surgical items are missing. In some examples, the surgical hub 5104 may be configured to determine the relative distance or relative position of a modular device 5102 and a patient monitoring device 5124, for example, by proximity sensors. The surgical hub 5104 can compare the relative positions of the devices to a recommended or expected layout for a particular surgical procedure. If any discontinuity exists between the layouts, the surgical hub 5104 may be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout.

[0099] The situational awareness surgical hub 5104 can determine whether a surgeon (or other healthcare professional) is making an error or deviating from a set of actions expected during a surgical procedure. For example, the surgical hub 5104 may be configured to determine the type of surgical procedure being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the surgical procedure with the steps or instruments expected for the type of surgical procedure that the surgical hub 5104 has determined is being performed. In some examples, the surgical hub 5104 may be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgical procedure.

[0100] Surgical instruments (and other modular devices 5102) may be adjusted to the specific context of each surgical procedure (e.g., adjustment for different tissue types) to verify their operation during the surgical procedure. Subsequent steps, data, and display adjustments may be provided to the surgical instruments (and other modular devices 5102) in the surgical field according to the specific context of the procedure.

[0101] Figure 10 shows an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each step of the surgical procedure. Refer also to Figure 9 in the following description of the timeline 5200 shown in Figure 9. The timeline 5200 may show typical steps that nurses, surgeons, and other healthcare professionals might take during a lung segmentectomy, beginning with setting up the operating room and ending with transferring the patient to the postoperative recovery room. The contextually aware surgical hub 5104 may receive data from data source 5126 throughout the surgical procedure, including data generated each time a healthcare professional uses a modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 receives this data from the paired modular device 5102 and other data sources 5126, and can continuously derive estimations (i.e., contextual information) about the procedure in progress as new data is received, such as which steps of the procedure are being performed at any given time. The contextual awareness system of the surgical hub 5104 may, for example, record data about the procedure to generate a report, verify the steps being taken by the healthcare worker, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions as specified herein.

[0102] As a first step 5202 in this exemplary procedure, hospital staff may read the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic procedure. In a second step 5204, staff may scan incoming medical equipment for the procedure. The surgical hub 5104 cross-references the scanned equipment with a list of equipment that may be used in various types of procedures to confirm that the combination of equipment corresponds to a thoracic procedure. Furthermore, the surgical hub 5104 may also determine that the procedure is not a wedge resection because the incoming materials either do not contain specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection. In a third step 5206, healthcare workers may scan a patient band via a scanner 5128 that is communicably connected to the surgical hub 5104. Next, the surgical hub 5104 can identify the patient based on the scanned data. In the fourth step 5208, the medical staff turns on the auxiliary equipment. The auxiliary equipment used may vary depending on the type of surgical procedure and the technique used by the surgeon, but in this exemplary case, examples include a fume exhauster, an air insulator, and a medical imaging device. When activated, the auxiliary equipment, which is a modular device 5102, can automatically pair with the surgical hub 5104, which may be located within a specific vicinity of the modular device 5102, as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 that is paired with it during this pre-operative or initialization phase. In this particular embodiment, the surgical hub 5104 may determine that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, a list of medical supplies to be used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure to be performed by the surgical team.Once the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing. In the fifth step 5210, personnel attach the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 may be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, it may verify that the patient is in the operating room, for example, as described in process 5207. In the sixth step 5212, healthcare personnel may administer anesthesia to the patient. The surgical hub 5104 can estimate that the patient is under anesthesia based on data from the modular device 5102 and / or the patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 5212 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion begins.

[0103] In the seventh step 5214, the lung of the patient being operated on may collapse (while ventilation is switched to the contralateral lung). The surgical hub 5104 can, for example, infer from the ventilator data that the patient's lung has collapsed. The surgical hub 5104 can compare the detection of the patient's lung collapse with the expected steps of the procedure (which can be accessed or read in advance), so it can infer that the surgical portion of the procedure has commenced, and thereby determine that lung collapse may be the first surgical step in this particular procedure. In the eighth step 5216, a medical imaging device 5108 (e.g., a scope) may be inserted, and video footage from the medical imaging device may be initiated. The surgical hub 5104 may receive medical imaging device data (i.e., video or image data) through a connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has commenced. Furthermore, the surgical hub 5104 can determine that a particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resections have not been taken into consideration by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Contextual information regarding the type of procedure being performed can be determined in various ways using data from the medical imaging device 124 (Figure 2), for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structures, by monitoring the number or type of medical imaging device being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used. For example, one technique for performing a VATS lobectomy may involve positioning the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while another technique for performing a VATS segmentectomy may involve positioning the camera in an anterior intercostal position relative to the segmental fissure. A situational awareness system can be trained, for example, using pattern recognition or machine learning techniques to recognize the positioning of medical imaging devices according to the visualization of a patient's anatomical structure.An exemplary technique for performing VATS lobectomy may utilize a single medical imaging device. An exemplary technique for performing VATS segmentectomy may utilize multiple cameras. An exemplary technique for performing VATS segmentectomy utilizes an infrared light source (which can be communicably connected to a surgical hub as part of a visualization system) to visualize the segmental fissure, but this is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0104] In step 9, 5218, the surgical team may initiate the incision step of the procedure. The surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the previously considered steps of the procedure are completed) corresponds to the incision step. In step 10, 5220, the surgical team may proceed to the ligation step of the procedure. The surgical hub 5104 may receive data from surgical stapling and cutting instruments indicating that instruments are being emitted, and can therefore infer that the surgeon is ligating arteries and veins. As in the previous step, the surgical hub 5104 can derive this inference by cross-referencing the received data from the surgical stapling and cutting instruments with the read-out steps in the process. In the eleventh step 5222, the segmental resection portion of the procedure can be performed. Based on data from the surgical stapling and cutting instruments, including data from its cartridge, the surgical hub 5104 can infer that the surgeon is transversely incising the parenchymal tissue. The cartridge data may correspond, for example, to the size or type of staples being fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data may indicate the type of tissue being stapled and / or transversely incised. In this case, the type of staples being fired is used for parenchymal tissue (or other similar tissue types), thereby allowing the surgical hub 5104 to infer that the segmental resection portion of the procedure is being performed. Next, in the twelfth step 5224, the nodule incision step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being fired, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leakage test.In this particular procedure, the RF or ultrasound instrument used after the parenchymal tissue has been transversely incised corresponds to the nodular incision step, thereby allowing the surgical hub 5104 to make this estimation. Since different instruments are better suited to specific tasks, the surgeon should note that they should periodically alternate between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasound) instruments depending on the specific step in the procedure. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Upon completion of the twelfth step 5224, the incision may be closed, and the postoperative portion of the procedure may begin.

[0105] In the 13th step 5226, the patient can be de-anesthetized. The surgical hub 5104 may estimate that the patient is waking from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase). Finally, the 14th step 5228 may be the step in which a healthcare worker removes various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 may estimate that the patient is being transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. As can be seen from this exemplary procedure description, the surgical hub 5104 can determine or estimate when each step of a given surgical procedure is occurring, according to the data received from various data sources 5126 that are communicably connected to the surgical hub 5104.

[0106] As shown in the first step 5202 of the timeline 5200 shown in Figure 10, in addition to estimating the type of surgical procedure to be performed using patient data from the EMR database(s), the patient data can also be used by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0107] Figure 11 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. In one aspect, the computer-implemented interactive surgical system may be configured to monitor and analyze data related to the operation of various surgical systems, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented interactive surgical system may include a cloud-based analytics system. The cloud-based analytics system may be described as a surgical system, but is not necessarily limited to that, and may generally be a cloud-based medical system. As shown in Figure 11, the cloud-based analytics system may include a plurality of surgical instruments 7012 (which may be the same as or similar to instrument 112), a plurality of surgical hubs 7006 (which may be the same as or similar to hub 106), and a surgical data network 7001 (which may be the same as or similar to network 201) for connecting the surgical hubs 7006 to a cloud 7004 (which may be the same as or similar to cloud 204). Each of the plurality of surgical hubs 7006 may be communicably connected to one or more surgical instruments 7012. Hub 7006 may also be communicably connected to a cloud 7004 of a computer-implemented interactive surgical system via network 7001. Cloud 7004 may be a remote, centralized source of hardware and software for storing, manipulating, and communicating data generated based on the operation of various surgical systems. As shown in Figure 11, access to Cloud 7004 may be achieved via network 7001, which may be the Internet or another suitable computer network. The surgical hub 7006, which may be connected to Cloud 7004, can be considered the client side of a cloud computing system (i.e., a cloud-based analytics system). Surgical instruments 7012 may be paired with the surgical hub 7006 for controlling and performing various surgical procedures or actions described herein.

[0108] In addition, the surgical instrument 7012 may include transceivers (which may also include transceivers) for transmitting data to and from the corresponding surgical hub 7006. The combination of the surgical instrument 7012 and the corresponding hub 7006 may indicate a specific location, such as an operating room within a medical facility (e.g., a hospital) for providing medical surgery. For example, the memory of the surgical hub 7006 may store location data. As shown in Figure 11, the cloud 7004 includes a central server 7013 (which may be the same as or similar to the remote server 7013), a hub application server 7002, a data analysis module 7034, and an input / output ("I / O") interface 7006. The central server 7013 of the cloud 7004 collectively manages the cloud computing system, which includes monitoring requests from client surgical hubs 7006 and managing the processing capacity of the cloud 7004 to perform those requests. Each of the central servers 7013 may include one or more processors 7008 coupled to a preferred memory device 7010, which may include volatile memory such as random-access memory (RAM) and non-volatile memory such as magnetic storage devices. The memory device 7010 may include machine-executable instructions that, when executed, cause the processor 7008 to run a data analysis module 7034 for cloud-based data analysis, operation, recommendation, and other operations described below. Furthermore, the processor 7008 can run the data analysis module 7034 independently or in conjunction with a hub application that runs independently by the hub 7006. The central server 7013 may also include a database 2212 of aggregated medical data that may reside in memory 2210.

[0109] Based on connections to various surgical hubs 7006 via network 7001, cloud 7004 can aggregate data from various surgical instruments 7012 and specific data generated by their corresponding hubs 7006. Such aggregated data may be stored in the aggregated medical database 7012 of cloud 7004. Specifically, cloud 7004 may, advantageously, perform data analysis and operations on the aggregated data to derive insights and / or perform functions that individual hubs 7006 cannot achieve on their own. For this purpose, as shown in Figure 11, cloud 7004 and surgical hubs 7006 are communicated together to send and receive information. The I / O interface 7006 is connected to multiple surgical hubs 7006 via network 7001. In this way, the I / O interface 7006 can be configured to transfer information between the surgical hubs 7006 and the aggregated medical data database 7011. Thus, the I / O interface 7006 can facilitate read / write operations of the cloud-based analysis system. Such read / write operations may be performed in response to requests from the hub 7006. These requests may be sent to the hub 7006 through the hub application. The I / O interface 7006 may include one or more high-speed data ports, including a universal serial bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 7004 to the hub 7006. The hub application server 7002 of the cloud 7004 may be configured to host and supply shared functions to a software application (e.g., a hub application) running on the surgical hub 7006. For example, the hub application server 7002 may manage requests from the hub application through the hub 7006, control access to the aggregated medical data database 7011, and perform load balancing. The data analysis module 7034 will be described in more detail with reference to Figure 12.

[0110] The configurations of the specific cloud computing systems described in this disclosure may be designed to address a variety of problems that arise in the context of medical surgeries and procedures performed using medical devices such as surgical instruments 7012, 112. Specifically, surgical instrument 7012 may be a digital surgical device configured to interact with the cloud 7004 in order to implement techniques to improve surgical outcomes. Various surgical instruments 7012 and / or surgical hubs 7006 may include a touch-controlled user interface so that a clinician can control the manner of interaction between the surgical instruments 7012 and the cloud 7004. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0111] FIG. 12 is a block diagram showing the functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. The cloud-based analysis system may include a plurality of data analysis modules 7034 that can be executed by a processor 7008 of a cloud 7004 to provide data analysis solutions for problems that particularly occur in the medical field. As shown in FIG. 12, the functions of the cloud-based data analysis module 7034 may be assisted via a hub application 7014 hosted by a hub application server 7002 accessible on a surgical hub 7006. The cloud processor 7008 and the hub application 7014 may operate in cooperation to execute the data analysis module 7034. An application program interface (API) 7016 may define a set of protocols and routines corresponding to the hub application 7014. Additionally, the API 7016 may manage the storage and retrieval of data to and from a medical database 7012 aggregated for the operation of the application 7014. A cache 7018 may also store data (e.g., temporarily) and may be coupled to the API 7016 for more efficient retrieval of data used by the application 7014. The data analysis module 7034 of FIG. 12 may include modules for resource optimization 7020, data collection and aggregation 7022, authorization and security 7024, control program update 7026, patient outcome analysis 7028, recommendations 7030, and data classification and prioritization 7032. Other suitable data analysis modules may also be implemented by the cloud 7004 in some aspects. In one aspect, the data analysis module may be used for specific recommendations based on the analysis of trends, results, and other data.

[0112] For example, the data collection and aggregation module 7022 may be used to generate self-describing data (e.g., metadata), including the identification of prominent features or configurations (e.g., trends), the management of redundant datasets, and the storage of data into paired datasets that can be grouped by surgical procedures but do not necessarily correspond to actual surgical procedure dates and surgeons. In particular, paired datasets generated from the operation of surgical instruments 7012 may include applying a binary classification, such as bleeding or non-bleeding events. More generally, the binary classification may be characterized as either a desirable event (e.g., a successful surgical procedure) or an undesirable event (e.g., a surgical instrument 7012 misfired or misused). The aggregated self-describing data may correspond to individual data received from various groups or subgroups of the surgical hub 7006. Thus, the data collection and aggregation module 7022 can generate aggregated metadata or other organized data based on the raw data received from the surgical hub 7006. For this purpose, the processor 7008 can be operationally linked to the hub application 7014 and the aggregated medical data database 7011 in order to run the data analysis module 7034. The data acquisition and aggregation module 7022 may store the aggregated and organized data in the aggregated medical data database 2212.

[0113] The resource optimization module 7020 can be configured to analyze this aggregated data to determine the optimal use of resources for a particular healthcare facility or group of healthcare facilities. For example, the resource optimization module 7020 can determine the optimal reorder point for surgical staple fasteners 7012 for a group of healthcare facilities based on the corresponding predicted demand for those staple fasteners 7012. The resource optimization module 7020 can also evaluate the resource use or other operational configurations of various healthcare facilities to determine whether resource use can be improved. Similarly, the recommendation module 7030 can be configured to analyze the aggregated data from the data collection and aggregation module 7022 to provide recommendations. For example, the recommendation module 7030 can recommend to a healthcare facility (e.g., a healthcare service provider such as a hospital) that a particular surgical instrument 7012 should be upgraded to an improved version based, for example, on the error rate being higher than predicted. Additionally, the recommendation module 7030 and / or the resource optimization module 7020 can recommend better supply chain parameters, such as product reorder points, and provide suggestions for different surgical instruments 7012, their use, or procedural steps to improve surgical outcomes. Healthcare facilities can receive such recommendations via the corresponding surgical hub 7006. More specific recommendations regarding the parameters or configurations of various surgical instruments 7012 can also be provided. Each hub 7006 and / or surgical instrument 7012 may also have a display screen that shows data or recommendations provided by the cloud 7004.

[0114] The patient outcome analysis module 7028 can analyze surgical outcomes associated with the currently used operating parameters of the surgical instrument 7012. The patient outcome analysis module 7028 may also analyze and evaluate other potential operating parameters. In this regard, the recommendation module 7030 may use these other potential operating parameters to make recommendations based on the fact that they result in better surgical outcomes, such as better sealing or less bleeding. For example, the suggestion module 7030 may send a suggestion to the surgical instrument 7006 regarding the use of a particular cartridge with the corresponding stapled surgical instrument 7012. Thus, the cloud-based analysis system may be configured to analyze large amounts of raw data collected while controlling for common variables and to provide centralized recommendations across multiple healthcare facilities (favorably determined based on aggregated data). For example, the cloud-based analysis system can analyze, evaluate, and / or aggregate things such as the type of medical procedure, the type of patient, the number of patients, and geographical similarities among healthcare providers using similar types of instruments in a way that a single healthcare facility cannot analyze independently. The control program update module 7026 can be configured to implement recommendations for various surgical instruments 7012 when the corresponding control program is updated. For example, the patient outcome analysis module 7028 can identify correlations linking specific control parameters to successful (or unsuccessful) outcomes. Such correlations may be addressed when the updated control program is transmitted to the surgical instrument 7012 via the control program update module 7026. Updates to the instrument 7012, which may be transmitted via the corresponding hub 7006, may incorporate aggregated outcome data collected and analyzed by the data collection and aggregation module 7022 of the cloud 7004. Additionally, the patient outcome analysis module 7028 and the recommendation module 7030 can identify improved ways of using the instrument 7012 based on the aggregated outcome data.

[0115] The cloud-based analytics system may include security features implemented by Cloud 7004. These security features may be managed by the authorization and security module 7024. Each surgical hub 7006 may have associated unique credentials, such as a username, password, and other preferred security credentials. These credentials may be stored in memory 7010 and associated with authorized cloud access levels. For example, based on providing accurate credentials, a surgical hub 7006 may be granted access to communicate with the cloud to a predetermined extent (e.g., to send or receive certain defined types of information). For this purpose, the aggregated medical data database 7011 of Cloud 7004 may include a database of authorized credentials to verify the accuracy of the provided credentials. Different credentials may be associated with various levels of authorization for interaction with Cloud 7004, such as predetermined access levels for receiving data analysis generated by Cloud 7004. Furthermore, for security purposes, the cloud may maintain a database of hubs 7006, instruments 7012, and other devices, which may include a "blacklist" of prohibited devices. Specifically, surgical hubs 7006 listed on the blacklist may not be permitted to interact with the cloud, while surgical instruments 7012 listed on the blacklist may not have functional access to the corresponding hub 7006 and / or may be prevented from fully functioning when paired with the corresponding hub 7006. Additionally or alternatively, the cloud 7004 may flag instruments 7012 based on non-conformity or other specified criteria. In this way, counterfeit medical devices and the improper reuse of such devices across the entire cloud-based analysis system can be identified and addressed.

[0116] The surgical instrument 7012 may use a wireless transceiver to transmit a wireless signal that may represent, for example, authorization credentials for access to the corresponding hub 7006 and the cloud 7004. A wired transceiver may also be used to transmit a signal. Such authorization credentials can be stored in the respective memory devices of the surgical instrument 7012. The authorization and security module 7024 can determine whether the authorization credentials are accurate or forged. The authorization and security module 7024 may also dynamically generate authorization credentials for enhanced security. The credentials can also be encrypted, for example, by using hash-based encryption. Upon transmitting appropriate authorization, the surgical instrument 7012 may signal to the corresponding hub 7006 and ultimately the cloud 7004 to indicate that the instrument 7012 is ready to acquire and transmit medical data. In response, the cloud 7004 may transition to a state in which it can receive medical data for storage in the aggregated medical data database 7011. This readiness for data transmission can be indicated, for example, by an optical indicator on the instrument 7012. Cloud 7004 can also send signals to surgical instruments 7012 to update their associated control programs. Cloud 7004 can send signals directed to specific categories of surgical instruments 7012 (e.g., electrosurgical instruments) to ensure that software updates for control programs are sent only to the appropriate surgical instruments 7012. Furthermore, Cloud 7004 may be used to implement system-wide solutions to address local or global issues based on selective data transmission and authorization credentials. For example, if a group of surgical instruments 7012 are identified as having a common manufacturing defect, Cloud 7004 may modify the authorization credentials corresponding to this group to implement operational lockout for that group.

[0117] A cloud-based analytics system can enable monitoring of multiple healthcare facilities (e.g., healthcare facilities such as hospitals) to determine improved practices and recommend changes accordingly (e.g., via suggestion module 2030). Thus, the processor 7008 of Cloud 7004 can analyze data associated with individual healthcare facilities to identify facilities and aggregate that data with other data associated with other healthcare facilities. Groups can be defined, for example, based on similar operational practices or geographical location. In this way, Cloud 7004 may provide a broad range of analytics and recommendations to healthcare facility groups. The cloud-based analytics system can also be used for enhanced contextual awareness. For example, the processor 7008 may predictively model the effect of cost and effectiveness recommendations for a particular facility (compared to overall operations and / or various medical procedures). The cost and effectiveness associated with that particular facility can also be compared to the corresponding local areas of other facilities or any other equivalent facilities.

[0118] The data classification and prioritization module 7032 may prioritize and classify data based on severity (e.g., the severity, unexpectedness, or suspiciousness of the medical event associated with the data). This classification and prioritization may be used in conjunction with the functions of other data analysis modules 7034 described herein to improve the cloud-based analysis and operations described herein. For example, the data classification and prioritization module 7032 can assign priorities to data analyses performed by the data collection and aggregation module 7022 and the patient outcome analysis module 7028. Different priority levels may result in specific responses from the cloud 7004 (corresponding to the level of urgency), such as increased priority for rapid response, special processing, exclusion from the aggregated medical data database 7011, or other preferred responses. Furthermore, if necessary, the cloud 7004 may send requests (e.g., push messages) via the hub application server for additional data from the corresponding surgical instrument 7012. Push messages may result in notifications displayed on the corresponding hub 7006 to request support or additional data. This push message may be needed when the cloud detects a significant anomaly or outlier and the cloud is unable to determine the cause of that anomaly. The central server 7013 may be programmed to trigger this push message in certain critical situations, such as when data is determined to differ from a predicted value by a predetermined threshold, or when security is suspected to be involved.

[0119] Further illustrative details regarding the various functions described are provided in the following descriptions. Each of the descriptions may utilize a cloud architecture, as shown in Figures 11 and 12, as an example of a hardware and software implementation.

[0120] Figure 13 shows a block diagram of a computer-implemented adaptive surgical system 9060 configured to adaptively generate control program updates for modular devices 9050, according to at least one aspect of the present disclosure. In some examples, the surgical system may include a surgical hub 9000, a plurality of modular devices 9050 communicably connected to the surgical hub 9000, and an analysis system 9100 communicably connected to the surgical hub 9000. While a single surgical hub 9000 may be shown, it should be noted that the surgical system 9060 may include any number of surgical hubs 9000 that can be connected to form a network of surgical hubs 9000 communicably connected to the analysis system 9010. In some examples, the surgical hub 9000 may include a processor 9010 connected to memory 9020 for executing stored instructions, and a data relay interface 9030 to which data is transmitted to the analysis system 9100. In some examples, the surgical hub 9000 may further include a user interface 9090 having an input device 9092 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user, and an output device 9094 (e.g., a display screen) for providing output to the user. The output may include data from queries entered by the user, suggestions for products or mixtures of products for use in a given procedure, and / or instructions for actions to be performed before, during, or after a surgical procedure. The surgical hub 9000 may further include an interface 9040 for communicatively connecting modular devices 9050 to the surgical hub 9000. In one embodiment, the interface 9040 may include a transceiver that can be communicatively connected to the modular devices 9050 via a wireless communication protocol. The modular devices 9050 may include, for example, surgical stapling and cutting instruments, electrosurgical instruments, ultrasound instruments, inhalers, ventilators, and display screens. In some examples, the surgical hub 9000 can be further connected to one or more patient monitoring devices 9052, such as an EKG monitor or BP monitor, in a communicative manner.In some examples, the surgical hub 9000 can be further connected to one or more databases 9054 or external computer systems, such as the EMR database of the medical facility where the surgical hub 9000 is located, in a communicative manner.

[0121] When the modular device 9050 is connected to the surgical hub 9000, the surgical hub 9000 can sense or receive perioperative data from the modular device 9050 and then associate the received perioperative data with surgical procedure outcome data. The perioperative data may indicate how the modular device 9050 was controlled during the course of the surgical procedure. The procedure outcome data includes data associated with the outcome from the surgical procedure (or the process thereof), and this data may include whether the surgical procedure (or the process thereof) had a positive or negative outcome. For example, outcome data may include whether the patient suffered postoperative complications from a particular procedure, or whether there was leakage (e.g., bleeding or air leakage) at a particular staple line or incision line. The surgical hub 9000 can obtain surgical procedure outcome data by receiving data from an external source (e.g., from the EMR database 9054), by directly detecting the outcome (e.g., via one of the connected modular devices 9050), or by estimating the occurrence of the outcome through a situational awareness system. For example, data on postoperative complications can be read from the EMR database 9054, and data on staple or incision line leakage can be directly detected or estimated by the situational awareness system. Surgical procedure outcome data can be estimated by the situational awareness system from data received from various data sources, including the modular device 9050 itself, the patient monitoring device 9052, and the database 9054 to which the surgical hub 9000 is connected.

[0122] The surgical hub 9000 can transmit data and result data from associated modular devices 9050 to the analysis system 9100 for processing on the analysis system 9100. By transmitting both perioperative data showing how the modular devices 9050 are controlled and procedure result data, the analysis system 9100 can correlate different modes of control of the modular devices 9050 with surgical outcomes for specific procedure types. In some examples, the analysis system 9100 may include a network of analysis servers 9070 configured to receive data from the surgical hub 9000. Each of the analysis servers 9070 may include memory and a memory-contiguous processor that executes instructions stored in memory to analyze the received data. In some examples, the analysis servers 9070 may be connected in a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 9100 can then learn the optimal or preferred operating parameters for various types of modular devices 9050, generate adjustments to the control programs of the modular devices 9050 in the field, and then transmit (or "push") updates to the control programs of the modular devices 9050.

[0123] Further details regarding the computer-implemented interactive surgical system 9060, including the surgical hub 9000 and various modular devices 9050 that can be connected thereto, are described in relation to Figures 5 and 6.

[0124] Figure 14 provides a surgical system 6500 according to the present disclosure, which may include a surgical instrument 6502 capable of communicating with a console 6522 or a portable device 6526 via a local area network 6518 or a cloud network 6520 via a wired or wireless connection. In various embodiments, the console 6522 and the portable device 6526 may be any suitable computing device. The surgical instrument 6502 may include a handle 6504, an adapter 6508, and a loading unit 6514. The adapter 6508 is releasably coupled to the handle 6504, and the loading unit 6514 is releasably coupled to the adapter 6508 so that the adapter 6508 transmits force from the drive shaft to the loading unit 6514. The adapter 6508 or the loading unit 6514 may include an internally disposed force gauge (not explicitly shown) for measuring the force exerted on the loading unit 6514. The loading unit 6514 may include an end effector 6530 having a first jaw 6532 and a second jaw 6534. The loading unit 6514 may also be a multi-firing loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without having to remove the loading unit 6514 from the surgical site to reload the loading unit 6514.

[0125] The first jaw 6532 and the second jaw 6534 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 6532 may be configured to include a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) which may be configured to fire at least one fastener multiple times, or which may be fired two or more times before being replaced. The second jaw 6534 may include an anvil that deforms the fasteners or otherwise secures them around the tissue as they are ejected from the multi-fire fastener cartridge.

[0126] The handle 6504 may include a motor connected to the drive shaft so as to affect the rotation of the drive shaft. The handle 6504 may include a control interface for selectively operating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces that a clinician can engage to operate the motor.

[0127] The control interface of the handle 6504 may communicate with the controller 6528 of the handle 6504 to selectively operate the motor and affect the rotation of the drive shaft. The controller 6528 may be located within the handle 6504 and is configured to receive input from the control interface and adapter data from the adapter 6508 or loading unit data from the loading unit 6514. The controller 6528 may analyze the input from the control interface and the data received from the adapter 6508 and / or loading unit 6514 in order to selectively operate the motor. The handle 6504 may also include a display that can be viewed by a clinician while the handle 6504 is in use. The display may be configured to show a portion of the adapter or loading unit data before, during, or after firing the instrument 6502.

[0128] The adapter 6508 may include an internally disposed adapter identification device 6510, and the loading unit 6514 may include an internally disposed loading unit identification device 6516. The adapter identification device 6510 may communicate with the controller 6528, and the loading unit identification device 6516 may communicate with the controller 6528. It will be understood that the loading unit identification device 6516 may communicate with the adapter identification device 6510 which relays or bypasses communication from the loading unit identification device 6516 to the controller 6528.

[0129] The adapter 6508 may also include a plurality of sensors 6512 (one shown) disposed around it to detect various conditions of the adapter 6508 or the environment (e.g., when the adapter 6508 is connected to a loading unit, when the adapter 6508 is connected to a handle, when the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside the adapter 6508, the number of times the adapter 6508 has fired, the peak force of the adapter 6508 during firing, the total amount of force applied to the adapter 6508, the peak recoil force of the adapter 6508, the number of pauses of the adapter 6508 during firing, etc.). The plurality of sensors 6512 may provide input to the adapter identification device 6510 in the form of data signals. The data signals from the plurality of sensors 6512 may be stored in the adapter identification device 6510 or used to update adapter data stored in the adapter identification device 6510. The data signals from the plurality of sensors 6512 may be analog or digital. Multiple sensors 6512 may include force gauges for measuring the force exerted on the loading unit 6514 during firing.

[0130] The handle 6504 and adapter 6508 can be configured to interconnect the adapter identification device 6510 and the loading unit identification device 6516 with the controller 6528 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals between them). Additionally or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductively). It is also intended that the adapter identification device 6510 and the controller 6528 may be able to wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0131] The handle 6504 may include a transmitter 6506 configured to transmit instrument data from the controller 6528 to other components of the system 6500 (e.g., LAN 6518, cloud 6520, console 6522, or portable device 6526). The transmitter 6506 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 6500. For example, the controller 6528 may transmit instrument data to the console 6528 that includes the serial number of an attached adapter (e.g., adapter 6508) attached to the handle 6504, the serial number of a loading unit attached to the adapter (e.g., loading unit 6514), and the serial number of a multi-fire fastener cartridge (e.g., multi-fire fastener cartridge) loaded into the loading unit. The console 6522 may then send back data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively, to the controller 6528. The controller 6528 can display a message on the local device display, or send a message via the transmitter 6506 to the console 6522 or portable device 6526, which can then display the message on the display 6524 or the portable device screen, respectively.

[0132] Figure 15A shows an exemplary flow for determining the operating mode and operating in the determined mode. Computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems may be configured to be updatable. Such updates may include the inclusion of features and benefits that were not available to the user before the update. These updates may be established by any method of hardware, firmware, and software updates suitable for introducing functionality to the user. For example, computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems may be updated using interchangeable / swappable (e.g., hot-swappable) hardware components, flashable firmware devices, and updatable software systems.

[0133] Renewal may be conditional on any preferred criteria or set of criteria. For example, renewal may be conditional on one or more hardware capabilities of the system, such as processing power, bandwidth, or resolution. For example, renewal may be conditional on one or more software aspects, such as the purchase of specific software code. For example, renewal may be conditional on a purchased service tier. A service tier may represent the functions and / or sets of functions that a user is entitled to use in connection with a computer-implemented interactive surgical system. A service tier may be determined by a license code, an e-commerce server authentication interaction, a hardware key, a username / password combination, a biometric authentication interaction, a public / private key exchange interaction, etc.

[0134] In 10704, system parameters / device parameters may be identified. System parameters / device parameters may be any element or set of elements on which updates are conditional. For example, a computer-implemented interactive surgical system may detect a specific bandwidth of communication between a modular device and a surgical hub. For example, a computer-implemented interactive surgical system may detect an indication of purchase of a specific service tier.

[0135] In 10708, the operating mode may be determined based on identified system parameters / device parameters. This determination may be made by a process that maps system parameters / device parameters to operating modes. The process may be manual and / or automated. The process may be the result of local and / or remote computing. For example, the operating mode may be determined based on identified system parameters / device parameters using client / server interaction. For example, the operating mode may be determined based on identified system parameters / device parameters using local software and / or locally embedded firmware. For example, the operating mode may be determined based on identified system parameters / device parameters using a hardware key, such as a secure microprocessor.

[0136] In 10710, operation may proceed according to a determined operating mode. For example, the system or device may proceed to operate in the default operating mode. For example, the system or device may proceed to operate in an alternative operating mode. The operating mode may be indicated by control hardware, firmware, and / or software already present in the system or device. The operating mode may be indicated by newly installed / updated control hardware, firmware, and / or software.

[0137] Figure 15B shows an exemplary functional block diagram for changing the operating mode. The upgradeable element 10714 may include an initialization component 10716. The initialization component 10716 may include any hardware, firmware, and / or software suitable for determining the operating mode. For example, the initialization component 10716 may be part of the system or device startup procedure. The initialization component 10716 may be involved in interactions to determine the operating mode of the upgradeable element 10714. For example, the initialization component 10716 may interact with, for example, user 10730, external resource 10732, and / or local resource 10718. For example, the initialization component 10716 may receive a license key from user 10730 to determine the operating mode. The initialization component 10716 may query an external resource 10732, such as a server, using the serial number of the upgradeable device 10714 to determine the operating mode. For example, the initialization component 10716 may query the local resource 10718, such as a local query to determine the amount of available bandwidth and / or a local query for a hardware key to determine the operating mode.

[0138] The upgradeable element 10714 may include one or more operational components 10720, 10722, 10726, 10728 and an operational pointer 10724. The initialization component 10716 may instruct the operational pointer 10724 to instruct the operational components 10720, 10722, 10726, 10728 corresponding to the determined operational mode to operate the upgradeable element 10741. The initialization component 10716 may instruct the operational pointer 10724 to instruct the operation of the upgradeable element to the default operational component 10720. For example, the default operational component 10720 may be selected on the condition that no other alternative operational modes have been determined. For example, the default operational component 10720 may be selected on the condition of a failure of the initialization component and / or a failure of interaction. The initialization component 10716 may instruct the operation pointer 10724 to instruct the resident operation component 10722 to operate the upgradeable element 10714. For example, a particular function may be present in the upgradeable element 10714 but requires startup to operate. The initialization component 10716 may instruct the operation pointer 10724 to instruct the operation of the upgradeable element 10714 to install a new operation component 10728 and / or a newly installed operation component 10726. For example, new software and / or firmware may be downloaded. The new software and / or firmware may include code to enable the function represented by the selected operating mode. For example, a new hardware component may be installed to enable the selected operating mode.

[0139] A surgical visualization system may include hierarchical access to specific capabilities. The surgical visualization system may be used to analyze at least a portion of the surgical field. The surgical visualization system may include a processor configured to operate in a first operating mode. The processor may receive control parameters and, based on those parameters, decide to operate in a second operating mode. The first operating mode may also relate to determining and / or displaying metrics representing the current state of moving particles in that portion of the surgical field. The second operating mode may also relate to determining and / or displaying metrics representing the aggregation state of moving particles in that portion of the surgical field, and / or metrics representing the state of moving particles at selectable tissue depths. The control parameters that determine the operating mode may include, for example, processing capacity or bandwidth, and / or system aspects such as the identification of an appropriate service layer for the surgical visualization system.

[0140] The surgical visualization system may include hierarchical access functionality. The surgical visualization system may be used to analyze at least a portion of the surgical field. Based on control parameters, the system may evaluate the current state of moving particles in that portion of the surgical field, evaluate the aggregation state of moving particles, and / or evaluate moving particles at selectable tissue depths. The control parameters may include system aspects such as processing capacity or bandwidth, and / or identification of appropriate service layers.

[0141] Figures 16A–16D and 17A–17F show various embodiments of an example of a visualization system 2108 that may be incorporated into a surgical system. The visualization system 2108 may include an imaging control unit 2002 and a hand unit 2020. The imaging control unit 2002 may include one or more illumination sources, power supplies for one or more illumination sources, one or more types of data communication interfaces (including USB, Ethernet, or wireless interfaces 2004), and one or more video outputs 2006. The imaging control unit 2002 may further include interfaces such as a USB interface 2010 configured to transmit integrated video and image acquisition data to a USB-enabled device. The imaging control unit 2002 may also include a processor unit, a temporary memory unit, a non-temporary memory unit, an image processing unit, a bus structure for forming a data linkage among computing components, and any interface (e.g., input and / or output) devices necessary to receive information from the imaging control unit and transmit information to components not included in the imaging control unit. The non-temporary memory may further include instructions that, when executed by the processor unit, allow data received from computing devices not included in the hand unit 2020 and / or imaging control unit to perform any number of operations.

[0142] The illumination source may include a white light source 2012 and one or more laser light sources. The imaging control unit 2002 may include one or more optical and / or electrical interfaces for optical and / or electrical communication with the hand unit 2020. The one or more laser light sources may, in non-limiting examples, include any one or more of a red laser light source, a green laser light source, a blue laser light source, an infrared laser light source, and an ultraviolet laser light source. In some non-limiting examples, the red laser light source may provide illumination with a peak wavelength that may be in the range of 635 nm to 660 nm, including values ​​at both ends. Non-limiting examples of red laser peak wavelengths may include about 635 nm, about 640 nm, about 645 nm, about 650 nm, about 655 nm, about 660 nm, or any value or range of values ​​in between. In some non-limiting examples, the green laser light source may provide illumination with a peak wavelength that may be in the range of 520 nm to 532 nm, including values ​​at both ends. Non-limiting examples of green laser peak wavelengths may include approximately 520 nm, 522 nm, 524 nm, 526 nm, 528 nm, 530 nm, 532 nm, or any value or range between them. In some non-limiting examples, a blue laser source may provide illumination with a peak wavelength that may be in the range of 405 nm to 445 nm, including both ends. Non-limiting examples of blue laser peak wavelengths may include approximately 405 nm, 410 nm, 415 nm, 420 nm, 425 nm, 430 nm, 435 nm, 440 nm, 445 nm, or any value or range between them. In some non-limiting examples, an infrared laser source may provide illumination with a peak wavelength that may be in the range of 750 nm to 3000 nm, including both ends. Non-limiting examples of infrared laser peak wavelengths may include approximately 750 nm, approximately 1000 nm, approximately 1250 nm, approximately 1500 nm, approximately 1750 nm, approximately 2000 nm, approximately 2250 nm, approximately 2500 nm, approximately 2750 nm, 3000 nm, or any value or range of values ​​in between. In some non-limiting examples, an ultraviolet laser light source may provide illumination with a peak wavelength that may be in the range of 200 nm to 360 nm, including values ​​at both ends.Non-limiting examples of ultraviolet laser peak wavelengths may include approximately 200 nm, approximately 220 nm, approximately 240 nm, approximately 260 nm, approximately 280 nm, approximately 300 nm, approximately 320 nm, approximately 340 nm, approximately 360 nm, or any value or range of values ​​in between.

[0143] In one non-limiting embodiment, the hand unit 2020 may include a body 2021, a camera scope cable 2015 attached to the body 2021, and an elongated camera probe 2024. The body 2021 of the hand unit 2020 may include hand unit control buttons 2022, or other control units that enable a medical professional to use the hand unit 2020 to control the operation of the hand unit 2020 or other components of the imaging control unit 2002, for example, a light source. The camera scope cable 2015 may include one or more conductors and one or more optical fibers. The camera scope cable 2015 may be terminated at a camera head connector 2008 at its proximal end, where the camera head connector 2008 is configured to mate with one or more optical and / or electrical interfaces of the imaging control unit 2002. The conductor may power the hand unit 2020, which includes the main body 2021 and the elongated camera probe 2024, and / or any electrical components inside the hand unit 2020, which includes the main body 2021 and / or the elongated camera probe 2024. The conductor may also function to provide bidirectional data communication between the hand unit 2020 and any one or more components of the imaging control unit 2002. One or more optical fibers may conduct illumination from one or more within the imaging control unit 2002 through the hand unit main body 2021 to the distal end of the illumination source of the elongated camera probe 2024. In some non-limiting embodiments, one or more optical fibers may also conduct light from the surgical site that has been reflected or refracted to one or more optical sensors disposed within the elongated camera probe 2024, the hand unit 2021, and / or the imaging control unit 2002.

[0144] Figure 16B (plan view) shows in more detail several embodiments of the hand unit 2020 of the visualization system 2108. The hand unit body 2021 may be made of plastic material. The hand unit control buttons 2022 or other control units may have a rubber overmolding to protect the control units while allowing the surgeon to operate them. The camera scope cable 2015 may have an optical fiber integrated with a conductor, and the camera scope cable 2015 may have a protective and flexible overcoat such as PVC. In some non-limiting examples, the camera scope cable 2015 may be about 10 feet long for ease of use during surgical procedures. The length of the camera scope cable 2015 may range from about 5 feet to about 15 feet. Non-limiting examples of the length of the camera scope cable 2015 may be about 5 feet, about 6 feet, about 7 feet, about 8 feet, about 9 feet, about 10 feet, about 11 feet, about 12 feet, about 13 feet, about 14 feet, about 15 feet, or any length or range of lengths in between. The elongated camera probe 2024 may be made from a rigid material such as stainless steel. In some embodiments, the elongated camera probe 2024 may be joined to the hand unit body 2021 via a rotatable collar 2026. The rotatable collar 2026 may allow the elongated camera probe 2024 to rotate relative to the hand unit body 2021. In some embodiments, the elongated camera probe 2024 may be terminated at its distal end with an epoxy-sealed plastic window 2028.

[0145] The side plan view of the hand unit shown in Figure 16C shows that the optical or image sensor 2030 may be located within the distal end 2032a of the elongated camera probe or within the hand unit body 2032b. In some alternative embodiments, the optical or image sensor 2030 may include additional optical elements located within the imaging control unit 2002. Figure 16C further shows an example of the optical sensor 2030, including a CMOS image sensor 2034 located within a mount 2036 having a radius of approximately 4 mm. Figure 16D shows an embodiment of the CMOS image sensor 2034, showing the active area 2038 of the image sensor. Although the CMOS image sensor in Figure 16C is shown located within a mount 2036 having a radius of approximately 4 mm, it can be recognized that such a sensor and mount combination may be of any useful size, located within the elongated camera probe 2024, the hand unit body 2021, or within the image control unit 2002. Some non-limiting examples of such alternative mounts may include a 5.5mm mount 2136a, a 4mm mount 2136b, a 2.7mm mount 2136c, and a 2mm mount 2136d. It can also be recognized that the image sensor may include a CCD image sensor. The CMOS sensor or CCD sensor may include an array of individual light-sensing elements (pixels).

[0146] Figures 17A to 17F show various embodiments of several examples of lighting sources and their control units that may be incorporated into the visualization system 2108.

[0147] Figure 17A shows one embodiment of a laser illumination system having multiple laser beams emitting multiple wavelengths of electromagnetic energy. As can be seen from the figure, the illumination system 2700 may include a red laser beam 2720, a green laser beam 2730, and a blue laser beam 2740, all optically connected together through an optical fiber 2755. As can be seen from the figure, each of the laser beams may have a corresponding photosensing element or electromagnetic sensor 2725, 2735, 2745 for sensing the output of a particular laser beam or wavelength.

[0148] Further disclosures relating to the laser illumination system shown in Figure 17A for use in the surgical visualization system 2108 can be found in U.S. Patent Application Publication No. 2014 / 0268860, entitled "CONTROLLING THE INTEGRAL LIGHT ENERGY OF A LASER PULSE," filed on March 15, 2014, and issued as U.S. Patent No. 9,777,913 on October 3, 2017, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0149] Figure 17B shows the operating cycle of the sensor used in rolling readout mode. The X direction corresponds to time, and the diagonal lines 2202 will be understood to indicate the activity of the internal pointer that reads each frame of data, the time for each row. The same pointer is involved in resetting each row of pixels for the next exposure period. The net integral times of each row 2219a-c are equivalent, but they are time-shifted relative to each other due to the rolling reset and readout process. Therefore, for any scenario where adjacent frames are required to represent light of different configurations, the only option for ensuring consistency across each row is to pulse the light between readout cycles 2230a-c. More specifically, the maximum available period corresponds to the sum of the blanking time and the time during which optical black or optical blind (OB) rows (2218, 2220) are served at the beginning or end of a frame.

[0150] Figure 17B shows the operating cycle of the sensor used in rolling readout mode or during sensor readout 2200. Frame readout may begin and be represented by a vertical line 2210. The readout period is represented by a diagonal or slanted line 2202. The sensor may be read out row by row, with the top of the downward slanted edge being the top row of the sensor 2212 and the bottom of the downward slanted edge being the bottom row of the sensor 2214. The time between the last row readout and the next readout cycle may be called blanking times 2216a-d. It should be understood that blanking times 2216a-d may be the same or different between successful readout cycles. Note that some of the sensor pixel rows may be covered with a light shield (e.g., a metallic coating or a substantially black layer of any other kind of material). These covered pixel rows may be referred to as optically black rows 2218 and 2220. The optically black rows 2218 and 2220 may be used as inputs for the correction algorithm.

[0151] As shown in Figure 17B, these optically black rows 2218 and 2220 may be located at the top or bottom of the pixel array, or at both the top and bottom of the pixel array. In some embodiments, it may be desirable to control the amount of electromagnetic radiation, e.g., light that is exposed to the pixels and thereby integrated or accumulated by the pixels. It will be understood that photons are the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into charge or electric current. In some embodiments, integration time 2219(a~c) may be initiated by resetting the pixels using an electron shutter or rolling shutter. The light is then integrated until the next readout stage. In some embodiments, the position of the electron shutter can be moved between two readout cycles 2202 to control pixel saturation for a given amount of light. In some alternative embodiments without an electron shutter, the integration time 2219a~c of the incident light may be initiated during the first readout cycle 2202 and may be terminated in the next readout cycle 2202, which also defines the start of the next integration. In some alternative embodiments, the amount of light accumulated by each pixel may be controlled by the time 2230a-2230d during which the light was pulsed during the blanking time 2216a-2216d. This ensures that all rows see the same light emitted from the same light pulse 2230a-2230c. In other words, each row begins its integration in a first dark environment 2231, which may be in the row 2220 behind the optical black of the readout frame (m) with the maximum light pulse width, and then ends its integration in a second dark environment 2232, which may be in the row 2218 ahead of the optical black of the next subsequent readout frame (m+1) with the maximum light pulse width. Thus, the image generated from the light pulses 2230a-c is available only during the readout of frame (m+1) without interference with either frame (m) or (m+2).

[0152] It should be noted that having optical pulses 2230a to c that are read only within one frame and do not interfere with adjacent frames is a state in which the given optical pulses 2230a to c are emitted during the blanking time 2216. Since the optical black rows 2218 and 2220 are insensitive to light, the frame time (m) of the optical black rear row 2220 and the frame time (m+1) of the optical black front row 2218 can be added to the blanking time 2216 to determine the maximum range of the emission time of the optical pulses 2230.

[0153] In some embodiments, Figure 17B shows an example of a timing diagram for continuous frame acquisition by a conventional CMOS sensor. Such a CMOS sensor may incorporate a Bayer pattern of color filters, as shown in Figure 17C. The Bayer pattern is recognized as providing luminance detail higher than chromaticity. It may be further recognized that the sensor has reduced spatial resolution because a total of four adjacent pixels are required to generate color information for the aggregated spatial portion of the image. In an alternative approach, the color image may be constructed by rapidly stroving the visualization region at high speed with various light sources (either lasers or light-emitting diodes) having different central light wavelengths.

[0154] The optical stroving system may be under the control of the camera system and may include a specially designed CMOS sensor with high-speed readout. The main benefit is that the sensor can achieve the same spatial resolution with significantly fewer pixels compared to conventional Bayer or three-sensor cameras. Thus, the physical space occupied by the pixel array can be reduced. The actual pulse periods (2230a to 2230c) may differ within the repeating pattern, as shown in Figure 17B. This is useful, for example, to allocate more time to components that require greater light energy or components with weaker light sources. As long as the average capture frame rate is an integer multiple of the required final system frame rate, the data may simply be buffered as appropriate within the signal processing chain.

[0155] The equipment for reducing the CMOS sensor chip area to an acceptable level by combining all of these methods is particularly attractive for small-diameter (approximately 3-10 mm) endoscopes. Specifically, it enables endoscope designs in which the sensor is located within a spatially constrained distal end, thereby significantly reducing the complexity and cost of the optical component while providing high-definition images. The result of this approach is that data must be temporally fused from three separate snapshots to reconstruct each final full-color image. Any movement in the scene relative to the optical frame of reference of the endoscope will generally degrade the perceived resolution because the edges of the object appear to be in slightly different positions within each captured component. This disclosure describes means for mitigating this problem by taking advantage of the fact that spatial resolution is much more important to luminance information than to chromaticity information.

[0156] The basis of this approach is that, instead of emitting monochromatic light in each frame, a combination of three wavelengths is used to provide all the luminance information within a single image. Chromaticity information is derived from separate frames with repeating patterns, such as Y-Cb-Y-Cr (Figure 17D). While a clever selection of pulse ratios can provide pure luminance data, the same cannot be said for chromaticity.

[0157] In one embodiment, as shown in Figure 17D, the endoscope system 2300a may include a pixel array 2302a having uniform pixels, and the system 2300a may be operated to receive Y (luminance pulse) 2304a, Cb (ChromaBlue) 2306a, and Cr (ChromaRed) 2308a pulses.

[0158] To complete a full-color image, two chromaticity components must also be provided. However, the same algorithm applied to luminance cannot be directly applied to chromaticity images because they are encoded, as reflected in the fact that some of the RGB coefficients are negative. The solution is to add a sufficiently large amount of luminance so that all of the final pulse energies are positive. As long as the color fusion process in the ISP recognizes the composition of the chromaticity frames, they can be decoded by subtracting an appropriate amount of luminance from the adjacent frames. The pulse energy ratios are as follows:

[0159]

number

[0160] When the λ coefficient is equal to 0.552, both the red and green components cancel each other out exactly, in which case the Cb information can be provided with pure blue light. Similarly, setting δ = 0.650 cancels out the blue and green components of Cr, resulting in pure red light. This particular example is shown in Figure 17E, which also...

[0161]

number

[0162] In the Y-Cb-Y-Cr pulsed scheme, the image data is already in the YCbCr space after color fusion. Therefore, this means that the previous luminance and chromaticity-based operations are performed before converting to linear RGB for color correction, etc.

[0163] The color fusion process is simpler than demosaicing required by Bayer patterns (see Figure 17C) because there is no spatial interpolation. Frame buffering is required to have all the necessary information available for each pixel. In one common embodiment, Y-Cb-Y-Cr pattern data may be pipelined to generate one full-color image for every two raw capture images. This is achieved by using each chromaticity sample twice. Figure 17F shows a specific example of a 120Hz frame capture rate providing a final image at 60Hz.

[0164] Further disclosures relating to the control of the laser components of the illumination system shown in Figures 17B–17F for use in the surgical visualization system 108 can be found in U.S. Patent Application Publication No. 2014 / 0160318, entitled “YCBCR PULSED ILLUMINATION SCHEME IN A LIGHT DEFICIENT ENVIRONMENT,” issued on 6 December 2016 as U.S. Patent No. 9,516,239 and filed on 26 July 2013, entitled “CONTINUOUS VIDEO IN A LIGHT DEFICIENT ENVIRONMENT,” issued on 22 August 2017 as U.S. Patent No. 9,743,016 and filed on 26 July 2013, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0165] Imaging of subsurface blood vessels During surgical procedures, surgeons may need to manipulate tissues to achieve the desired medical outcome. The surgeon's actions are limited by what is visually observable at the surgical site. Therefore, a surgeon may not be aware, for example, of the arrangement of vascular structures beneath the tissue being manipulated during the procedure.

[0166] Because surgeons cannot visualize the vascular structure beneath the surgical site, they may accidentally sever one or more important blood vessels during the procedure.

[0167] Therefore, it is desirable to have a surgical visualization system that can acquire imaging data of the surgical site for presentation to the surgeon, and whose presentation can include information related to the presence of vascular structures located beneath the surface of the surgical site.

[0168] Some aspects of the present disclosure further provide control circuits configured to control the illumination of a surgical site using one or more illumination sources, such as laser light sources, and to receive imaging data from one or more image sensors. In some aspects, the present disclosure provides a non-temporary computer-readable medium for storing computer-readable instructions that, when executed, cause a device to detect blood vessels in tissue and determine the depth below the surface of the tissue.

[0169] In some embodiments, the surgical imaging system may include a plurality of illumination sources, each configured to emit light having a specific central wavelength; a photosensor configured to receive a portion of the light reflected from a tissue sample when illuminated by one or more of the plurality of illumination sources; and a computing system. The computing system may receive data from the photosensor when the tissue sample is illuminated by each of the plurality of illumination sources, and may determine the depth location of structures within the tissue sample based on the data received by the photosensor when the tissue sample is illuminated by each of the plurality of illumination sources, and calculate visualization data relating to the structures and their depth locations. In some embodiments, the visualization data may have a data format that can be used by a display system, and the structures may include one or more vascular tissues.

[0170] Vascular imaging using NIR spectroscopy In one embodiment, the surgical imaging system may image one or more tissues within the surgical site at different times and depths, including an independent color cascade of illumination sources including visible light and light outside the visible range. The surgical imaging system may further detect or calculate the properties of light reflected and / or refracted from the surgical site. The properties of the light may be used to provide a composite image of the tissues within the surgical site and to provide analysis of underlying tissues that are not directly visible on the surface of the surgical site. The surgical imaging system may determine the tissue depth position without requiring a separate measuring device.

[0171] In one embodiment, the characteristics of light reflected and / or refracted from the surgical site may be the amount of absorbance of light at one or more wavelengths. The various chemical components of individual tissues may result in specific light absorption patterns that are wavelength-dependent.

[0172] In one embodiment, the illumination source may include a red laser source and a near-infrared laser source, and one or more tissues to be imaged may include vascular tissue such as veins or arteries. In some embodiments, a red laser source (within the visible range) may be used to image several embodiments of the underlying vascular tissue based on spectroscopy in the visible-red range. In some non-limiting examples, the red laser source may provide illumination having a peak wavelength that may be in the range of 635 nm to 660 nm, including both extreme values. Non-limiting examples of the red laser peak wavelength may include about 635 nm, about 640 nm, about 645 nm, about 650 nm, about 655 nm, about 660 nm, or any value or range of values ​​in between. In some other embodiments, a near-infrared laser source may be used to image the underlying vascular tissue based on near-infrared spectroscopy. In some non-limiting examples, the near-infrared laser source may have a wavelength that may be in the range of 750 nm to 3000 nm, including both extreme values. Non-limiting examples of infrared laser peak wavelengths may include approximately 750 nm, 1000 nm, 1250 nm, 1500 nm, 1750 nm, 2000 nm, 2250 nm, 2500 nm, 2750 nm, 3000 nm, or any value or range of values ​​in between. It may be recognized that the underlying vascular tissue can be probed using a combination of red and infrared spectroscopy. In some examples, vascular tissue may be probed using a red laser source with a peak wavelength of approximately 660 nm and a near-IR laser source with a peak wavelength of approximately 750 nm or approximately 850 nm.

[0173] Near-infrared spectroscopy (NIRS) is a non-invasive technique that enables the determination of tissue oxygenation based on the spectral quantification of oxygenated and deoxygenated hemoglobin within tissues. In some embodiments, NIRS can be used to directly image vascular tissue based on the difference in illumination absorbance between vascular and non-vascular tissues. Alternatively, vascular tissue can be indirectly visualized based on the difference in illumination absorbance of blood flow in the tissue before and after the application of physiological interventions such as arterial and venous occlusion.

[0174] Instruments for near-iron (NIR) spectroscopy may be similar to those for the ultraviolet-visible and intermediate-iron ranges. Such spectroscopic instruments may include an illumination source, a detector, and a dispersion element for selecting a specific near-iron wavelength for illuminating a tissue sample. In some embodiments, the light source may include an incandescent or quartz halogen light source. In some embodiments, the detector may include a semiconductor (e.g., InGaAs) photodiode or photoarray. In some embodiments, the dispersion element may include a prism or, more generally, a diffraction grating. Fourier transform NIR instruments using an interferometer are also common, and are particularly common for wavelengths beyond about 1000 nm. Depending on the sample, the spectrum can be measured in either reflection mode or transmission mode.

[0175] Figure 18 schematically shows an example of an apparatus 2400 similar to those used in NIR spectroscopy for the ultraviolet-visible and intermediate IR ranges. The light source 2402 can emit illumination 2404 over a broad spectral range that can strike a dispersing element 2406 (such as a prism or diffraction grating). The dispersing element 2406 may operate to select a narrow wavelength portion 2408 of the light emitted by the broad-spectrum light source 2402, and the selected portion 2408 of the light may illuminate the tissue 2410. The light reflected from the tissue 2412 may be directed to a detector 2416 (e.g., by a dichroic mirror 2414), and the intensity of the reflected light 2412 may be recorded. The wavelength of the light illuminating the tissue 2410 may be selected by the dispersing element 2406. In some embodiments, the tissue 2410 may be illuminated only by a single narrow wavelength portion 2408 selected by the dispersing element 2406 to form the light source 2402. In other embodiments, the tissue 2410 may be scanned in various narrow wavelength segments 2408 selected by the dispersive element 2406. In this way, the spectroscopic analysis of the tissue 2410 may be obtained over a range of NIR wavelengths.

[0176] Figure 19 schematically shows an example of an instrument 2430 for determining NIRS based on Fourier transform infrared imaging. In Figure 19, a laser source emitting 2432 light in the near-IR range 2434 illuminates a tissue sample 2440. The light 2436 reflected by the tissue 2440 is reflected to a beam splitter 2446 by a mirror such as a dichroic mirror 2444. The beam splitter 2446 directs one portion 2448 of the light reflected by the tissue 2440 to a stationary mirror 2450 and one portion 2452 of the light 2436 reflected by the tissue 2440 to a moving mirror 2454. The moving mirror 2454 may vibrate in a fixed position based on a fixed piezoelectric transducer actuated by a sinusoidal voltage having a voltage frequency. The position of the moving mirror 2454 in space corresponds to the frequency of the sinusoidal operating voltage of the piezoelectric transducer. Light reflected from the moving mirror and the stationary mirror may be recombined at the beam splitter 2446 and directed to the detector 2456. The computational components can receive the signal output of the detector 2456 and perform the Fourier transform (time) of the received signal. Since the wavelength of the light received from the moving mirror 2454 changes over time relative to the wavelength of the light received from the stationary mirror 2450, the time-based Fourier transform of the recombined light corresponds to the wavelength-based Fourier transform of the recombined light 2458. In this way, the spectrum based on the wavelength of the light reflected from the tissue 2440 can be determined, and the spectral characteristics of the light 2436 reflected from the tissue 2440 can be obtained. Therefore, changes in the absorbance of the illumination in the spectral components from the light reflected from the tissue 2440 may indicate the presence or absence of tissue having specific light-absorbing properties (such as hemoglobin).

[0177] An alternative to near-infrared light for determining hemoglobin oxygenation is the use of monochromatic red light to determine the red light absorbance characteristics of hemoglobin. The red light absorbance characteristics of hemoglobin with a central wavelength of approximately 660 nm can indicate whether the hemoglobin is oxygenated (arterial blood) or deoxygenated (venous blood).

[0178] In some alternative surgical procedures, contrast agents can be used to improve the data collected for oxygenation and tissue oxygen consumption. In one non-limiting example, NIRS technology may be used in combination with a bolus injection of a near-IR contrast agent such as indocyanine green (ICG), which has a peak absorbance at approximately 800 nm. ICG is used in several medical procedures to measure cerebral blood flow.

[0179] Vascular imaging using laser Doppler flow measurement In one embodiment, the characteristics of the light reflected and / or refracted from the surgical site may be the Doppler shift of the light wavelength from the illumination source.

[0180] Laser Doppler flow metering can be used to visualize and characterize the flow of particles moving against an effectively stationary background. Therefore, laser light scattered by moving particles such as blood cells may have a different wavelength than the original illumination laser source. In contrast, laser light scattered by an effectively stationary background (e.g., vascular tissue) may have the same wavelength as the original illumination laser source. The change in wavelength of scattered light from blood cells may reflect both the laser source and the direction of blood cell flow relative to blood cell velocity.

[0181] Figures 20A to 20C show the changes in the wavelength of light scattered from blood cells as they move away from the laser light source (Figure 20A) or toward the laser light source (Figure 20C).

[0182] In each of Figures 20A and 20C, the original illumination light 2502 with a relative central wavelength of 0 is shown. From Figure 20A, it can be observed that the light scattered from blood cells moving away from the laser source 2504 has a wavelength that is shifted by a certain amount 2506 greater than the wavelength of the laser source (and therefore red-shifted). Also, from Figure 20C, it can be observed that the light scattered from blood cells moving toward the laser source 2508 has a wavelength that is shifted by a certain amount 2510 shorter than the wavelength of the laser source (and therefore blue-shifted). The amount of wavelength shift (e.g., 2506 or 2510) may depend on the velocity of the blood cells. In some embodiments, the amount of red shift (2506) in some blood cells may be approximately the same as the amount of blue shift (2510) in some other blood cells. Alternatively, the amount of red shift (2506) in some blood cells may differ from the amount of blue shift (2510) in some other blood cells. Therefore, as shown in Figure 20A, the velocity of blood cells flowing away from the laser source may be less than the velocity of blood cells flowing toward the laser source, as shown in Figure 26C, based on the relative magnitude of the wavelength shifts (2506 and 2510). In contrast, as shown in Figure 26B, light scattered from tissue that is not moving relative to the laser light source (e.g., blood vessels 2512 or non-vascular tissue 2514) may not show any change in wavelength.

[0183] Figure 21 shows one embodiment of an instrument 2530 that may be used to detect the Doppler shift of laser light scattered from a portion of tissue 2540. Light 2534 emitted from laser 2532 may pass through beam splitter 2544. A portion of the laser light 2536 may be transmitted by beam splitter 2544 and illuminate the tissue 2540. Another portion of the laser light may be reflected at 2546 by beam splitter 2544 to collide with detector 2550. The backscattered light 2542 from tissue 2540 may be directed by beam splitter 2544 and may also collide with detector 2550. The combination of light 2534 emitted from laser 2532 and the backscattered light 2542 from tissue 2540 may result in an interference pattern detected by detector 2550. The interference pattern received by detector 2550 may include interference fringes resulting from a combination of light 2534 emitted from laser 2532 and Doppler-shifted (and therefore wavelength-shifted) backscattered light 2452 from tissue 2540.

[0184] It can be recognized that the backscattered light 2542 from tissue 2540 may also include backscattered light from the boundary layer within tissue 2540 and / or wavelength-specific light absorption by the material within tissue 2540. As a result, the interference pattern observed in detector 2550 may incorporate interference fringe features from these additional optical effects, and therefore, calculations of Doppler shifts may be found unless properly analyzed.

[0185] Figure 22 illustrates some of these additional optical effects. It is well known that light passing through a first optical medium having a first refractive index n1 may be reflected at the interface with a second optical medium having a second refractive index n2. Light transmitted through the second optical medium has a transmission angle with respect to the interface that is different from the angle of incident light, based on the difference between refractive indices n1 and n2 (Snell's Law). Figure 22 illustrates the effect of Snell's Law on light striking the surface of a multicomponent tissue 2150, as may be presented in the surgical field. The multicomponent tissue 2150 may consist of an outer tissue layer 2152 having a refractive index n1 and an implanting tissue such as a blood vessel having a blood vessel wall 2156. The blood vessel wall 2156 can be characterized by a refractive index n2. Blood can flow into the lumen of the blood vessel 2160. In some embodiments, it may be important during the surgical procedure to determine the location of the blood vessel 2160 beneath the surface 2154 of the outer tissue layer 2152 and to characterize the blood flow using Doppler shift techniques.

[0186] The incident laser beam 2170a may be used to probe the blood vessel 2160 and may be directed onto the upper surface 2154 of the outer tissue layer 2152. A portion 2172 of the incident laser beam 2170a may be reflected at the upper surface 2154. Another portion 2170b of the incident laser beam 2170a may penetrate the outer tissue layer 2152. The reflected portion 2172 at the upper surface 2154 of the outer tissue layer 2152 has the same path length as the incident light 2170a and therefore has the same wavelength and phase as the incident light 2170a. However, the portion 2170b of light transmitted through the outer tissue layer 2152 has a different transmission angle than the incident angle of light hitting the tissue surface because the outer tissue layer 2152 has a refractive index n1 that is different from that of air.

[0187] When a portion of the light transmitted through the outer tissue layer 2152 collides with, for example, the second tissue surface 2158 of the blood vessel wall 2156, some portions 2174a,b of the light are reflected toward the light source 2170a of the incident light. Thus, the light 2174a reflected at the interface between the outer tissue layer 2152 and the blood vessel wall 2156 has the same wavelength as the incident light 2170a, but is phase-shifted due to the change in optical path length. Projecting the reflected light 2174a,b along with the incident light on the sensor from the interface between the outer tissue layer 2152 and the blood vessel wall 2156 generates an interference pattern based on the phase difference between the two light sources.

[0188] Furthermore, a portion of the incident light 2170c can penetrate the vessel wall 2156 and into the vessel lumen 2160. This portion of the incident light 2170c may interact with moving blood cells in the vessel lumen 2160 and may be reflected at 2176a-2176c toward a light source of impacting light having a wavelength Doppler shifted according to the velocity of the blood cells, as disclosed above. The Doppler-shifted light 2176a-c reflected from the moving blood cells may be projected together with the incident light on the sensor, resulting in an interference pattern having stripes based on the wavelength difference between the two light sources.

[0189] In Figure 22, the optical path 2178 presents light colliding with red blood cells in the vascular lumen 2160, assuming no change in refractive index between the emitted light and the light reflected by the moving blood cells. In this example, only the Doppler shift of the reflected light wavelength can be detected. However, the light reflected by the blood cells 2176(a-c) may incorporate a phase shift due to variations in tissue refractive index, in addition to the wavelength change due to the Doppler effect.

[0190] Therefore, when an optical sensor receives incident light, if it receives light reflected from one or more tissue interfaces (2172 and 2174a, b) and Doppler-shifted light shifted from blood cells (2176a-c), it can be understood that the interference pattern generated on the optical sensor may include effects due to Doppler shift (wavelength change) and changes in refractive index (phase change) within the tissue. As a result, if the effects of changes in refractive index within the sample are not compensated for, Doppler analysis of light reflected by the tissue sample may produce erroneous results.

[0191] Figure 23 shows an example of the effect of Doppler analysis of light impacting 2250 on a tissue sample to determine the depth and location of underlying blood vessels. When no intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be primarily due to changes in wavelength reflected from migrating blood cells. As a result, the spectrum 2252 derived from the interference pattern may generally only reflect the Doppler shift of the blood cells. However, when intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be due to a combination of changes in wavelength reflected from migrating blood cells and a phase shift due to the refractive index of the intervening tissue. The spectrum 2254 derived from such an interference pattern may result in a calculation of the Doppler shift found due to the additional phase change of the reflected light. In some embodiments, if information regarding the properties (thickness and refractive index) of the intervening tissue is known, the resulting spectrum 2256 may be corrected to provide a more accurate calculation of the wavelength change.

[0192] It is recognized that the tissue penetration depth of light depends on the wavelength of light used. Therefore, the wavelength of the laser source light may be selected to detect particle motion (such as blood cells) at a specific range of tissue depths.

[0193] Figures 24A to 24C schematically illustrate means for detecting moving particles such as blood cells at various tissue depths based on laser wavelength. As shown in Figure 24A, the laser source 2340 may direct the incident beam of laser light 2342 onto the surface 2344 of the surgical site. Blood vessels 2346 (veins or arteries, etc.) may be located within the tissue 2348 at a depth delta of some extent from the tissue surface. The penetration depth 2350 of the laser into the tissue 2348 may depend at least partially on the laser wavelength. Thus, laser light with a wavelength in the red range of approximately 635 nm to approximately 660 nm can penetrate tissue 2351a to a depth of approximately 1 mm. Laser light with a wavelength in the green range of approximately 520 nm to approximately 532 nm can penetrate tissue 2351b to a depth of approximately 2 to 3 mm. Laser light with a wavelength in the blue range of approximately 405 nm to approximately 445 nm can penetrate tissue 2351C to a depth of approximately 4 mm or more. In the examples shown in Figures 30A to 30C, blood vessel 2346 may be located at a depth delta of approximately 2-3 mm below the tissue surface. Red laser light does not penetrate to this depth and therefore does not detect blood cells flowing within this vessel. However, both green and blue laser light can penetrate to this depth. Therefore, scattered green and blue laser light from blood cells within blood vessel 2346 may exhibit a Doppler shift in wavelength.

[0194] Figure 24B illustrates how the Doppler shift 2355 at the wavelength of the reflected laser light can manifest. The synchrotron radiation (or laser source light 2342) impacting the tissue surface 2344 may have a central wavelength 2352. For example, light from a green laser may have a central wavelength 2352 in the range of approximately 520 nm to approximately 532 nm. The reflected green light may have a central wavelength 2354 that is shifted to a longer wavelength (red shift) if it is reflected from particles such as red blood cells moving away from the detector. The difference between the central wavelength 2352 and the central wavelength 2354 of the emitted laser light includes the Doppler shift 2355.

[0195] As disclosed above in relation to Figures 22 and 23, the laser light reflected from the structure within the tissue 2348 may also exhibit a phase shift of the reflected light due to changes in refractive index resulting from changes in the tissue structure or composition. The synchrotron radiation (or laser source light 2342) striking the tissue surface 2344 may have a first phase characteristic 2356. The reflected laser light may have a second phase characteristic 2358. It can be recognized that blue laser light 2351c, which can penetrate the tissue to a depth of about 4 mm or more, may encounter a wider variety of tissue structures than red laser light (about 1 mm, 2351a) or green laser light (about 2-3 mm, 2351b). Therefore, as shown in Figure 30C, the phase shift 2358 of the reflected blue laser light may be significant, at least due to the depth of penetration.

[0196] Figure 24D shows an embodiment of continuous illumination of tissue with red 2360a, green 2360b, and blue 2360c laser light. In some embodiments, the tissue may be continuously probed with red 2360a, green 2360b, and blue 2360c laser illumination. In some alternative examples, as shown in Figures 17D to 17F and disclosed above, one or more combinations of red 2360a, green 2360b, and blue 2360c laser light may be used to illuminate the tissue according to a defined illumination sequence. Figure 24D shows the effect of such illumination on CMOS imaging sensors 2362a to 2362d over time. Thus, at a first time t.sub.1, CMOS sensor 2362a may be illuminated by the red 2360a laser. At a second time t.sub.2, CMOS sensor 2362b may be illuminated by the green 2360b laser. At the third time t.sub.3, the CMOS sensor 2362c may be illuminated by a blue 2360c laser. The illumination cycle may then be repeated, starting from the fourth time t.sub.4, at which point the CMOS sensor 2362d may again be illuminated by a red 2360a laser. It can be recognized that continuous illumination of tissue with laser illumination at different wavelengths may enable Doppler analysis at various tissue depths over time. The surgical site may be illuminated using red 2360a, green 2360b, and blue 2360c laser sources, but it can also be recognized that other wavelengths outside the visible light range (such as the infrared or ultraviolet region) may be used to illuminate the surgical site for Doppler analysis.

[0197] Figure 25 shows an example of the use of Doppler imaging to detect the presence of blood vessels that cannot be seen by other means at a surgical site 2600. In Figure 25, the surgeon may wish to excise a tumor 2602 located within the right upper posterior lobe 2604 of the lung. Because the lung is highly vascular, care must be taken to identify only these vessels associated with the tissue and to seal only those vessels without impairing blood flow to the unaffected portion of the lung. In Figure 25, the surgeon has identified the edge 2606 of the tumor 2604. The surgeon may then excise an initial incision area 2608 within the edge region 2606, and exposed blood vessels 2610 may be observed for excision and sealing. A Doppler imaging detector 2620 may be used to locate and identify non-observable blood vessels 2612 within the incision area. The imaging system may receive data from the Doppler imaging detector 2620 for analysis and display of data acquired from the surgical site 2600. In some embodiments, the imaging system may include a display illustrating the surgical site 2600, including a visible image of the surgical site 2600, along with an image of hidden blood vessels 2612 on the image of the surgical site 2600 overlay.

[0198] In the scenario disclosed above with respect to Figure 25, the surgeon wants to sever the blood vessels supplying oxygen and nutrients to the tumor while preserving blood vessels associated with non-cancerous tissue. Additionally, the blood vessels may be located at different depths within or around the surgical site 2600. Therefore, the surgeon must identify the location (depth) of the blood vessels and determine whether they are suitable for resection. Figure 26 shows one method for identifying deep blood vessels based on the Doppler shift of light from blood cells flowing through the vessels. As disclosed above, red laser light has a penetration depth of approximately 1 mm, and green laser light has a penetration depth of approximately 2-3 mm. However, blood vessels with a subsurface depth of 4 mm or more are outside the penetration depths at these wavelengths. However, blue laser light can detect such blood vessels based on their blood flow.

[0199] Figure 26 shows the Doppler shift of laser light reflected from a blood vessel at a specific depth below the surgical site. This site may be illuminated by red, green, and blue laser light. The central wavelength 2630 of the illumination light may be normalized to the relative center 3631. If the blood vessel is at a depth of 4 mm or more below the surface of the surgical site, neither the red nor the green laser light will be reflected by the blood vessel. As a result, the central wavelength 2632 of the reflected red light and the central wavelength 2634 of the reflected green light are not significantly different from the central wavelength 2630 of the illumination red or green light, respectively. However, if this site is illuminated by a blue laser light, the central wavelength 2638 of the reflected blue light 2636 is different from the central wavelength 2630 of the illumination blue light. In some cases, the amplitude of the reflected blue light 2636 may also be significantly reduced from the amplitude of the illumination blue light. Thus, the surgeon can determine the presence of deep blood vessels along with their approximate depth, thereby avoiding deep blood vessels during surface tissue incision.

[0200] Figures 27 and 28 schematically illustrate the use of laser sources with different central wavelengths (colors) to determine the approximate depth of blood vessels beneath the surface of a surgical site. Figure 27 shows a first surgical site 2650 having a surface 2654 and blood vessels 2656 located beneath the surface 2654. In one method, blood vessels 2656 may be identified based on the Doppler shift of light striking the flow of blood cells 2658 within the blood vessels 2656. The surgical site 2650 may be illuminated by light from a plurality of lasers 2670, 2676, 2682, each laser characterized by emitting light at one of several different central wavelengths. As mentioned above, illumination by the red laser 2670 can penetrate the tissue by only about 1 mm. Therefore, if the blood vessels 2656 are located at a depth of less than 1 mm 2672 beneath the surface 2654, the red laser illumination is reflected at 2674, and the Doppler shift of the reflected red illumination 2674 can be determined. Furthermore, as mentioned above, illumination by the green laser 2676 can penetrate only the tissue by about 2-3 mm. If the blood vessel 2656 is located at a depth of about 2-3 mm 2678 below the surface 2654, the green laser illumination will be reflected at 2680, while the red laser illumination 2670 will not be reflected, and the Doppler shift of the reflected green illumination 2680 can be determined. However, as shown in Figure 27, the blood vessel 2656 is located at a depth of about 4 mm 2684 below the surface 2654. Therefore, neither the red laser illumination 2670 nor the green laser illumination 2676 is reflected. Instead, only the blue laser illumination is reflected (2686), and the Doppler shift of the reflected blue illumination 2686 can be determined.

[0201] In contrast to vessel 2656 shown in Figure 27, vessel 2656' shown in Figure 28 is located closer to the surface of the tissue at the surgical site. Vessel 2656' can also be distinguished from vessel 2656 in that vessel 2656' is exemplified as having a much thicker wall 2657. Thus, vessel 2656 may be an example of a vein, while vessel 2656' may be an example of an artery, since arterial walls are known to be thicker than venous walls. In some examples, arterial walls can have a thickness of about 1.3 mm. As disclosed above, red laser illumination 2670' can penetrate the tissue to a depth of about 1 mm 2672'. Therefore, even if vessel 2656' is exposed at the surgical site (see 2610 in Figure 25), red laser light reflected from the surface of vessel 2656' (2674') may not be able to visualize the blood flow 2658' within vessel 2656' under Doppler analysis due to the thickness of the vessel wall 2657. However, as disclosed above, a green laser beam impact 2676' on the surface of tissue can penetrate to a depth of approximately 2-3 mm 2678'. Furthermore, a blue laser beam impact 2682' on the surface of tissue can penetrate to a depth of approximately 4 mm 2684'. Therefore, the green laser beam may be reflected (2680') from blood cells flowing (2658') within the blood vessels 2656', and the blue laser beam may be reflected (2686') from blood cells flowing (2658') within the blood vessels 2656'. As a result, Doppler analysis of the reflected green light 2680' and the reflected blue light 2686' may provide information about blood flow in near-surface blood vessels, particularly the approximate depth of the vessels.

[0202] As disclosed above, the depth of blood vessels beneath the surgical site may be probed based on wavelength-dependent Doppler imaging. The amount of blood flow through such vessels may be determined by speckle contrast (interference) analysis. The Doppler shift may indicate moving particles relative to a stationary light source. As disclosed above, the Doppler wavelength shift may be an indicator of the velocity of particle motion. Individual particles, such as blood cells, may not be observable separately. However, the velocity of each blood cell may generate a proportional Doppler shift. The interference pattern may be generated by a combination of backscattered light from multiple blood cells, resulting from the difference in the Doppler shifts of the backscattered light from each of the blood cells. The interference pattern may be an indicator of the number density of blood cells in the visualization frame. The interference pattern is sometimes called speckle contrast. Speckle contrast analysis may be calculated using a full-frame 300×300 CMOS imaging array, and the speckle contrast may be directly related to the amount of moving particles (e.g., blood cells) interacting with the laser light over a given exposure period.

[0203] The CMOS image sensor may be coupled to a digital signal processor (DSP). Each pixel of the sensor may be multiplexed and digitized. The Doppler shift of light may be analyzed by observing the light source laser beam compared to the Doppler-shifted light. Larger Doppler shifts and speckles may be associated with a greater number of blood cells in the blood vessels and their velocity.

[0204] Figure 29 shows one embodiment of a composite visual display 2800 that may be presented to a surgeon during a surgical procedure. The composite visual display 2800 may be constructed by overlaying a white light image 2830 of the surgical site using a Doppler analysis image 2850.

[0205] In some embodiments, the white light image 2830 can make the surgical site 2832, one or more surgical incisions 2834, and tissue 2836 easily visible within the surgical incision 2834. The white light image 2830 may be generated by illuminating the surgical site 2832 with a white light source 2838 and receiving the reflected white light 2842 with a photodetector. The surface of the surgical site may be illuminated using the white light source 2838, but in one embodiment, the surface of the surgical site may be visualized using an appropriate combination of red 2854, green 2856, and blue 2858 laser light, as disclosed above with respect to Figures 17C to 17F.

[0206] In some embodiments, the Doppler analysis image 2850 may include vessel depth information (from speckle analysis) along with blood flow information 2852. As disclosed above, vessel depth and blood flow velocity may be obtained by illuminating the surgical site with laser light of multiple wavelengths and determining vessel depth and blood flow based on the known penetration depth of light of a particular wavelength. Generally, the surgical site 2832 may be illuminated by light emitted by one or more lasers such as a red laser 2854, a green laser 2856, and a blue laser 2858. The CMOS detector 2872 may receive light (2862, 2866, 2870) reflected from the surgical site 2832 and the surrounding tissue. The Doppler analysis image 2850 may be constructed 2874 based on analysis of multiple pixel data from the CMOS detector 2872.

[0207] In one embodiment, a red laser 2854 may emit red laser illumination 2860 onto the surgical site 2832, and the reflected light 2862 can expose the surface or the smallest subsurface structure. In another embodiment, a green laser 2856 may emit green laser illumination 2864 onto the surgical site 2832, and the reflected light 2866 can reveal deeper subsurface characteristics. In yet another embodiment, a blue laser 2858 may emit blue laser illumination 2868 onto the surgical site 2832, and the reflected light 2870 can expose, for example, blood flow in deeper vascular structures. In addition, speckle contrast analysis may provide the surgeon with information on the volume and velocity of blood flow through deeper vascular structures.

[0208] Although not shown in Figure 29, it can be understood that the imaging system may also illuminate the surgical site with light outside the visible range. Such light may include infrared and ultraviolet light. In some embodiments, the infrared or ultraviolet light source may include a broadband source (such as a tungsten source, a tungsten halogen source, or a deuterium source). In some other embodiments, the infrared or ultraviolet light source may include a narrowband source (an IR diode laser, a UV gas laser, or a dye laser).

[0209] Figure 30 is a flowchart 2900 of a method for determining the depth of surface features within a tissue sample. The image acquisition system may illuminate the tissue with a first ray having a first center frequency 2910 and receive a first reflected light from the tissue illuminated by the first ray 2912. The image acquisition system may then calculate a first Doppler shift 2914 based on the first ray and the first reflected light. The image acquisition system may then illuminate the tissue with a second ray having a second center frequency 2916 and receive a second reflected light from the tissue illuminated by the second ray 2918. The image acquisition system may then calculate a second Doppler shift 2920 based on the second ray and the second reflected light. The image acquisition system may then calculate the depth of the tissue features 2922 based at least partially on the first center wavelength, the first Doppler shift, the second center wavelength, and the second Doppler shift. In some embodiments, tissue features may include the presence of migrating particles, such as blood cells moving within blood vessels, as well as the direction and velocity of the flow of these migrating particles. It can be understood that this method may be extended to include illumination of the tissue with any one or more additional rays. Furthermore, the system may compute an image that includes a combination of an image of the tissue surface and an image of structures located within the tissue.

[0210] In some embodiments, multiple visual displays may be used. For example, a 3D display may provide a composite image showing a combined white light (or a suitable combination of red, green, and blue laser light) and a laser Doppler image. Additional displays may provide only a white light display or only a combined white light display and a NIRS display to visualize only the tissue's blood oxygenation response. However, a NIRS display may not be required for every cycle that allows the tissue response to be enabled.

[0211] Characterization of surface structure using multispectral OCT During surgical procedures, surgeons may use “smart” surgical devices for manipulating tissue. Such devices may be considered “smart” in that they include automated functions to direct, control, and / or modify the operation of device-based parameters related to their use. Parameters may include the type and / or composition of the tissue being manipulated. If the type and / or composition of the tissue being manipulated is unknown, the operation of the smart device may be inappropriate for the tissue being manipulated. As a result, the tissue may be damaged, or the manipulation of the tissue may be invalid due to improper settings of the smart device.

[0212] Surgeons may manually attempt to change the parameters of smart devices through trial and error, resulting in inefficient and lengthy surgical procedures.

[0213] Therefore, it is desirable to have a surgical visualization system that can probe the tissue structures beneath the surgical site to determine their structural and compositional properties and provide such data to smart surgical instruments used in surgical procedures.

[0214] Some aspects of the present disclosure further provide control circuits configured to control illumination of a surgical site using one or more illumination sources, such as laser light sources, and to receive imaging data from one or more image sensors. In some aspects, the present disclosure provides a non-temporary computer-readable medium for storing computer-readable instructions, which, when executed, cause a device to characterize subsurface structures at a surgical site and determine the depth of subsurface structures in tissue.

[0215] In some embodiments, the surgical imaging system may include a plurality of illumination sources, each configured to emit light having a specific central wavelength; an optical sensor configured to receive a portion of the light reflected from a tissue sample when it is illuminated by one or more of the plurality of illumination sources; and a computing system. The computing system may be configured to receive data from the optical sensor when the tissue sample is illuminated by each of the plurality of illumination sources, to calculate structural data relating to the properties of structures within the tissue sample based on the data received by the optical sensor when the tissue sample is illuminated by each of the illumination sources, and to transmit structural data relating to structural features to be received by the smart surgical device. In some embodiments, the structural features are surface features or structural composition.

[0216] In one embodiment, the surgical system may include multiple laser light sources capable of receiving laser light reflected from tissue. The light reflected from the tissue can be used by the system to calculate the surface characteristics of components placed within the tissue. The characteristics of the components placed within the tissue may include metrics related to the composition of the components and / or the surface roughness of the components.

[0217] In one embodiment, the surgical system may transmit data relating to the composition of the components and / or metrics relating to the surface irregularities of the components to a second instrument used on tissue to modify the control parameters of the second instrument.

[0218] In some embodiments, the second device may be a forward energy device, and modifications to the control parameters may include clamp pressure, operating power level, operating frequency, and transducer signal amplitude.

[0219] As disclosed above, blood vessels can be detected beneath the surface of the surgical site based on the Doppler shift of light reflected by blood cells moving within the vessels.

[0220] Laser Doppler flow measurement can be used to visualize and characterize the flow of particles moving against an effectively stationary background. Therefore, laser light scattered by moving particles such as blood cells may have a different wavelength than the original illumination laser source. In contrast, laser light scattered by an effectively stationary background (e.g., vascular tissue) may have the same wavelength as the original illumination laser source. The change in wavelength of scattered light from blood cells can reflect both the laser source and the direction of blood cell flow relative to blood cell velocity. As previously disclosed, Figures 20A–20C show the change in wavelength of scattered light from blood cells, which may be moving away from the laser light source (Figure 20A) or toward the laser light source (Figure 20C).

[0221] In each of Figures 20A to 20C, the original illumination light 2502 with a relative central wavelength of 0 is shown. From Figure 20A, it can be observed that the light scattered from blood cells moving away from the laser source 2504 has a wavelength that is shifted by a certain amount 2506 greater than the wavelength of the laser source (and therefore red-shifted). Also, from Figure 20C, it can be observed that the light scattered from blood cells moving toward the laser source 2508 has a wavelength that is shifted by a certain amount 2510 shorter than the wavelength of the laser source (and therefore blue-shifted). The amount of wavelength shift (e.g., 2506 or 2510) may depend on the velocity of the blood cells. In some embodiments, the amount of red shift (2506) in some blood cells may be approximately the same as the amount of blue shift (2510) in some other blood cells. Alternatively, the amount of red shift (2506) in some blood cells may differ from the amount of blue shift (2510) in some other blood cells. As shown in Figure 24A, the velocity of blood cells flowing away from the laser source may be less than the velocity of blood cells flowing toward the laser source, as shown in Figure 20C, based on the relative magnitude of the wavelength shifts (2506 and 2510). In contrast, as shown in Figure 20B, light scattered from tissue not moving relative to the laser source (e.g., blood vessels 2512 or non-vascular tissue 2514) may not show any change in wavelength.

[0222] As previously disclosed, Figure 21 shows one embodiment of an instrument 2530 that may be used to detect the Doppler shift of laser light scattered from a portion of tissue 2540. Light 2534 emitted from laser 2532 may pass through beam splitter 2544. A portion of the laser light 2536 may be transmitted by beam splitter 2544 to illuminate tissue 2540. Another portion of the laser light may be reflected by beam splitter 2544 to collide with detector 2550 (2546). Backscattered light 2542 from tissue 2540 may be directed by beam splitter 2544 and can also collide with detector 2550. The combination of light 2534 emitted from laser 2532 and backscattered light 2542 from tissue 2540 may result in an interference pattern detected by detector 2550. The interference pattern received by detector 2550 may include interference fringes resulting from a combination of light 2534 emitted from laser 2532 and Doppler-shifted (and therefore wavelength-shifted) backscattered light 2452 from tissue 2540.

[0223] It can be recognized that the backscattered light 2542 from tissue 2540 may also include backscattered light from the boundary layer within tissue 2540 and / or wavelength-specific light absorption by the material within tissue 2540. As a result, the interference pattern observed in detector 2550 may incorporate interference fringe features from these additional optical effects, and therefore, calculations of Doppler shifts may be found unless properly analyzed.

[0224] It can be recognized that the light reflected from the tissue may also include backscattered light from the boundary layer within the tissue and / or wavelength-specific light absorption by the material within the tissue. As a result, the interference pattern observed in the detector may incorporate fringe features that, unless properly analyzed, can lead to calculations of Doppler shifts.

[0225] As previously disclosed, Figure 22 illustrates some of these additional optical effects. It is well known that light passing through a first optical medium having a first refractive index n1 may be reflected at the interface with a second optical medium having a second refractive index n2. Light transmitted through the second optical medium has a transmission angle with respect to the interface that is different from the angle of incident light, based on the difference between refractive indices n1 and n2 (Snell's Law). Figure 20 illustrates the effect of Snell's Law on light striking the surface of a multicomponent tissue 2150, as may be presented in the surgical field. The multicomponent tissue 2150 may consist of an outer tissue layer 2152 having a refractive index n1 and an implanting tissue such as a blood vessel having a blood vessel wall 2156. The blood vessel wall 2156 can be characterized by a refractive index n2. Blood can flow into the lumen of the blood vessel 2160. In some embodiments, it may be important during the surgical procedure to determine the location of the blood vessel 2160 beneath the surface 2154 of the outer tissue layer 2152 and to characterize the blood flow using Doppler shift techniques.

[0226] The incident laser beam 2170a may be used to probe the blood vessel 2160 and may be directed onto the upper surface 2154 of the outer tissue layer 2152. A portion 2172 of the incident laser beam 2170a may be reflected at the upper surface 2154. Another portion 2170b of the incident laser beam 2170a may penetrate the outer tissue layer 2152. The reflected portion 2172 at the upper surface 2154 of the outer tissue layer 2152 has the same path length as the incident light 2170a and therefore has the same wavelength and phase as the incident light 2170a. However, the portion 2170b of light transmitted through the outer tissue layer 2152 has a different transmission angle than the incident angle of light hitting the tissue surface because the outer tissue layer 2152 has a refractive index n1 that is different from that of air.

[0227] When a portion of the light transmitted through the outer tissue layer 2152 collides with, for example, the second tissue surface 2158 of the blood vessel wall 2156, some portions 2174a,b of the light are reflected toward the light source 2170a of the incident light. Thus, the light 2174a reflected at the interface between the outer tissue layer 2152 and the blood vessel wall 2156 has the same wavelength as the incident light 2170a, but is phase-shifted due to the change in optical path length. Projecting the reflected light 2174a,b along with the incident light on the sensor from the interface between the outer tissue layer 2152 and the blood vessel wall 2156 generates an interference pattern based on the phase difference between the two light sources.

[0228] Furthermore, a portion of the incident light 2170c can penetrate the vessel wall 2156 and into the vessel lumen 2160. This portion of the incident light 2170c may interact with moving blood cells in the vessel lumen 2160 and may be reflected at 2176a-2176c toward a light source of impacting light having a wavelength Doppler shifted according to the velocity of the blood cells, as disclosed above. The Doppler-shifted light 2176a-c reflected from the moving blood cells may be projected together with the incident light on the sensor, resulting in an interference pattern having stripes based on the wavelength difference between the two light sources.

[0229] In Figure 22, the optical path 2178 presents light colliding with red blood cells in the vascular lumen 2160, assuming no change in refractive index between the emitted light and the light reflected by the moving blood cells. In this example, only the Doppler shift of the reflected light wavelength can be detected. However, the light reflected by the blood cells 2176(a-c) may incorporate a phase shift due to variations in tissue refractive index, in addition to the wavelength change due to the Doppler effect.

[0230] Therefore, when an optical sensor receives incident light, if it receives light reflected from one or more tissue interfaces (2172 and 2174a, b) and Doppler-shifted light shifted from blood cells (2176a-c), it can be understood that the interference pattern generated on the optical sensor may include effects due to Doppler shift (wavelength change) and changes in refractive index (phase change) within the tissue. As a result, if the effects of changes in refractive index within the sample are not compensated for, Doppler analysis of light reflected by the tissue sample may produce erroneous results.

[0231] As previously disclosed, Figure 23 shows an example of the effect on Doppler analysis of light impacting 2250 on a tissue sample to determine the depth and location of underlying blood vessels. When no intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be primarily due to changes in wavelength reflected from migrating blood cells. As a result, the spectrum 2252 derived from the interference pattern may generally only reflect the Doppler shift of the blood cells. However, when intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be a combination of changes in wavelength reflected from migrating blood cells and a phase shift due to the refractive index of the intervening tissue. The spectrum 2254 derived from such an interference pattern may result in a calculation of the Doppler shift found due to the additional phase change of the reflected light. In some embodiments, if information regarding the properties (thickness and refractive index) of the intervening tissue is known, the resulting spectrum 2256 may be corrected to provide a more accurate calculation of the wavelength change.

[0232] It can be recognized that the phase shift of reflected light from tissue, regardless of the Doppler effect, may provide additional information about the underlying tissue structure.

[0233] Surgical visualization systems using the imaging techniques disclosed herein may benefit from ultra-high sampling and display frequencies. The sampling rate may be associated with the capability of the underlying device performing the sampling. A general-purpose computing system with software may be associated with a first range of achievable sampling rates. A purely hardware implementation (e.g., a dedicated application-specific integrated circuit, ASIC) may be associated with a second range of achievable sampling rates. The second range associated with a purely hardware implementation is generally higher (e.g., much higher) than the first range associated with a general-purpose computing software implementation.

[0234] Surgical visualization systems using the imaging techniques disclosed herein may benefit from adaptable and / or updatable imaging algorithms (e.g., transformation and imaging processes). General-purpose computing systems with software may be associated with a high degree of adaptability and / or updatableness. Pure hardware implementations (e.g., dedicated application-specific integrated circuits, ASICs) may generally be associated with a lower degree of adaptability and / or updatableness than general-purpose computing systems with software. This may be partly due to the general ease with which software can be adapted and / or updated (which may include compiling and loading different software, and / or updating modular components) compared to pure hardware implementations (where new hardware components are physically designed, built, added, and / or swapped).

[0235] Surgical visualization systems using the imaging techniques disclosed herein may benefit from solutions that balance higher sampling rates with the adaptability and / or updatableness of software systems, associated with hardware-based implementations. Such surgical visualization systems may employ a mixture of hardware and software solutions. For example, a surgical visualization system may use various hardware implementation conversions with software selectors. A surgical visualization system may also use a field-programmable gate array (FPGA). An FPGA may include a hardware device that may contain one or more logic elements. These logic elements may be composed of bitstreams to implement various functions. For example, logic elements may be configured to perform specific individual logic functions, or they may be configured to perform them in a specific order and interconnection. Once configured, the FPGA may perform its functions using the hardware logic elements without further configuration. Alternatively, once configured, the FPGA may be reconfigured with different bitstreams to implement different functions. Similarly, once reconfigured, the FPGA may perform these different functions using the hardware logic elements.

[0236] Figure 31 shows an exemplary surgical visualization system 10000. The surgical visualization system 10000 may be used to analyze at least a portion of the surgical field. For example, the surgical visualization system 10000 may be used to analyze tissue 10002 in at least a portion of the surgical field. The surgical visualization system 10000 may include a field-programmable gate array (FPGA) 10004, a processor (e.g., a processor 10006 local to the FPGA 10004), memory 10008, a laser light illumination source 10010, an optical sensor 10012, a display 10014, and / or a processor 10016 located remotely from the FPGA. The surgical visualization system 10000 may include components and functions described, for example, in relation to Figures 16A to 16D.

[0237] System 10000 may use FPGA 10004 to transform reflected laser light through frequency conversion and determine moving particles, for example, by identifying the Doppler shift of the light. This transformed data may be displayed (for example, in real time). This transformed data may be displayed, for example, as a graph and / or metric 10020 representing the number of moving particles per second. System 10000 may include communication between a local processor 10006 to FPGA 10004 and a remote processor 10016 of FPGA 10004. For example, the remote processor 10016 of FPGA 10004 may aggregate the data (for example, data over multiple seconds). The system may also be able to display this aggregate of data, for example, as a graph and / or metric 10026 representing the movement trend. This graph and / or metric 10026 may be overlaid on real-time data. Such trend information may be used to identify occlusion, the vascular sealing / clamping efficiency of the device, an overview of the vascular tree, and even the magnitude of the amplitude of movement over time. FPGA10004 may be configured to be updatable on the fly, for example, with different (e.g., more advanced) transformations. These updates may come from a local or remote communication server. These updates may change the analysis of the transformations, for example, from refractive index (e.g., analysis of cellular irregularity) to blood flow, multiple simultaneous depth analysis, etc.

[0238] FPGA updates may include transformations that implement various imaging options for the user. These imaging options may include standard composite visible light, tissue refractive power, Doppler shift, motion artifact correction, improved dynamic range, improved local clarity, super-resolution, NIR fluorescence, multispectral imaging, confocal laser endoscopy, optical coherence tomography, Raman spectroscopy, photoacoustic imaging, or any combination thereof. The imaging options include U.S. Patent Application No. 15 / 940,742, titled "DUAL CMOS ARRAY IMAGING," filed March 29, 2018; U.S. Patent Application No. 13 / 952,564, titled "WIDE DYNAMIC RANGE USING MONOCHROMATIC SENSOR," filed July 26, 2013; U.S. Patent Application No. 14 / 214,311, titled "SUPER RESOLUTION AND COLOR MOTION ARTIFACT CORRECTION IN A PULSED COLOR IMAGING SYSTEM," filed March 14, 2014; and "CAMERA SYSTEM WITH MINIMAL AREA MONOLITHIC CMOS IMAGE," filed July 26, 2013. This may include any of the options presented in either U.S. Patent Application No. 13 / 952,550, entitled “SENSOR”, each of which is incorporated herein by reference in whole. Doppler wavelength shift may be used, for example, to identify the number, size, speed, and / or direction of moving particles. Doppler wavelength shift may be used in conjunction with multiple laser wavelengths, for example, to correlate tissue depth and moving particles. Tissue refractive power may be used, for example, to identify irregularities or variability in the surface and subsurface aspects of tissue. In surgical practice, tissue refractive power may be useful in identifying tumor margins, infections, destroyed surface tissue, adhesions, changes in tissue composition, etc. NIR fluorescence may include techniques in which a systemically injected drug is preferentially absorbed by target tissue. When illuminated with the appropriate wavelength of light, they fluoresce and can be imaged by an NIR-compatible scope / camera.Hyperspectral and / or multispectral imaging may involve illuminating and evaluating tissue across many wavelengths of the entire electromagnetic spectrum to provide real-time images. This may be used to distinguish between target tissues. This may also allow imaging depths of, for example, 0–10 mm. Confocal laser endomicroscopy (CLE) can use light to capture high-resolution cellular-level resolution without penetrating the tissue. It can provide real-time histopathological diagnosis of tissue. It is a technique that uses light to capture 3D images with micrometer resolution from within the tissue. Optical coherence tomography (OCT) may use NIR light. OCT may allow imaging of tissue at depths of, for example, 1–2 mm. Raman spectroscopy may involve a technique that measures the photon shift caused by monochromatic laser illumination of tissue. This may be used to identify specific molecules. Photoacoustic imaging may involve exposing tissue to laser pulses such that a portion of the energy causes thermoelastic expansion and ultrasonic emission. These resulting ultrasounds may be detected and analyzed to form an image.

[0239] These updates can be performed automatically based on user input or system compliance checks. These real-time aggregation and updatable functions of System 10000 may be selectively enabled based on any aspect of the system configuration, such as system capacity, power availability, free memory access, transfer capacity, software level, tiered purchase level, etc.

[0240] The laser light illumination source 10010 may include any illumination source of laser light suitable for analyzing human tissue. The laser light illumination source 10010 may include devices such as the light source laser emitter shown in Figures 17A to 17F. The laser light illumination source 10010 may use one or more wavelengths of laser light to illuminate the tissue 10002. For example, the laser light illumination source 10010 may use a combination of red-blue-green-ultraviolet 1-ultraviolet 2-infrared. For example, this combination with a sampling and operating speed of 360-480 Hz allows each light source to have multiple frames at a combined frame rate of 60 Hz for the end user. Combinations of laser light wavelengths with independent light sources may increase the resolution from a single array and allow for various depth penetrations.

[0241] The tissue 10002 may be, for example, human tissue within a portion of the surgical field. The laser light may be reflected from the tissue 10002, resulting in reflected laser light. The reflected laser light may be received by the optical sensor 10012. The optical sensor 10012 may be configured to receive reflected laser light from at least a portion of the surgical field. The optical sensor 10012 may be configured to receive laser light from the entire surgical field. The optical sensor may be configured to receive reflected laser light from a selectable portion of the surgical field. For example, a user such as a surgeon may direct the optical sensor and the optical laser light source and / or laser light source to analyze a specific portion of the surgical field.

[0242] The optical sensor 10012 may be any device suitable for sensing reflected laser light and outputting corresponding information. For example, the optical sensor 10012 may detect one or more characteristics of the reflected laser light, such as amplitude, frequency, wavelength, Doppler shift, and / or other time-domain or frequency-domain qualities. The laser light sensor 10012 source may include, for example, a device such as the optical sensor disclosed in relation to Figures 16A to 16D.

[0243] The laser light sensor 10012 may include one or more sensor modules 10013. The sensor modules 10013 may be configured to measure a wide range of wavelengths. The sensor modules 10013 may be tuned and / or filtered to measure specific wavelengths, for example. The sensor modules 10013 may include, for example, individual sensors, a collection of sensors, a sensor array, a combination of sensor arrays, etc. For example, the sensor modules 10013 may include semiconductor components such as photodiodes, CMOS (complementary metal-oxide-semiconductor) image sensors, and CCD (charge-coupled device) image sensors.

[0244] The laser light sensor 10012 may include a dual CMOS array. Figure 31B shows an exemplary laser light sensor 10030. The laser light sensor 10030 may include two sensor modules 10032 and 10034. The sensor modules 10032 and 10034 may be implemented as a dual parallel CMOS array. For example, the laser light sensor 10030 is incorporated into the shape factor of a surgical scope 10031 (e.g., a 7 mm diameter surgical scope) having two sensor modules 10032 and 10034 (e.g., two parallel 4 mm sensors). The laser light sensor 10030 may be configured to allow shifts between imaging modes. Modes may include, for example, three-dimensional stereoscopic imaging and two-dimensional simultaneous imaging (e.g., visual imaging with imaging for refractive power analysis and / or Doppler analysis). Modes may include, for example, imaging with a narrower or wider visualization range. The modes may include, for example, imaging with lower resolution or higher resolution and / or artifact correction. Sensor modules 10032 and 10034 may include different types of sensors. For example, the first sensor module 10032 may be a CMOS device. The second sensor module 10034 may be a different CMOS device. Differences in CMOS devices can enable greater diversity in light-gathering capabilities. For example, different CMOS devices can enable wider light contrast and / or better light-gathering. For example, the first sensor array 10032 may have more pixel detectors than the second sensor array 10034. The surgical scope 10031 may include one or more light sources 10036, such as a laser light illumination source.

[0245] Figure 31C is a graphical representation of exemplary operation of a pixel array for multiple frames. A sensor module (e.g., a CMOS sensor module) may incorporate patterns and / or techniques for light sensing. Light sensing techniques associated with the operation of the sensor module may incorporate filtering. Light sensing techniques associated with the sensor module may incorporate stroving of a light source. Examples of these techniques may include, for example, the techniques disclosed herein in relation to Figures 17C and 17D. A pattern of a stroving light source associated with the sensor module may be used to measure reflected light and generate information indicating the reflected light. The pixel array may be captured by rapidly stroving a visualization region at high speed with various light sources (either lasers or light-emitting diodes) having various central light wavelengths.

[0246] Stroving may cause the sensor to capture each pixel array associated with the corresponding wavelength. For example, in the first pattern 10038, red, green, and blue, as well as infrared (e.g., near-infrared) wavelength light may be stroved. Such stroving may cause the sensor to capture, for example, a first pixel array 10040 associated with the red wavelength, a second pixel array 10042 associated with the green wavelength, a third pixel array 10044 associated with the blue wavelength, a fourth pixel array 4046 associated with the green wavelength, a fifth pixel array 10048 associated with the infrared (e.g., near-infrared) wavelength, a sixth pixel array 10050 associated with the green wavelength, and a seventh pixel array 10052 associated with the blue wavelength. For example, in the second pattern 10054, red, green, and blue, as well as infrared (e.g., near-infrared) wavelength light may be stroved. Such stroving may cause the sensor to capture, for example, an eighth pixel array 10056 associated with the red wavelength, a ninth pixel array 10058 associated with the green wavelength, a tenth pixel array 10060 associated with the blue wavelength, an eleventh pixel array 10062 as...

Claims

1. A program for controlling the digital aspects of a surgical system comprising a processor, wherein when the program is loaded into the processor, the processor: It operates in the first operating mode, Receiving control parameters, Based on the received control parameters, it is decided to operate in a second operating mode, and the following is performed: Operating in the first operating mode means that This includes determining a first metric that represents the current state of the moving particles in at least a portion of the surgical field, Operating in the second operating mode means that A program that determines the first and second measurement criteria, wherein the second measurement criterion represents either the aggregation state of moving particles in at least a portion of the surgical field, or the current state of moving particles in at least a portion of the surgical field at a selectable tissue depth.

2. The program according to claim 1, wherein the control parameters include parameters indicating any one of power capacity, memory capacity, bandwidth capacity, and processing suitability.

3. The program according to claim 1, wherein the control parameters include parameters indicating processing suitability, and the processing suitability indicates a functional layer associated with either a user or an instrument.

4. The program according to any one of claims 1 to 3, further comprising displaying the first measurement criterion and the second measurement criterion as an overlay on an image including at least a portion of the surgical field.

5. Operating in the second operating mode means that Receiving situational awareness information, The program according to any one of claims 1 to 4, further comprising reducing noise based on the aforementioned situation recognition information.

Images