Surgical visualization and particle trend analysis system

Abstract

The surgical visualization system (108) may include particle trend analysis. The surgical visualization system may include a field programmable gate array (FPGA) configured to convert sensor information of the backscattered laser light into real-time information of particle movement (e.g., blood cells) in a portion of the surgical field. The system may communicate the real-time information from the FPGA to a remote processor for aggregation and analysis. The surgical visualization system may display both metrics representing the current state of the moving particles and the aggregation state of the moving particles. The system may display both the velocity and acceleration of particle movement for use by the surgeon.

Description

Technical Field

[0001] (Cross - reference to related applications) This application is related to the following, the content of each of which is incorporated herein by reference: U.S. Patent Application No. 15 / 940,663, filed on March 29, 2018, entitled "Surgical System Distributed Processing", U.S. Patent Application No. 15 / 940,704, filed on March 29, 2018, entitled "Use Of Laser Light And Red - Green - Blue Coloration To Determine Properties Of Back Scattered Light", U.S. Patent Application (Attorney Docket No. END9287USNP1), filed together with this application, entitled "Method for Operating Tiered Operation Modes in A Surgical System", U.S. Patent Application (Attorney Docket No. END9287USNP2), filed together with this application, entitled "Tiered - Access Surgical Visualization System", and U.S. Patent Application (Attorney Docket No. END9287USNP4), filed together with this application, entitled "Field Programmable Surgical Visualization System".

Background Art

[0002] Surgical systems often incorporate imaging systems that allow clinicians(s) to view the surgical site and / or one or more parts thereof on one or more displays, such as monitors. These displays may be localized in the surgical theater and / or remote. The imaging system may include a scope equipped with a camera that views the surgical site and transmits the view to a display accessible to the clinician. Examples of scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, coloscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, and exoscopy. The imaging system may be limited by the information that can be recognized by and / or transmitted to the clinician(s). For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognizable during surgery by certain imaging systems. Additionally, certain imaging systems may be unable to communicate and / or transmit certain information to the clinician(s) during surgery. [Overview of the project] [Means for solving the problem]

[0003] A surgical visualization system may include particle trend analysis. The surgical visualization system may include a field-programmable gate array (FPGA) configured to convert sensor information from backscattered laser light into real-time information of particle movement (e.g., blood cells) in a portion of the surgical field. The system may communicate the real-time information from the FPGA to a remote processor for aggregation and analysis. The surgical visualization system may display both a metric representing the current state of the moving particles and the aggregation state of the moving particles. For example, the system may display both the velocity and acceleration of particle movement for use by the surgeon.

[0004] According to various embodiments of the present invention, the following examples are provided. 1. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is: A laser light illumination source configured to illuminate at least a portion of the surgical field with laser light, A light sensor configured to receive reflected laser light, A field-programmable gate array configured to convert information indicating reflected laser light into information indicating moving particles within at least a portion of the surgical field, A display configured to show an image including at least a portion of the surgical field, A processor configured to control a display to overlay and display a first metric and a second metric onto an image, The first metric represents the current state of the moving particle in at least a portion of the surgical field. The second metric is a surgical visualization system that represents the aggregation state of moving particles in at least a portion of the surgical field. 2. The surgical visualization system according to Example 1, including data trends, of the aggregation state of moving particles in at least a portion of the surgical field. 3. The second metric represents aggregation over several seconds and is shown as a trend, as described in Example 1 of the surgical visualization system. 4. The surgical visualization system according to Example 1, wherein the second metric includes the acceleration of moving particles in at least a portion of the surgical field. 5. A surgical visualization system according to any one of Examples 1 to 4, wherein the second metric is suitable for identifying any of the following: occlusion, vascular sealing / clamping efficiency of the instrument, vascular tree overview, and magnitude of vibration of movement over time. 6. A surgical visualization system according to any one of Examples 1 to 5, wherein the processor is configured to control the display to show a graphical trend animation superimposed on the image. 7. A surgical visualization system according to any one of Examples 1 to 6, further comprising a user interface for receiving a user selection of the type of unit to be used to display a second metric. 8. A surgical visualization system according to any one of Examples 1 to 7, further comprising a user interface for receiving user selection of tagged particle groups, wherein a second metric represents the aggregation state of the tagged particle groups. 9. A surgical visualization system according to any one of Examples 1 to 8, wherein the processor comprises a first processor associated with an image acquisition module, the system further comprises a second processor associated with an external processing resource, information indicating moving particles within at least a portion of the surgical field is transmitted to the second processor for aggregation analysis, and information indicating a second metric is transmitted from the second processor to the first processor. 10. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is A field-programmable gate array configured to convert information indicating reflected laser light into information indicating moving particles within at least a portion of the surgical field, A first processor, housed together with a field-programmable gate array and configured to generate a first metric representing the current state of a moving particle in at least a portion of the surgical field, A surgical visualization system comprising: a second processor located remotely from a field-programmable gate array, configured to receive information indicating moving particles within at least a portion of the surgical field, and to generate a second metric representing the aggregation state of the moving particles within at least a portion of the surgical field. 11. The surgical visualization system according to Embodiment 10, further comprising a display, wherein a second processor transmits a second metric to a first processor, and the first processor instructs the display to display the first metric and the second metric superimposed on an image including at least a portion of the surgical field. 12. The surgical visualization system according to Example 10 or 11, further comprising a laser light illumination source configured to illuminate at least a portion of the surgical field with laser light, and an optical sensor configured to receive reflected laser light. 13. A surgical visualization system according to any one of Examples 10 to 12, including data trends, of the aggregation state of moving particles in at least a portion of the surgical field. 14. The second metric represents aggregation over several seconds and is shown as a trend, as described in any one of Examples 10-12 of the surgical visualization system. 15. A surgical visualization system according to any one of Examples 10 to 12, wherein the second metric includes the acceleration of moving particles in at least a portion of the surgical field. 16. A surgical visualization system as described in any one of Examples 10-15, wherein the second metric is suitable for identifying any of the following: occlusion, instrument vascular sealing / clamping efficiency, vascular tree overview, and magnitude of motion vibration over time. 17. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is A display configured to show an image including at least a portion of the surgical field, A processor configured to control a display to overlay and display a first metric and a second metric onto an image, The first metric represents the current state of the moving particle in at least a portion of the surgical field. The second metric is a surgical visualization system that represents the aggregation state of moving particles in at least a portion of the surgical field. 18. The surgical visualization system according to Example 17, including data trends, of the aggregation state of moving particles in at least a portion of the surgical field. 19. The second metric represents aggregation over several seconds and is shown as a trend, as described in Example 17 of the surgical visualization system. 20. A surgical visualization system according to any one of Examples 17-19, further comprising a user interface for receiving user selection of tagged particle groups, wherein a second metric represents the aggregation state of the tagged particle groups. 21. The optical sensor is a system according to any one of Examples 1 to 10 and Examples 12 to 16, which provides information indicating reflected laser light. 22. The first metric represents information indicating a moving particle, provided by a field-programmable gate array, as described in any one of Examples 1 to 16. 23. The system according to any one of Examples 1 to 22, wherein the information indicating the reflected laser light includes one or more of the following: amplitude, frequency, wavelength, Doppler shift, and / or other time-domain or frequency-domain qualities. 24. The system according to any one of Examples 1 to 23, wherein the information indicating the moving particles includes one or more of the number of moving particles per unit time, particle speed, particle velocity, and / or volume. 25. The system according to any one of Examples 1 to 24, wherein the second metric is calculated by aggregating information indicating the movement of the particle over time and performing other statistical methods such as least-squares regression, polynomial fitting, mean, arithmetic mean, mode, maximum, minimum, variance and / or similar, or by calculating a value representing acceleration.

[0005] The above embodiments, particularly embodiments 1, 10, and 17, make it possible to acquire and display real-time data while simultaneously calculating and displaying aggregated data, and to inform clinicians or users about trends in clinically relevant data. This improves the clinician's or user's understanding of the current state of the imaginged surgical field, and proportionally improves the safety and effectiveness of the procedures performed by the clinician or user.

[0006] According to the above embodiments, particularly embodiments 1 and 10, the function of converting optical data from the surgical field into real-time data is performed by a field-programmable gate array, while the display of the real-time data and the processing of that data, aggregating and calculating that data, and displaying the aggregated data are performed by separate processors to mitigate the possibility that excessive processing load, power consumption, etc., may interfere with the proper acquisition and conversion of information.

[0007] According to the above embodiments, particularly embodiments 1, 11, and 17, additional information is presented as an overlay on the surgical field image, so that the clinician or user can always have both the image and clinically relevant information within their field of view. This improves the clinician or user's ability to monitor the procedure and ensure safety.

[0008] In the above embodiments, particularly Embodiment 6, the aggregated data is displayed in the form of trend animation, which may be more readily understandable to clinicians or users of surgical visualization systems than, for example, statically displayed metrics (e.g., values). This can improve clinicians' or users' ability to more easily assess the current and trending conditions within the imaged portion of the surgical field, thereby enabling them to make appropriate clinical decisions quickly and safely, and consequently improving the safety and effectiveness of the procedures they are performing.

[0009] In the embodiments described above, particularly in Embodiment 7, clinicians or users of the system can select the display of the unit most appropriate for the specific procedure they are performing. This not only improves the clinician's or user's ability to perform the procedure but also avoids unnecessary processing power or bandwidth being used for calculating or displaying irrelevant metrics.

[0010] In the above embodiments, particularly in embodiments 8 and 20, a clinician or system user can select a specific group of particles to be tagged, enabling them to track trends or other aggregated metrics within the group of particles that have a specific clinical relevance in the procedure being performed, thereby improving the clinician or user's ability to perform the procedure.

[0011] In the above embodiments, particularly in Embodiment 9 or 10, the first processor is associated with the image acquisition module, and the aggregation analysis is executed on the second processor. Since the potentially more computationally intensive task of aggregation analysis is executed by the second processor, it is possible to reduce the risk that the computational load of the aggregation analysis interferes with the display of real-time metrics. Without this risk reduction, there could be a risk of impairing the ability of a clinician or user to perform a procedure using the system, thereby improving the safety of the procedure being performed and / or reducing the time required for the procedure.

Brief Description of the Drawings

[0012] [Figure 1] It is a block diagram of an exemplary computer-implemented interactive surgical system. [Figure 2] It is a diagram of an exemplary surgical system being used to perform a surgical procedure in an operating room. [Figure 3] It is a diagram of an exemplary surgical hub paired with a visualization system, a robotic system, and intelligent instruments. [Figure 4] It is a diagram showing an exemplary surgical data network comprising a modular communication hub configured to connect a modular device located in one or more operating rooms of a healthcare facility or any room within a healthcare facility equipped with specialized equipment for surgical procedures to the cloud. [Figure 5] It is a diagram showing an exemplary computer-implemented interactive surgical system. [Figure 6] It is a diagram showing an exemplary surgical hub comprising a plurality of modules connected to a modular control tower. [Figure 7] It is a logic diagram showing an exemplary control system for a surgical instrument or tool. [Figure 8] It is a diagram showing a surgical instrument or tool comprising a plurality of motors that can be activated to perform various functions. [Figure 9] It is a diagram showing an exemplary situation awareness surgical system. [Figure 10]This figure shows an exemplary surgical procedure and reasoning timeline that a surgical hub can create from data detected at each step of the surgical procedure. [Figure 11] This is a block diagram of an exemplary computer-implemented interactive surgical system. [Figure 12] This is a block diagram illustrating the functional architecture of an exemplary computer-implemented interactive surgical system. [Figure 13] This is a block diagram of an exemplary computer-implemented interactive surgical system configured to adaptively generate updates to control programs for modular devices. [Figure 14] This figure shows an exemplary surgical system including a handle having a controller and a motor, an adapter releasably connected to the handle, and a loading unit releasably connected to the adapter. [Figure 15A] This is an illustrative flowchart for determining the operating mode. [Figure 15B] This is an illustrative functional block diagram for changing the operating mode. [Figure 16A] This is a diagram illustrating an exemplary visualization system. [Figure 16B] This is a diagram illustrating an exemplary visualization system. [Figure 16C] This is a diagram illustrating an exemplary visualization system. [Figure 16D] This is a diagram illustrating an exemplary visualization system. [Figure 17A] This is a diagram of multiple laser emitters that can be incorporated into an exemplary visualization system. [Figure 17B] This is a diagram illustrating the illumination of an image sensor with a Bayer pattern of a color filter. [Figure 17C] This is a graphical representation of the behavior of a pixel array across multiple frames. [Figure 17D] This is a schematic diagram showing an example of the operation sequence for chromaticity and luminance frames. [Figure 17E] This is a diagram showing an example of a sensor and emitter pattern. [Figure 17F] This is a graphical representation of the operation of a pixel array. [Figure 18] This figure shows exemplary equipment for NIR spectroscopy. [Figure 19] This figure shows exemplary equipment for measuring NIRS based on Fourier transform infrared imaging. [Figure 20A] This figure shows the change in the wavelength of light scattered by moving blood cells. [Figure 20B] This figure shows the change in the wavelength of light scattered by moving blood cells. [Figure 20C] This figure shows the change in the wavelength of light scattered by moving blood cells. [Figure 21] This figure shows exemplary instruments that may be used to detect the Doppler shift of laser light scattered from a tissue. [Figure 22] This figure shows an exemplary optical effect of light striking a tissue that has a subsurface structure. [Figure 23] This figure shows an exemplary optical effect of light striking a tissue that has a subsurface structure. [Figure 24A] This figure shows the detection of migrating blood cells at tissue depth based on laser Doppler analysis at various laser wavelengths. [Figure 24B] This figure shows the detection of migrating blood cells at tissue depth based on laser Doppler analysis at various laser wavelengths. [Figure 24C] This figure shows the detection of migrating blood cells at tissue depth based on laser Doppler analysis at various laser wavelengths. [Figure 24D] This figure shows the detection of migrating blood cells at tissue depth based on laser Doppler analysis at various laser wavelengths. [Figure 25] This figure shows an example of how Doppler imaging can be used to detect the presence of subsurface blood vessels. [Figure 26] This figure shows the Doppler shift of blue light caused by blood cells flowing in blood vessels below the surface. [Figure 27]This figure shows an example of the location of deep subsurface blood vessels. [Figure 28] This figure shows an example of the location of superficial blood vessels beneath the surface. [Figure 29] This figure shows an exemplary composite image including a surface image and an image of subsurface blood vessels. [Figure 30] This figure shows an exemplary method for determining the depth of a feature area on the surface of a tissue sample. [Figure 31A] This is a diagram illustrating an exemplary visualization system. [Figure 31B] This figure shows an exemplary laser light sensor having two sensor modules. [Figure 31C] This figure shows a graphical representation of the exemplary behavior of a pixel array across multiple frames. [Figure 32] This figure shows an exemplary method for determining the operating mode. [Figure 33] This figure shows an exemplary method for displaying real-time and trend information to users. [Figure 34] This diagram illustrates an exemplary user interface where real-time and / or trend information is displayed. [Figure 35] This is a diagram illustrating an exemplary upgrade framework. [Figure 36] This figure shows an exemplary method for reconfiguring a field-programmable gate array. [Modes for carrying out the invention]

[0013] The surgical hub may have collaborative interaction with one of several means of displaying images from a laparoscope and information from one of several other smart devices. The hub may have the ability to interact with multiple displays using algorithms or control programs that enable combined display and control of data distributed across multiple displays communicating with the hub.

[0014] Referring to Figure 1, the computer-implemented interactive surgical system 100 may include one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 which may include a remote server 113 connected to a storage device 105). Each surgical system 102 may include at least one surgical hub 106, which communicates with the cloud 104 which may include the remote server 113. In one example, as shown in Figure 1, the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some embodiments, the surgical system 102 may include M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P are integers of 1 or more.

[0015] In various embodiments, the visualization system 108 may include one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically positioned relative to the sterile field, as shown in Figure 2. In one embodiment, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described under the heading “Advanced Imaging Acquisition Module” in U.S. Patent Provisional Application No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed December 28, 2017, and in U.S. Patent Publication No. 2019-0200844(A1), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed December 4, 2018, both of which are incorporated herein by reference in their entirety.

[0016] As shown in Figure 2, the main display 119 is positioned in the sterile field so that it can be viewed by the operator on the operating table 114. In addition, a visualization tower 111 is positioned outside the sterile field. The visualization tower 111 may include a first non-sterile display 107 and a second non-sterile display 109, facing opposite directions from each other. A visualization system 108 guided by a hub 106 is configured to utilize displays 107, 109, and 119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 106 can cause the visualization system 108 to display snapshots of the surgical site recorded by the imaging device 124 on the non-sterile displays 107 or 109 while maintaining live video of the surgical site on the main display 119. The snapshots on the non-sterile displays 107 or 109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0017] In one embodiment, the hub 106 may also be configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 to a main display 119 in the sterile field, which can then be viewed by a sterile operator at the operating table. In one example, the input may take the form of modifications to a snapshot displayed on the non-sterile display 107 or 109, which can then be sent to the main display 119 by the hub 106.

[0018] Referring to Figure 2, the surgical instrument 112 may be used as part of the surgical system 102 in a surgical procedure. The hub 106 may also be configured to coordinate the flow of information to the display of the surgical instrument 112. For example, disclosures are found in U.S. Patent Provisional Application No. 62 / 611,341, entitled "INTERACTIVE SURGICAL PLATFORM," filed December 28, 2017, and U.S. Patent Publication No. 2019-0200844(A1), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018, both of which are incorporated herein by reference in their entirety. Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 can be sent by the hub 106 to the surgical instrument display 115 in the sterile field, which can then be viewed by the operator of the surgical instrument 112. Examples of surgical instruments suitable for use with the surgical system 102 are described, for example, in the section "Surgical Instrument Hardware" in U.S. Patent Provisional Application No. 62 / 611,341, entitled "INTERACTIVE SURGICAL PLATFORM," filed December 28, 2017, and in U.S. Patent Publication No. 2019-0200844(A1), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018. Both of these disclosures are incorporated herein by reference in their entirety.

[0019] Figure 2 shows an example of a surgical system 102 used to perform a surgical procedure on a patient lying on an operating table 114 in an operating room 116. A robotic system 110 may be used as part of the surgical system 102 in a surgical procedure. The robotic system 110 may include a surgeon's console 118, a patient-side cart 120 (surgical robot), and a surgical robot hub 122. While the surgeon views the surgical site through the surgeon's console 118, the patient-side cart 120 can manipulate at least one detachably connected surgical instrument 117 through a minimally invasive incision in the patient's body. Images of the surgical site are acquired by a medical imaging device 124, which can be manipulated by the patient-side cart 120 to orient the imaging device 124. The robotic hub 122 can be used to process images of the surgical site, which can then be displayed to the surgeon through the surgeon's console 118.

[0020] Other types of robotic systems can be easily adapted for use with the surgical system 102. Various examples of robotic systems and surgical tools suitable for use together are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed on 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0021] Various examples of cloud-based analytical methods performed by Cloud 104 and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed 4 December 2018, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” the disclosure of which is incorporated herein by reference in its entirety.

[0022] In various embodiments, the imaging device 124 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0023] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0024] One or more illuminators may be configured to illuminate electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0025] The invisible spectrum (e.g., the non-emission spectrum) is the portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0026] In various embodiments, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0027] The imaging device may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is required in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process described above is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient who is ready for surgical treatment. The sterile field may include hand-washed team members wearing appropriate clothing, as well as all equipment and restraints within that area.

[0028] Referring here to Figure 3, a hub 106 is shown that communicates with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, and a storage array 134, and an operating room mapping module 133. In certain embodiments, as shown in Figure 3, the hub 106 further includes a fume extraction module 126 and / or aspiration / irrigation module 128. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, aspiration of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Valuable time can be lost dealing with this problem during surgical procedures. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The modular enclosure 136 of the hub provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines. An aspect of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within a docking station of the hub enclosure. The docking station includes data contacts and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed in a single unit. In one aspect, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid, and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the fume exhaust component.In one embodiment, the above-mentioned fluid line is a first fluid line, and a second fluid line extends from the remote surgical site to a suction and irrigation module slidably received within the hub enclosure. In one embodiment, the hub enclosure comprises a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Embodiments of the present disclosure present a solution in which a modular enclosure 136 of the hub is configured to house various generators and facilitate interactive communication between them. One advantage of the modular enclosure 136 of the hub is that it allows for the rapid removal and / or replacement of various modules. Embodiments of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. A modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the first power and data contacts. Further, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station having a second docking port including second data and power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is slidably movable to disengage from the second power and data contacts.In addition, the modular surgical enclosure also includes a communication bus between a first docking port and a second docking port, configured to facilitate communication between a first energy generator module and a second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented relating to a modular enclosure 136 of a hub that enables modular integration of a generator module 140, a smoke exhaust module 126, and a suction / irrigation module 128. The modular enclosure 136 of the hub further facilitates interactive communication between modules 140, 126, and 128. The generator module 140 may be a generator module comprising integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the modular enclosure 136 of the hub. The generator module 140 may be configured to connect to a unipolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules interacting via the modular enclosure 136 of the hub. The modular enclosure 136 of the hub may be configured to facilitate the insertion of multiple generators and interactive communication between generators docked to the modular enclosure 136 of the hub, so that multiple generators function as a single generator.

[0029] Figure 4 shows a surgical data network 201 comprising a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to a cloud-based system (e.g., a cloud 204 which may include a remote server 213 connected to a storage device 205). In one embodiment, the modular communication hub 203 comprises a network hub 207 and / or a network switch 209 that communicate with a network router. The modular communication hub 203 can also be connected to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to travel from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional mechanisms that constitute each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0030] Modular devices 1a-1n located in the operating room may be connected to a modular communication hub 203. A network hub 207 and / or a network switch 209 may be connected to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to a local computer system 210 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be connected to a network switch 209. The network switch 209 may be connected to a network hub 207 and / or a network router 211 to connect devices 2a-2m to a cloud 204. Data associated with devices 2a-2n may be transferred to a cloud 204 via the network router 211 for data processing and manipulation. Data associated with devices 2a-2m may also be transferred to a local computer system 210 for local data processing and manipulation.

[0031] It will be understood that the surgical data network 201 can be extended by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. A modular communication hub 203 may be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. The modular communication hub 203 is connected to a display 212 to display, for example, images acquired by some of the devices 1a-1n / 2a-2m during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 203 of the surgical data network 201, such as an imaging module 138 connected to an endoscope, a generator module 140 connected to an energy-based surgical device, a smoke extraction module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module.

[0032] In one embodiment, the surgical data network 201 may include a combination of network hubs, network switches, and network routers that connect devices 1a-1n / 2a-2m to the cloud. One or all of the devices 1a-1n / 2a-2m connected to the network hub or network switch can collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” can be used as a metaphor for “internet,” but the term is not limited to that. Therefore, the term “cloud computing” may be used herein to refer to “a type of internet-based computing” in which various services such as servers, storage, and applications are delivered via the internet to a modular communication hub 203 and / or computer system 210 located in a surgical setting (e.g., a fixed, mobile, temporary, or field operating room or space), and to devices connected to the modular communication hub 203 and / or computer system 210. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located in the operating room. The hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0033] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, surgical data networks can provide improved surgical outcomes, reduced costs, and increased patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud 204 or a local computer system 210, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcome of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be performed on tissue-specific sites and conditions. Such data analysis may further involve prognostic analysis processing, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0034] The operating room devices 1a-1n may be connected to the modular communication hub 203 via a wired or wireless channel, depending on the configuration of devices 1a-1n with respect to the network hub. In one embodiment, the network hub 207 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same operating room network. The network hub 207 may collect data in packet form but transmit them to the router in half-duplex mode. The network hub 207 does not need to store any media access control / Internet Protocol (MAC / IP) for transferring any device data. Only one of devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 does not need to have a routing table or intelligence regarding the destination of information and broadcasts all network data to each connection and to the remote server 213 (Figure 4) on the cloud 204. While the Network Hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and could cause bottlenecks.

[0035] Operating room devices 2a-2m may be connected to network switch 209 via a wired or wireless channel. Network switch 209 operates within the data link layer of the OSI model. Network switch 209 may be a multicast device for connecting devices 2a-2m located in the same operating room to the network. Network switch 209 may transmit data in the form of frames to network router 211, but operates in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 209. Network switch 209 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0036] The network hub 207 and / or network switch 209 may be connected to the network router 211 to connect to the cloud 204. The network router 211 operates within the network layer of the OSI model. The network router 211 creates a route for sending data packets received from the network hub 207 and / or network switch 211 to cloud-based computing resources for further processing and manipulation of data collected by one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, such as different networks located in different operating rooms of the same medical facility or different operating rooms of different medical facilities. The network router 211 can send data in packet form to the cloud 204 and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to transfer data.

[0037] In one example, the network hub 207 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 207 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0038] In some embodiments, the operating room devices 1a-1n / 2a-2m can exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with the modular communication hub 203 via the Bluetooth wireless technology standard to build a personal area network (PAN). The operating room devices 1a-1n / 2a-2m can communicate with the modular communication hub 203 via a number of wireless or wired communication standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long-Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G and beyond. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and Ev-DO.

[0039] The modular communication hub 203 can function as a central connection point for one or all of the operating room devices 1a-1n / 2a-2m and can handle a data type known as a frame. A frame can carry data generated by devices 1a-1n / 2a-2m. When a frame is received by the modular communication hub 203, it is amplified and transmitted to the network router 211, which then transfers this data to cloud computing resources using a number of wireless or wired communication standards or protocols as described herein.

[0040] The modular communication hub 203 may be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Because the modular communication hub 203 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room devices 1a-1n / 2a-2m.

[0041] Figure 5 shows a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204 which may include a remote server 213. In one embodiment, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to a plurality of operating room devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 236 includes a modular communication hub 203 connected to a computer system 210.

[0042] As shown in the embodiment of Figure 5, the modular control tower 236 can be connected to an imaging module 238 which can be connected to an endoscope 239, a generator module 240 which can be connected to an energy device 241, a fume exhaust module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 which is optionally connected to a display 237, and a non-contact sensor module 242. The operating room devices can be connected to cloud computing resources and data storage via the modular control tower 236. The robot hub 222 may also be connected to the modular control tower 236 and cloud computing resources. In particular, the device / instrument 235 and the visualization system 208 may be connected to the modular control tower 236 via wired or wireless communication standards or protocols as described herein. The modular control tower 236 may be connected to a hub display 215 (e.g., a monitor, screen) to display and overlay images received from imaging modules, device / instrument displays, and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with the images and overlaid images.

[0043] Figure 6 shows a surgical hub 206 comprising multiple modules connected to a modular control tower 236. The modular control tower 236 may comprise a modular communication hub 203, such as a network connectivity device, and a computer system 210, for example, to perform local processing, visualization, and imaging. As shown in Figure 6, the modular communication hub 203 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 203, and data associated with the modules may be transferred to the computer system 210, cloud computing resources, or both. As shown in Figure 6, each network hub / switch within the modular communication hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and local displays 217. Communication to the cloud 204 can be done via either a wired communication channel or a wireless communication channel.

[0044] The surgical hub 206 may use a non-contact sensor module 242 to measure the dimensions of the operating room and to generate a map of the surgical site using either an ultrasonic non-contact measuring device or a laser non-contact measuring device. As described in the section "Surgical Hub Spatial Awareness Within an Operating Room" of U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," and in U.S. Patent Application Publication No. 2019-0200844(A1), filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the entirety of both disclosures is incorporated herein by reference, an ultrasonic-based non-contact sensor module may scan an operating room by transmitting bursts of ultrasound and receiving echoes as the bursts of ultrasound reflect off the outer walls of the operating room, wherein the sensor module is configured to determine the size of the operating room and adjust the distance limits for Bluetooth pairing. A laser-based non-contact sensor module can, for example, scan an operating room by transmitting laser light pulses, receive the laser light pulses reflected from the outer wall of the operating room, compare the phase of the transmitted pulses with the received pulses to determine the size of the operating room, and adjust the Bluetooth pairing distance limit.

[0045] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 may be connected via a system bus to a communication module 247, storage 248, memory 249, non-volatile memory 250, and an input / output interface 251. The system bus may be any of several types of bus structures, including memory buses or memory controllers, peripheral buses or external buses, and / or local buses, using any variety of available bus architectures, including, but not limited to, 9-bit buses, industrial standard architecture (ISA), micro-charmel architecture (MSA), extended ISA (EISA), intelligent drive electronics (IDE), VESA local bus (VLB), peripheral component interconnect (PCI), USB, advanced graphics port (AGP), personal computer memory card international association (PCMCIA), small computer systems interface (SCSI), or any other proprietary bus.

[0046] Processor 244 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which are available in the product datasheet.

[0047] In one embodiment, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0048] System memory can include volatile and non-volatile memory. The basic input / output system (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory can include random-access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0049] The computer system 210 may also include removable / non-removable, volatile / non-volatile computer storage media, such as disk storage devices. Disk storage devices may include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage devices may include the above-mentioned storage media independently or in combination with other storage media. Other storage media may include, but are not limited to, optical disk drives such as compact disc ROM devices (CD-ROM), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROM). Removable or non-removable interfaces may be used to facilitate connection of disk storage devices to the system bus.

[0050] It should be understood that the computer system 210 may include software that acts as an intermediary between the user and basic computer resources, as described in a preferred operating environment. Such software may include an operating system. An operating system, which may be stored on a disk storage device, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on a disk storage device. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0051] The user can input commands or information to the computer system 210 via input devices 251 connected to the I / O interface 251. Input devices may include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards, microphones, joysticks, gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor via the system bus through interface ports 251. Interface ports 251 may include, for example, serial ports, parallel ports, game ports, and USB ports. Output devices 251 may use some of the same types of ports as the input devices 251. Therefore, for example, a USB port may be used to provide input to the computer system and output information from the computer system to an output device. Output adapters may be provided to indicate the presence of several output devices, particularly monitors, displays, speakers, and printers, among others, which may require special adapters. Examples of output adapters include video and sound cards that provide a means of connection between the output device and the system bus, but these are illustrative and not limiting. Note that other devices and / or systems of devices, such as remote computers, may provide both input and output functions.

[0052] Computer system 210 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to the computer system. For brevity, only memory storage devices are shown along with the remote computers. Remote computers may be logically connected to the computer system via network interfaces, and subsequently physically connected via communication connections. Network interfaces may encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies may include fiber distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated services digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL).

[0053] In various embodiments, the computer system 210 in Figure 6, the imaging module 238 in Figures 5-6, and / or the visualization system 208, and / or the processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction, multiple data (SIMD) or multiple-instruction, multiple data (MIMD) technology. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0054] A communication connection(s) may refer to hardware / software used to connect a network interface to a bus. For the sake of clarity of the example, the communication connection(s) are shown as being inside the computer system, but they may also be outside the computer system 210. For illustrative purposes only, hardware / software required for connection to a network interface may include internal and external technologies such as standard telephone-grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0055] Figure 7 shows a logic diagram of a control system 470 for a surgical instrument or tool according to one or more embodiments of the present disclosure. The system 470 may include a control circuit. The control circuit may include a microcontroller 461 having a processor 462 and memory 468. For example, one or more of sensors 472, 474, and 476 provide real-time feedback to the processor 462. A motor 482 driven by a motor driver 492 drives an I-beam knife element by operably connecting a longitudinally movable displacement member. A tracking system 480 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to a processor 462 which can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the launch member, launch bar, and I-beam knife element. Additional motors may be provided in the tool driver interface to control the firing of the I-beam, the movement of the occluder, the rotation of the shaft, and joint movement. A display 473 may display various operating conditions of the instrument and may include a touchscreen function for data input. The information displayed on display 473 can be overlaid with images acquired via the endoscopic imaging module.

[0056] In one embodiment, the microcontroller 461 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 461 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance above 40MHz, 32KB single-cycle SRAM, internal ROM with StellarisWare® software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels.

[0057] In one embodiment, the microcontroller 461 may include a safety controller, which may include two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured specifically for IEC61508 and ISO26262 safety limit applications, in particular, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0058] The microcontroller 461 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 461 may include a processor 462 and memory 468. The electric motor 482 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor driver 492 may be the A3941 available from Allegro Microsystems, Inc. Other motor drivers can be readily substituted for use in the tracking system 480 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” which is incorporated herein by reference in its entirety.

[0059] The microcontroller 461 may be programmed to provide precise control over the speed and position of the displacement member and joint motion system. The microcontroller 461 may be configured to calculate the response within its software. The calculated response may be compared with the measured response of the actual system to obtain an "observed" response, which is used to determine the actual feedback. The observed response may be a well-adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, thereby enabling the detection of external influences on the system.

[0060] In some examples, the motor 482 may be controlled by a motor driver 492 and may be used by a surgical instrument or tool launching system. In various forms, the motor 482 may be, for example, a brushed DC-driven motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 482 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 492 may comprise, for example, an H-bridge driver including a field-effect transistor (FET). The motor 482 may be powered by a power supply assembly removably mounted on a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may comprise a battery that may include a number of battery cells connected in series, which can be used as a power source for powering a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be a lithium-ion battery that can be coupled to and detached from the power supply assembly.

[0061] The motor driver 492 may be the A3941, available from Allegro Microsystems, Inc. The A3941 492 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET), particularly designed for inductive loads such as brushed DC motors. The driver 492 may have a built-in charge pump regulator, which can provide full (over 10V) gate drive to battery voltages down to 7V, and can enable the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to provide the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side or low-side FET. The power FET can be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be easily substituted for use in the tracking system 480 with an absolute positioning system.

[0062] The tracking system 480 may comprise a controlled motor drive circuit arrangement comprising a position sensor 472 according to one aspect of the present disclosure. The position sensor 472 for the absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 472 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include a contact-type displacement sensor or a non-contact-type displacement sensor.A linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical detection system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical detection system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0063] The electric motor 482 may include a rotary shaft that operably interfaces with a gear assembly mounted in meshing engagement with a set of drive teeth or a rack on the displacement member. The sensor element may be operably connected to the gear assembly such that one rotation of the position sensor 472 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply may provide power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing engagement with the corresponding drive gear of the gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0064] One rotation of the sensor element associated with the position sensor 472 may correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor mechanism may be connected via a gear reduction that results in the position sensor 472 completing one or more rotations relative to the full stroke of the displacement member. The position sensor 472 can complete multiple rotations relative to the full stroke of the displacement member.

[0065] To provide a unique position signal for two or more rotations of the position sensor 472, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches may be fed back to the microcontroller 461, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 472 is provided to the microcontroller 461. The position sensor 472 of the sensor mechanism may include an array of analog rotation sensors such as a magnetic sensor or a potentiometer, or an array of analog Hall effect elements, which output a unique combination of position signals or values.

[0066] The position sensor 472 may comprise any number of magnetic sensing elements, such as magnetic sensors, which are classified according to whether they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors may include numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear perturbations, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0067] In one embodiment, the position sensor 472 of a tracking system 480 equipped with an absolute positioning system may be equipped with a magnetic rotation absolute positioning system. The position sensor 472 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 472 interfaces with a microcontroller 461 to provide an absolute positioning system. The position sensor 472 may be a low-voltage, low-power component, but includes four Hall effect elements in the area of ​​the position sensor 472 located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and Boulder algorithm, may be provided to implement a simple and efficient algorithm for calculating hyperbolic and trigonometric functions that requires only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 461 via a standard serial communication interface, such as a serial peripheral interface (SPI). The position sensor 472 may offer 12-bit or 14-bit resolution. The position sensor 472 may also be an AS5055 chip, available in a small QFN 16-pin 4x4x0.85mm package.

[0068] The tracking system 480, which includes an absolute positioning system, may also include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 472, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors(s) may include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. To predict what the state and output of the physical system will be by knowing the input, the calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, and inductive resistance.

[0069] The absolute positioning system can provide the absolute position of a displacement member when the device is powered on, without requiring the displacement member to be moved back or forward to a reset (zero or home) position, as may be necessary with conventional rotary encoders that simply count the number of forward or backward steps taken by the motor 482 to estimate the position of a device actuator, drive bar, knife, etc.

[0070] For example, a sensor 474, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of an end effector, such as the amplitude of strain exerted on the anvil during clamping, which can indicate the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to a processor 462. Instead of, or in addition to, sensor 474, a sensor 476, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. For example, sensor 476, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and deform into contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 478 may be used to measure the current drawn in by the motor 482. The force required to propel the launching member forward may correspond, for example, to the current drawn in by the motor 482. The measured force can be converted into a digital signal and provided to the processor 462.

[0071] In one embodiment, a strain gauge sensor 474 can be used to measure the force applied to tissue by the end effector. A strain gauge can be connected to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 474, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 474 can measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 462 of the microcontroller 461. A load sensor 476 can measure the force used to operate a knife element, for example, to cut tissue trapped between an anvil and a staple cartridge. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor can also be converted into a digital signal and provided to the processor 462.

[0072] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 474 and 476 respectively, can be used by the microcontroller 461 to characterize the selected position of the launcher and / or the corresponding values ​​of the launcher's velocity. In one example, memory 468 can store techniques, equations, and / or lookup tables that can be used by the microcontroller 461 during evaluation.

[0073] The control system 470 for surgical instruments or tools may also include a wired communication circuit or a wireless communication circuit for communicating with a modular communication hub 203, as shown in Figures 5 and 6.

[0074] Figure 8 shows a surgical instrument or tool equipped with multiple motors that can be activated to perform various functions. In a particular example, the first motor can be activated to perform a first function, the second motor can be activated to perform a second function, the third motor can be activated to perform a third function, the fourth motor can be activated to perform a fourth function, and so on. In a particular example, the multiple motors of the robotic surgical instrument 600 can be activated individually to produce firing, closing, and / or jointing motions in the end effector. The firing, closing, and / or jointing motions can be transmitted to the end effector, for example, via a shaft assembly.

[0075] In certain examples, the surgical instrument system or tool may include a firing motor 602. The firing motor 602 may be operably connected to a firing motor drive assembly 604, which can be configured to transmit the firing motion generated by the motor 602 to an end effector, specifically to displace an I-beam element. In certain examples, the firing motion generated by the motor 602 may, for example, deploy a staple from a staple cartridge into tissue captured by the end effector and / or advance the cutting edge of the I-beam element to cut the captured tissue. The I-beam element can be retracted by reversing the direction of the motor 602.

[0076] In certain examples, the surgical instrument or tool may include a closure motor 603. The closure motor 603 may be operably coupled to a closure motor drive assembly 605, which may be configured to specifically displace a closure tube to close the anvil and transmit the closure motion generated by the motor 603 to an end effector to compress tissue between the anvil and the staple cartridge. The closure motion allows the end effector to transition from an open configuration to an approach configuration, for example, to capture tissue. The end effector may be moved to an open position by reversing the direction of the motor 603.

[0077] In certain examples, a surgical instrument or tool may include, for example, one or more articular motion motors 606a, 606b. The motors 606a, 606b may be operably connected to corresponding articular motion motor drive assemblies 608a, 608b, which may be configured to transmit the articular motion generated by the motors 606a, 606b to an end effector. In certain examples, the articular motion may cause, for example, the end effector to articulate relative to a shaft.

[0078] As described herein, a surgical instrument or tool may include multiple motors that can be configured to perform various independent functions. In certain examples, multiple motors of a surgical instrument or tool can be activated individually or separately to perform one or more functions while other motors remain stopped. For example, articulation motors 606a and 606b can be activated to articulate an end effector while firing motor 602 remains stopped. Alternatively, firing motor 602 can be activated to fire multiple staples and / or advance a cutting edge while articulation motor 606 remains stopped. Furthermore, a closure motor 603 may be activated simultaneously with firing motor 602 to advance the closure tube and I-beam element distally, as described in more detail below herein.

[0079] In certain examples, a surgical instrument or tool may include a common control module 610 that can be used with multiple motors of the surgical instrument or tool. In certain examples, the common control module 610 may correspond to one of the multiple motors at a time. For example, the common control module 610 may be individually connectable and disconnectable to multiple motors of a robotic surgical instrument. In certain examples, multiple motors of a surgical instrument or tool may share one or more common control modules, such as the common control module 610. In certain examples, multiple motors of a surgical instrument or tool can engage with the common control module 610 individually and selectively. In certain examples, the common control module 610 can selectively switch between interfacing with one of the multiple motors of the surgical instrument or tool and interfacing with another of the multiple motors of the surgical instrument or tool.

[0080] In at least one example, the common control module 610 can be selectively switched between an operable engagement with the articulation motors 606a, 606b and an operable engagement with either the firing motor 602 or the closing motor 603. In at least one embodiment, as shown in Figure 8, the switch 614 can move or transition between multiple positions and / or states. For example, in a first position 616, the switch 614 may electrically connect the common control module 610 to the firing motor 602; in a second position 617, the switch 614 may electrically connect the common control module 610 to the closing motor 603; in a third position 618a, for example, the switch 614 may electrically connect the common control module 610 to the first articulation motor 606a; and in a fourth position 618b, the switch 614 may electrically connect the common control module 610 to the second articulation motor 606b. In certain examples, a separate common control module 610 may also be electrically connected to the launch motor 602, the closing motor 603, and the joint motion motors 606a, 606b. In certain examples, the switch 614 may be a mechanical switch, an electromechanical switch, a solid switch, or any preferred switching mechanism.

[0081] Each of the motors 602, 603, 606a, and 606b may be equipped with a torque sensor for measuring the output torque on the motor shaft. The force on the end effector may be sensed in any conventional manner, such as by force sensors on the outside of the jaws or by torque sensors on the motors that actuate the jaws.

[0082] In various examples, as shown in Figure 8, the common control module 610 may include a motor driver 626 which may comprise one or more H-bridge FETs. The motor driver 626 may modulate the power transmitted from the power supply 628 to the motor connected to the common control module 610, for example, based on input from a microcontroller 620 ("controller"). In certain examples, as described herein, the microcontroller 620 can be used, for example, to determine the current drawn by the motor while the motor is connected to the common control module 610.

[0083] In certain examples, the microcontroller 620 may include a microprocessor 622 ("processor") and one or more non-temporary computer-readable media or memory units 624 ("memory"). In certain examples, the memory 624 may store various program instructions, which, when executed, cause the processor 622 to perform some of the functions and / or calculations described herein. In certain examples, one or more of the memory units 624 may be linked to the processor 622, for example.

[0084] In certain examples, power supply 628 may be used to power, for example, a microcontroller 620. In certain examples, power supply 628 may include, for example, a battery (or "battery pack" or "power pack") such as a lithium-ion battery. In certain examples, the battery pack may be configured to be removably attached to a handle in order to power a surgical instrument 600. A number of battery cells connected in series may be used as power supply 628. In certain examples, power supply 628 may be, for example, replaceable and / or rechargeable.

[0085] In various examples, the processor 622 can control the motor driver 626 to control the position, direction of rotation, and / or speed of a motor connected to a common control module 610. In specific examples, the processor 622 can signal the motor driver 626 to stop and / or disable a motor connected to the common control module 610. The term “processor,” as used herein, should be understood to include any suitable microprocessor, microcontroller, or other basic computing device that integrates the functions of a computer’s central processing unit (CPU) on one or up to several integrated circuits. A processor can be a multipurpose programmable device that takes digital data as input, processes that data according to instructions stored in memory, and provides the results as output. It can be an example of sequential digital logic, as it has internal memory. A processor can operate with numbers and symbols represented in binary.

[0086] The processor 622 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In a particular example, the microcontroller 620 may be, for example, the LM 4F230H5QR available from Texas Instruments. In at least one embodiment, the Texas Instruments LM4F230H5QR is an ARM Cortex-M4F processor core that includes, among other features readily available in the product datasheet, 256KB of on-chip memory of single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and one or more 12-bit ADCs with 12 analog input channels. Other microcontrollers may be readily substituted for use with module 4410. Therefore, this disclosure should not be limited to this context.

[0087] Memory 624 may include program instructions for controlling each of the motors of the surgical instrument 600, which can be connected to a common control module 610. For example, memory 624 may include program instructions for controlling the firing motor 602, the closing motor 603, and the joint movement motors 606a, 606b. Such program instructions can cause the processor 622 to control the firing function, closing function, and joint movement function according to input from an algorithm or control program of the surgical instrument or tool.

[0088] For example, one or more mechanisms and / or sensors, such as sensor 630, can be used to inform the processor 622 of program instructions to be used in a particular setting. For example, sensor 630 can inform the processor 622 to use program instructions related to the firing, closing, and joint movement of the end effector. In a particular example, sensor 630 may include a position sensor that can be used to sense the position of switch 614, for example. Thus, if the processor 622 detects, for example, that switch 614 is in a first position 616 via sensor 630, it can use a program instruction associated with the firing of the end effector's I-beam; if the processor 622 detects, for example, that switch 614 is in a second position 617 via sensor 630, it can use a program instruction associated with the closing of the anvil; and if the processor 622 detects, for example, that switch 614 is in a third position 618a or a fourth position 618b via sensor 630, it can use a program instruction associated with the joint movement of the end effector.

[0089] Figure 9 shows a diagram of a situation-aware surgical system 5100 relating to at least one aspect of the present disclosure. In some examples, the data source 5126 may include, for example, a modular device 5102 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 5122 (e.g., an EMR database containing patient records), and a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor). The surgical hub 5104 may be configured to derive contextual information about a surgical procedure from the data, for example, based on a specific combination(s) of received data or a specific order in which the data is received from the data source 5126. Contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, a specific step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This function relating to several aspects of the surgical hub 5104 for deriving or inferring information about a surgical procedure from received data is sometimes referred to as “situational awareness.” For example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104, for deriving contextual information related to a surgical procedure from received data.

[0090] The situational awareness system of the surgical hub 5104 can be configured to derive contextual information from data received from the data source 5126 in various different ways. For example, the situational awareness system may include a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 5122, patient monitoring device 5124, and / or modular device 5102) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In some examples, the situational awareness system may include a lookup table that stores pre-characterized contextual information about the surgical procedure, associated with one or more inputs (or ranges of inputs) that correspond to that contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situational awareness system to control the modular device 5102. In some examples, contextual information received by the situation awareness system of the surgical hub 5104 is associated with a specific control adjustment, or a set of control adjustments, of one or more modular devices 5102. In some examples, the situation awareness system may include a further machine learning system, a lookup table, or other such system that generates or retrieves one or more control adjustments of one or more modular devices 5102 when contextual information is provided as input.

[0091] The surgical hub 5104, which incorporates a situational awareness system, can bring many advantages to the surgical system 5100. One advantage could be improved interpretation of the detected and collected data, which would improve processing accuracy during the surgical procedure and / or the use of the data. Returning to the previous example, the situational awareness surgical hub 5104 can determine what type of tissue is being operated on, so if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situational awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the type of tissue.

[0092] The type of tissue being operated on can affect the adjustments made to the compression rate and load threshold of surgical staple fasteners and cutting instruments for measuring specific interstitial gaps. The situational awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic or abdominal surgery, thereby allowing the surgical hub 5104 to determine whether the tissue being clamped by the end effector of the surgical staple fastener and cutting instrument is the lung (in the case of thoracic surgery) or the stomach (in the case of abdominal surgery). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical staple fastener and cutting instrument according to the type of tissue.

[0093] The type of body cavity being operated on during an aeration procedure can affect the function of the fume exhauster. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing aeration) and determine the type of procedure. Generally, since certain types of procedures may be performed in specific body cavities, the surgical hub 5104 can appropriately control the motor speed of the fume exhauster to match the body cavity being operated on. In this way, the situational awareness surgical hub 5104 can provide a consistent amount of fume exhaust for both thoracic and abdominal surgeries.

[0094] The type of procedure being performed can affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, arthroscopy may require a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The situation-aware surgical hub 5104 can determine whether the surgical procedure is an arthroscopy. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situation-aware surgical hub 5104 can determine what type of surgical procedure is being performed and then customize the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue profile for the surgical procedure. Furthermore, the situation-aware surgical hub 5104 can be configured to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument not simply for each procedure, but throughout the course of the surgical procedure. The situation-aware surgical hub 5104 can determine which steps of the surgical procedure are being performed or are continuing, and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type according to the steps of the surgical procedure.

[0095] In some cases, the surgical hub 5104 may also derive data from additional data sources 5126 to improve conclusions drawn from one data source 5126. The contextually aware surgical hub 5104 may enhance data received from the modular device 5102 with contextual information constructed from other data sources 5126 regarding the surgical procedure. For example, the contextually aware surgical hub 5104 may be configured to determine whether hemostasis has been achieved (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, video or image data may not be conclusive. Therefore, in one example, the surgical hub 5104 may be further configured to make a determination regarding the integrity of the staple line or tissue weld by comparing physiological measurements (e.g., blood pressure detected by a BP monitor communicably connected to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (Figure 2) communicably connected to the surgical hub 5104). In other words, the context-aware system of the surgical hub 5104 can provide additional context when analyzing visualization data by considering physiological measurement data. This additional context can be useful when the visualization data itself may not be definitive or may be incomplete.

[0096] For example, the situational awareness surgical hub 5104 can pre-activate the generator to which an RF electrosurgical instrument is connected if it determines that the use of such an instrument is required in a subsequent step of the procedure. By pre-activating the energy source, the instrument can be ready for use as soon as the preceding steps of the procedure are completed.

[0097] The situational awareness surgical hub 5104 can determine, according to the shape(s) of the surgical site that the surgeon is expected to need to see, whether the current or subsequent steps of the surgical procedure require different views or magnifications on the display. The surgical hub 5104 can then appropriately pre-modify the displayed view (e.g., supplied from a medical imaging device for the visualization system 108), thereby automatically adjusting the display throughout the surgical procedure.

[0098] The situational awareness surgical hub 5104 can determine which steps of a surgical procedure are being performed or will be performed next, and whether specific data or comparisons of data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0099] Errors can be checked during the setup of a surgical procedure or during the procedure itself. For example, the situation-aware surgical hub 5104 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read the corresponding checklist, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 5104 has determined to be performed. In some examples, the surgical hub 5104 can be configured to compare a list of items for the procedure and / or a list of devices paired with the surgical hub 5104 to a recommended or expected catalog of items and / or devices for a given surgical procedure. If discontinuities exist between the lists, the surgical hub 5104 can be configured to provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, and / or other surgical articles are missing. In some examples, the surgical hub 5104 may be configured to determine the relative distance or relative position of a modular device 5102 and a patient monitoring device 5124, for example, by proximity sensors. The surgical hub 5104 can compare the relative positions of the devices to a recommended or predicted layout for a particular surgical procedure. If a discontinuity exists between the layouts, the surgical hub 5104 may be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout.

[0100] The situational awareness surgical hub 5104 can determine whether a surgeon (or other healthcare professional) is making an error or deviating from a set of expected actions during a surgical procedure. For example, the surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the surgical procedure with the steps or instruments expected for the type of surgical procedure that the surgical hub 5104 has determined is being performed. In some examples, the surgical hub 5104 can be configured to provide alerts indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgical procedure.

[0101] Surgical instruments (and other modular devices 5102) may be adjusted to suit the specific circumstances of each surgical procedure (such as adjusting to suit different tissue types) and their operation during the surgical procedure may be verified. The following steps, data, and display adjustments may be provided to the surgical instruments (and other modular devices 5102) in the operating room, according to the specific context of the procedure.

[0102] Figure 10 shows an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each step of the surgical procedure. Refer also to Figure 9 in the following description of the timeline 5200 shown in Figure 9. The timeline 5200 may illustrate the typical steps that nurses, surgeons, and other healthcare professionals might take during a lung segmentectomy, beginning with setting up the operating room and ending with transferring the patient to the postoperative recovery room. The contextually aware surgical hub 5104 may receive data from data source 5126 throughout the surgical procedure, including data generated each time healthcare professionals use the modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 receives this data from the paired modular device 5102 and other data sources 5126, and can continuously derive estimations (i.e., contextual information) about the procedure in progress as new data is received, such as which step of the procedure is being performed at any given time. The contextual awareness system of the surgical hub 5104 may, for example, record data about the procedure to generate a report, verify the steps being taken by the medical personnel, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions described herein.

[0103] As a first step 5202 in this exemplary procedure, hospital staff may retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic surgery. In a second step 5204, staff may scan incoming medical supplies for the procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies that may be used in various types of procedures to confirm that the combination of supplies matches a thoracic procedure. Furthermore, the surgical hub 5104 may also determine that the procedure is not a wedge resection (because the incoming supplies either do not include specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection). In a third step 5206, healthcare workers may scan a patient band via a scanner 5128 that is communicably connected to the surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data. In the fourth part of 5208, a medical professional turns on an assistive device. The assistive devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they may include a fume exhauster, an air inlet, and a medical imaging device. Once activated, the assistive device, which is a modular device 5102, can automatically pair with a surgical hub 5104, which may be located within a specific vicinity of the modular device 5102, as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 paired with it during this pre-operative or initialization phase. In this particular embodiment, the surgical hub 5104 may determine that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team.When the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing. In the fifth step 5210, personnel attach EKG electrodes and other patient monitoring devices 5124 to the patient. The EKG electrodes and other patient monitoring devices 5124 may be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, it can confirm that the patient is in the operating room, for example, as described in process 5207. In the sixth step 5212, medical personnel may also administer anesthesia to the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from the modular device 5102 and / or the patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 5212 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion begins.

[0104] In Section 7, 5214, the lungs of the patient being operated on may collapse (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer, for example, that the patient's lungs have collapsed from ventilator data. The surgical hub 5104 can compare the detection of lung collapse with the expected steps of the procedure (which can be accessed or read in advance) and thus infer that the surgical portion of the procedure has begun and that thereby causing lung collapse may be the first surgical step in this particular procedure. In Section 8, 5216, a medical imaging device 5108 (e.g., a scope) may be inserted and video footage from the medical imaging device may be initiated. The surgical hub 5104 can receive medical imaging device data (i.e., video or image data) through a connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has begun. Furthermore, the surgical hub 5104 can determine that a particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resections have not been taken into consideration by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Contextual information regarding the type of procedure being performed can be determined in various ways using data from the medical imaging device 124 (Figure 2), for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structures, by monitoring the number or type of medical imaging device being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used. For example, one technique for performing a VATS lobectomy may position the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while one technique for performing a VATS segmentectomy may position the camera in an anterior intercostal position relative to the segmental fissure. A situational awareness system can be trained, for example, using pattern recognition or machine learning techniques to recognize the position of a medical imaging device according to the visualization of the patient's anatomical structure.An exemplary technique for performing VATS lobectomy may utilize a single medical imaging device. An exemplary technique for performing VATS segmentectomy may utilize multiple cameras. One exemplary technique for performing VATS segmentectomy utilizes an infrared light source (which can be communicably connected to a surgical hub as part of a visualization system) to visualize the segmental fissure, but this is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0105] In Section 9, 5218, the surgical team may initiate the incision phase of the procedure. The surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out phase of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the above-described phase of the procedure) corresponds to the incision phase. In Section 10, 5220, the surgical team may proceed to the ligation phase of the procedure. The surgical hub 5104 may receive data from surgical stapling and cutting instruments indicating that an instrument is being emitted, and can therefore infer that the surgeon is ligating arteries and veins. As with the previous phase, the surgical hub 5104 can derive this inference by cross-referencing the received data from the surgical stapling and cutting instruments with the phase in the read-out process. In Section 11, 5222, the segmental resection phase of the procedure may be performed. The surgical hub 5104 can infer that a surgeon is transversely incising parenchymal tissue based on data from surgical stapling and cutting instruments (including data from their cartridges). The cartridge data may correspond, for example, to the size or type of staples being fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data may indicate the type of tissue being stapled and / or transversely incised. In this case, the type of staples being fired is used for parenchymal tissue (or other similar tissue types), thereby allowing the surgical hub 5104 to infer that the segmental resection portion of the procedure is being performed. Subsequently, in the 12th 5224, the nodule incision step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being fired, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision step, thereby allowing the surgical hub 5104 to make this inference.It should be noted that different instruments are better suited to specific tasks, so surgeons should periodically alternate between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasound) instruments depending on the specific stage of the procedure. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which stage of the procedure the surgeon is performing. Once the 12th step 5224 is completed, the incision can be closed and the postoperative portion of the procedure can begin.

[0106] In step 13, 5226, the patient may be awakened from anesthesia. The surgical hub 5104 may estimate that the patient is waking from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase). Finally, step 14, 5228, may be the step in which a medical professional removes various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 may estimate that the patient is being transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. As can be seen from this exemplary procedure description, the surgical hub 5104 can determine or estimate when each step of a given surgical procedure is occurring, according to the data received from various data sources 5126 that are communicably connected to the surgical hub 5104.

[0107] As shown in the first step 5202 of the timeline 5200 shown in Figure 10, in addition to estimating the type of surgical procedure to be performed using patient data from the EMR database(s), the patient data can also be used by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0108] Figure 11 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. In one aspect, the computer-implemented interactive surgical system may be configured to monitor and analyze data relating to the operation of various surgical systems, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented interactive surgical system may include a cloud-based analytics system. The cloud-based analytics system may be described as a surgical system, but is not necessarily limited to that, and may generally be a cloud-based medical system. As shown in Figure 11, the cloud-based analytics system may include a plurality of surgical instruments 7012 (which may be the same as or similar to instrument 112), a plurality of surgical hubs 7006 (which may be the same as or similar to hub 106), and a surgical data network 7001 (which may be the same as or similar to network 201) for connecting the surgical hubs 7006 to a cloud 7004 (which may be the same as or similar to cloud 204). Each of the plurality of surgical hubs 7006 may be communicably connected to one or more surgical instruments 7012. Hub 7006 may also be communicably connected to a cloud 7004 of a computer-implemented interactive surgical system via network 7001. Cloud 7004 may be a remote, centralized source of hardware and software for storing, manipulating, and communicating data generated based on the operation of various surgical systems. As shown in Figure 11, access to Cloud 7004 may be achieved via network 7001, which may be the Internet or another suitable computer network. The surgical hub 7006, which may be connected to Cloud 7004, can be considered the client side of a cloud computing system (i.e., a cloud-based analytics system). Surgical instruments 7012 may be paired with the surgical hub 7006 for controlling and performing various surgical procedures or actions described herein.

[0109] In addition, the surgical instrument 7012 may be equipped with transceivers for data transmission to and from the corresponding surgical hub 7006 (which may also be equipped with transceivers). The combination of the surgical instrument 7012 and the corresponding hub 7006 can indicate a specific location, such as an operating room within a medical facility (e.g., a hospital) for providing medical surgery. For example, the memory of the surgical hub 7006 can store location data. As shown in Figure 11, the cloud 7004 includes a central server 7013 (which may be the same as or similar to the remote server 7013), a hub application server 7002, a data analysis module 7034, and an input / output ("I / O") interface 7006. The central server 7013 of the cloud 7004 collectively manages the cloud computing system, which includes monitoring requests from the client module 7006 and managing the processing capacity of the cloud 7004 to perform those requests. Each central server 7013 may comprise one or more processors 7008 coupled to a suitable memory device 7010, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage devices. The memory device 7010 may contain machine-executable instructions that, when executed, cause the processor 7008 to run a data analysis module 7034 for cloud-based data analysis, operation, recommendation, and other operations described below. Furthermore, the processor 7008 can run the data analysis module 7034 independently or in conjunction with a hub application running independently by the hub 7006. The central server 7013 may also include a database 2212 of aggregated medical data, which may reside in memory 2210.

[0110] Based on connections to various surgical hubs 7006 via network 7001, the cloud 7004 can aggregate data from various surgical instruments 7012 and the specific data generated by their corresponding hubs 7006. Such aggregated data may be stored in the aggregated medical database 7012 of the cloud 7004. Specifically, the cloud 7004 can advantageously perform data analysis and operations on the aggregated data to gain insights and / or perform functions that individual hubs 7006 may not be able to achieve on their own. For this purpose, as shown in Figure 11, the cloud 7004 and the surgical hubs 7006 are connected in a communicative manner to send and receive information. The I / O interface 7006 is connected to multiple surgical hubs 7006 via network 7001. In this way, the I / O interface 7006 can be configured to transfer information between the surgical hubs 7006 and the aggregated medical data database 7011. Thus, the I / O interface 7006 can facilitate read / write operations of the cloud-based analysis system. Such read / write operations may be performed in response to requests from the hub 7006. These requests may be sent to the hub 7006 via the hub application. The I / O interface 7006 may include one or more high-speed data ports, including a Universal Serial Bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 7004 to the hub 7006. The hub application server 7002 of the cloud 7004 may be configured to host and supply shared functions to a software application (e.g., the hub application) running on the surgical hub 7006. For example, the hub application server 7002 may manage requests from the hub application through the hub 7006, control access to the aggregated medical data database 7011, and perform load balancing. The data analysis module 7034 will be described in more detail with reference to Figure 12.

[0111] The configurations of specific cloud computing systems described in this disclosure may be designed to address a variety of problems arising in the context of medical surgeries and procedures performed using medical devices such as surgical instruments 7012, 112, etc. In particular, surgical instrument 7012 may be a digital surgical device configured to interact with the cloud 7004 in order to implement techniques to improve surgical outcomes. Various surgical instruments 7012 and / or surgical hubs 7006 may include a touch-controlled user interface so that a clinician can control the manner of interaction between the surgical instruments 7012 and the cloud 7004. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0112] Figure 12 is a block diagram showing the functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. The cloud-based analytics system may include a number of data analytics modules 7034 that can be executed by a processor 7008 of the cloud 7004 to provide data analytics solutions to problems that arise particularly in the medical field. As shown in Figure 12, the functionality of the cloud-based data analytics modules 7034 may be supported via a hub application 7014 hosted by a hub application server 7002 that can be accessed on a surgical hub 7006. The cloud processor 7008 and the hub application 7014 may work together to execute the data analytics modules 7034. An application program interface (API) 7016 may define a set of protocols and routines corresponding to the hub application 7014. In addition, the API 7016 may manage the storage and retrieval of data to a medical database 7012 aggregated for the operation of the application 7014. The cache 7018 may also store data (for example, temporarily) and may be linked to the API 7016 for more efficient retrieval of data used by the application 7014. The data analysis module 7034 in Figure 12 may include modules for resource optimization 7020, data collection and aggregation 7022, authorization and security 7024, control program updates 7026, patient outcome analysis 7028, recommendations 7030, and data classification and prioritization 7032. Other suitable data analysis modules may also be implemented by the cloud 7004 in several embodiments. In one embodiment, the data analysis module may be used to provide specific recommendations based on the analysis of trends, outcomes, and other data.

[0113] For example, the data collection and aggregation module 7022 may be used to generate self-describing data (e.g., metadata), including the identification of prominent features or configurations (e.g., trends), the management of redundant datasets, and the storage of data into paired datasets that can be grouped by surgery but do not necessarily correspond to actual surgical dates and surgeons. In particular, paired datasets generated from the operation of surgical instruments 7012 may include applying a binary classification, such as bleeding or non-bleeding events. More generally, the binary classification may be characterized as either a desirable event (e.g., a successful surgical procedure) or an undesirable event (e.g., a surgical instrument 7012 misfired or misused). The aggregated self-describing data may correspond to individual data received from various groups or subgroups of the surgical hub 7006. Thus, the data collection and aggregation module 7022 can generate aggregated metadata or other organized data based on the raw data received from the surgical hub 7006. For this purpose, the processor 7008 can be operationally linked to the hub application 7014 and the aggregated medical data database 7011 in order to execute the data analysis module 7034. The data acquisition and aggregation module 7022 may store the aggregated and organized data in the aggregated medical data database 2212.

[0114] The resource optimization module 7020 can be configured to analyze this aggregated data to determine the optimal use of resources for a particular healthcare facility or group of healthcare facilities. For example, the resource optimization module 7020 can determine the optimal reorder point for surgical staple fasteners 7012 for a group of healthcare facilities based on the corresponding predicted demand for those staple fasteners 7012. The resource optimization module 7020 can also evaluate the resource use or other operational configurations of various healthcare facilities to determine whether resource use can be improved. Similarly, the recommendation module 7030 can be configured to analyze the aggregated data from the data collection and aggregation module 7022 to provide recommendations. For example, the recommendation module 7030 can recommend to a healthcare facility (e.g., a healthcare service provider such as a hospital) that a particular surgical instrument 7012 should be upgraded to an improved version based, for example, on the error rate being higher than predicted. In addition, the recommendation module 7030 and / or the resource optimization module 7020 can recommend better supply chain parameters, such as product reorder points, and provide suggestions for different surgical instruments 7012, their use, or procedural steps to improve surgical outcomes. Medical facilities can receive such recommendations via the corresponding surgical hub 7006. More specific recommendations regarding the parameters or configurations of various surgical instruments 7012 can also be provided. The hub 7006 and / or the surgical instrument 7012 may each have a display screen that shows data or recommendations provided by the cloud 7004.

[0115] The patient outcome analysis module 7028 can analyze surgical outcomes associated with the currently used operating parameters of the surgical instrument 7012. The patient outcome analysis module 7028 may also analyze and evaluate other potential operating parameters. In this regard, the recommendation module 7030 may use these other potential operating parameters to make recommendations based on the resulting better surgical outcomes, such as better sealing or less bleeding. For example, the suggestion module 7030 may send a suggestion to the surgical instrument 7006 regarding the use of a particular cartridge with the corresponding stapled surgical instrument 7012. Thus, the cloud-based analytics system may be configured to analyze large amounts of collected raw data while controlling for common variables and to provide centralized recommendations across multiple healthcare facilities (favorably determined based on aggregated data). For example, the cloud-based analytics system can analyze, evaluate, and / or aggregate things such as the type of medical procedure, the type of patient, the number of patients, and geographical similarities among healthcare providers using similar types of instruments in a way that a single healthcare facility cannot analyze independently. The control program update module 7026 can be configured to implement various surgical instrument 7012 recommendations when the corresponding control program is updated. For example, the patient outcome analysis module 7028 can identify correlations linking specific control parameters to successful (or unsuccessful) outcomes. Such correlations may be addressed when the updated control program is transmitted to the surgical instrument 7012 via the control program update module 7026. Updates to the instrument 7012, which may be transmitted via the corresponding hub 7006, may incorporate aggregated outcome data collected and analyzed by the data collection and aggregation module 7022 of the cloud 7004. In addition, the patient outcome analysis module 7028 and the recommendation module 7030 can identify improved ways of using the instrument 7012 based on the aggregated outcome data.

[0116] The cloud-based analytics system may include security features implemented by Cloud 7004. These security features may be managed by the authorization and security module 7024. Each surgical hub 7006 may have associated unique credentials, such as a username, password, and other preferred security credentials. These credentials may be stored in memory 7010 and associated with permitted cloud access levels. For example, based on providing accurate credentials, a surgical hub 7006 may be granted access to communicate with the cloud to a predetermined extent (e.g., to send or receive certain defined types of information). For this purpose, the aggregated medical data database 7011 of Cloud 7004 may include a database of certified credentials to verify the accuracy of the provided credentials. Different credentials may be associated with various levels of authorization for interaction with Cloud 7004, such as a predetermined access level for receiving data analysis generated by Cloud 7004. Furthermore, for security purposes, the cloud may maintain a database of hubs 7006, instruments 7012, and other devices, which may include a "blacklist" of prohibited devices. Specifically, surgical hubs 7006 listed on the blacklist may not be permitted to interact with the cloud, while surgical instruments 7012 listed on the blacklist may not have functional access to the corresponding hub 7006 and / or may be prevented from fully functioning when paired with the corresponding hub 7006. Additionally or alternatively, the cloud 7004 may flag instruments 7012 based on non-conformity or other specified criteria. In this way, counterfeit medical devices and the improper reuse of such devices across the entire cloud-based analysis system can be identified and addressed.

[0117] The surgical instrument 7012 may use a wireless transceiver to transmit a wireless signal that may represent, for example, authorization credentials for access to the corresponding hub 7006 and the cloud 7004. A wired transceiver can also be used to transmit a signal. Such authorization credentials can be stored in the respective memory devices of the surgical instrument 7012. The authorization and security module 7024 can determine whether the authorization credentials are accurate or forged. The authorization and security module 7024 may also dynamically generate authorization credentials for enhanced security. The credentials may also be encrypted, for example, by using hash-based encryption. Upon transmitting appropriate authorization, the surgical instrument 7012 may signal to the corresponding hub 7006 and ultimately the cloud 7004 to indicate that the instrument 7012 is ready to acquire and transmit medical data. In response, the cloud 7004 may transition to a state in which it is ready to receive medical data for storage in the aggregated medical data database 7011. This readiness for data transmission can be indicated, for example, by an optical indicator on the instrument 7012. Cloud 7004 can also send signals to surgical instruments 7012 to update their associated control programs. Cloud 7004 can send signals directed to specific categories of surgical instruments 7012 (e.g., electrosurgical instruments) to ensure that software updates for control programs are sent only to the appropriate surgical instruments 7012. Furthermore, Cloud 7004 may be used to implement system-wide solutions to address local or global issues based on selective data transmission and authorization credentials. For example, if a group of surgical instruments 7012 are identified as having a common manufacturing defect, Cloud 7004 may modify the authorization credentials corresponding to this group to implement operational lockout for that group.

[0118] A cloud-based analytics system can enable monitoring of multiple healthcare facilities (e.g., healthcare facilities such as hospitals) to determine improved practices and recommend changes accordingly (e.g., via the suggestion module 2030). Thus, the processor 7008 of Cloud 7004 can analyze data associated with individual healthcare facilities to identify facilities and aggregate that data with other data associated with other healthcare facilities. Groups may be defined, for example, based on similar operational practices or geographical location. In this way, Cloud 7004 can provide a wide range of analytics and recommendations to healthcare facility groups. The cloud-based analytics system can also be used for enhanced contextual awareness. For example, the processor 7008 may predictively model the effect of cost and effectiveness recommendations for a particular facility (compared to overall operations and / or various medical procedures). The cost and effectiveness associated with that particular facility can also be compared to the corresponding local areas of other facilities or any other equivalent facilities.

[0119] The data classification and prioritization module 7032 may prioritize and classify data based on severity (e.g., the severity, unexpectedness, or suspiciousness of the medical event associated with the data). This classification and prioritization can be used in conjunction with the functionality of other data analysis modules 7034 described herein to improve the cloud-based analysis and operations described herein. For example, the data classification and prioritization module 7032 can assign priorities to data analyses performed by the data collection and aggregation module 7022 and the patient outcome analysis module 7028. Different priority levels may result in specific responses from the cloud 7004 (corresponding to the level of urgency), such as increased priority for rapid response, special processing, exclusion from the aggregated medical data database 7011, or other preferred responses. Furthermore, if necessary, the cloud 7004 may send requests (e.g., push messages) via the hub application server for additional data from the corresponding surgical instrument 7012. Push messages may result in notifications displayed on the corresponding hub 7006 to request support or additional data. This push message may be needed when the cloud detects a significant anomaly or outlier and the cloud is unable to determine the cause of that anomaly. The central server 7013 can be programmed to trigger this push message in certain critical situations, such as when data is determined to differ from a predicted value by a predetermined threshold, or when security is suspected to be involved.

[0120] Further illustrative details regarding the various functions described are provided in the following descriptions. Each of the descriptions may utilize a cloud architecture, as shown in Figures 11 and 12, as an example of a hardware and software implementation.

[0121] Figure 13 shows a block diagram of a computer-implemented adaptive surgical system 9060 configured to adaptively generate update updates for control programs for modular devices 9050, according to at least one aspect of the present disclosure. In some examples, the surgical system may include a surgical hub 9000, a plurality of modular devices 9050 communicably connected to the surgical hub 9000, and an analysis system 9100 communicably connected to the surgical hub 9000. While a single surgical hub 9000 may be shown, it should be noted that the surgical system 9060 may include any number of surgical hubs 9000, which may be connected to form a network of surgical hubs 9000 communicably connected to an analysis system 9010. In some examples, the surgical hub 9000 may include a processor 9010 connected to memory 9020 for executing stored instructions, and a data relay interface 9030 through which data is transmitted to the analysis system 9100. In some examples, the surgical hub 9000 may further include a user interface 9090 having an input device 9092 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 9094 (e.g., a display screen) for providing output to the user. The output may include data from queries entered by the user, suggestions for products or mixtures of products for use in a given procedure, and / or instructions for actions to be performed before, during, or after a surgical procedure. The surgical hub 9000 may further include an interface 9040 for communicatively connecting modular devices 9050 to the surgical hub 9000. In one embodiment, the interface 9040 may include transceivers that can be communicatively connected to the modular devices 9050 via a wireless communication protocol. The modular devices 9050 may include, for example, surgical stapling and cutting instruments, electrosurgical instruments, ultrasound instruments, inhalers, ventilators, and display screens.In some examples, the surgical hub 9000 may be further communicated to one or more patient monitoring devices 9052, such as EKG monitors or BP monitors. In some examples, the surgical hub 9000 may be further communicated to one or more databases 9054, such as an EMR database of the medical facility where the surgical hub 9000 is located, or to an external computer system.

[0122] When the modular device 9050 is connected to the surgical hub 9000, the surgical hub 9000 can sense or receive perioperative data from the modular device 9050 and then associate the received perioperative data with surgical procedure outcome data. The perioperative data may indicate how the modular device 9050 was controlled during the course of the surgical procedure. The procedure outcome data includes data associated with the outcome from the surgical procedure (or the process thereof), which may include whether the surgical procedure (or the process thereof) had a positive or negative outcome. For example, outcome data may include whether the patient suffered a postoperative complication from a particular procedure, or whether there was leakage (e.g., bleeding or air leakage) at a particular staple line or incision line. The surgical hub 9000 can obtain surgical procedure outcome data by receiving data from an external source (e.g., from the EMR database 9054), by directly detecting the outcome (e.g., via one of the connected modular devices 9050), or by inferring the occurrence of the outcome through a situational awareness system. For example, data on postoperative complications can be retrieved from the EMR database 9054, and data on staple line or incision line leakage can be directly detected or inferred by the situational awareness system. Surgical procedure outcome data can be inferred by the situational awareness system from data received from various data sources, including the modular device 9050 itself, the patient monitoring device 9052, and the database 9054 to which the surgical hub 9000 is connected.

[0123] The surgical hub 9000 can transmit data and result data from associated modular devices 9050 to the analysis system 9100 for processing on the analysis system 9100. By transmitting both perioperative data indicating how the modular devices 9050 are controlled and procedure result data, the analysis system 9100 can correlate different modes of control of the modular devices 9050 with surgical outcomes for specific procedure types. In some examples, the analysis system 9100 may include a network of analysis servers 9070 configured to receive data from the surgical hub 9000. Each of the analysis servers 9070 may include memory and a processor linked to the memory that executes instructions stored therein to analyze the received data. In some examples, the analysis servers 9070 may be connected in a distributed computing architecture and / or utilize a cloud computing architecture. Next, based on this paired data, the analysis system 9100 can learn the optimal or preferred operating parameters for various types of modular devices 9050, generate adjustments to the control program for the modular devices 9050 in the field, and then send (or "push") updates to the control program for the modular devices 9050.

[0124] Further details regarding the computer-implemented interactive surgical system 9060, including the surgical hub 9000 and various modular devices 9050 that can be connected thereto, will be described in relation to Figures 5 and 6.

[0125] Figure 14 provides a surgical system 6500 according to the present disclosure, which may include a surgical instrument 6502, the surgical instrument 6502 being able to communicate with a console 6522 or a portable device 6526 via a wired or wireless connection through a local area network 6518 or a cloud network 6520. In various embodiments, the console 6522 and the portable device 6526 may be any suitable computing device. The surgical instrument 6502 may include a handle 6504, an adapter 6508, and a loading unit 6514. The adapter 6508 is releasably coupled to the handle 6504, and the loading unit 6514 is releasably coupled to the adapter 6508 so that the adapter 6508 transmits force from the drive shaft to the loading unit 6514. The adapter 6508 or loading unit 6514 may include force gauges (not explicitly shown) disposed therein for measuring the force exerted on the loading unit 6514. The loading unit 6514 may include an end effector 6530 including a first jaw 6532 and a second jaw 6534. The loading unit 6514 may be a multi-firing loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without the loading unit 6514 having to be removed from the surgical site to reload the loading unit 6514.

[0126] The first jaws 6532 and the second jaws 6534 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaws 6532 may be configured to include a replaceable multi-fired fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) which may be configured to fire at least one fastener multiple times, or which may be fired two or more times before being replaced. The second jaws 6534 may include an anvil that deforms the fasteners or otherwise secures them around the tissue as they are ejected from the multi-fired fastener cartridge.

[0127] The handle 6504 may include a motor connected to the drive shaft so as to affect the rotation of the drive shaft. The handle 6504 may include a control interface for selectively starting the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces, which can be operated by a clinician to start the motor.

[0128] The control interface of the handle 6504 communicates with the controller 6528 of the handle 6504 to selectively start the motor and influence the rotation of the drive shaft. The controller 6528 may be located within the handle 6504 and is configured to receive input from the control interface and adapter data from the adapter 6508 or loading unit data from the loading unit 6514. The controller 6528 may analyze the input from the control interface and the data received from the adapter 6508 and / or loading unit 6514 in order to selectively start the motor. The handle 6504 may also include a display that can be viewed by a clinician while the handle 6504 is in use. The display may be configured to show some of the adapter data or loading unit data before, during, or after firing the instrument 6502.

[0129] The adapter 6508 may include an adapter identification device 6510 disposed therein, and the loading unit 6514 may include a loading unit identification device 6516 disposed therein. The adapter identification device 6510 may communicate with the controller 6528, and the loading unit identification device 6516 may communicate with the controller 6528. It will be understood that the loading unit identification device 6516 may communicate with the adapter identification device 6510 which relays or passes communication from the loading unit identification device 6516 to the controller 6528.

[0130] The adapter 6508 may also include a plurality of sensors 6512 (one of which is illustrated) disposed around it to detect various states of the adapter 6508 or the environment (e.g., when the adapter 6508 is connected to a loading unit, when the adapter 6508 is connected to a handle, when the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside the adapter 6508, the number of times the adapter 6508 has fired, the peak force of the adapter 6508 during firing, the total amount of force applied to the adapter 6508, the peak recoil force of the adapter 6508, the number of pauses of the adapter 6508 during firing, etc.). The plurality of sensors 6512 can provide input to the adapter identification device 6510 in the form of data signals. The data signals from the plurality of sensors 6512 may be stored in the adapter identification device 6510 or used to update adapter data stored in the adapter identification device 6510. The data signals from the plurality of sensors 6512 may be analog or digital. Multiple sensors 6512 may include force gauges for measuring the force exerted on the loading unit 6514 during firing.

[0131] The handle 6504 and adapter 6508 may be configured to interconnect the adapter identification device 6510 and the loading unit identification device 6516 with the controller 6528 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., may include electrical contacts that engage with each other to transmit energy and signals between them). In addition, or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductively). It is also intended that the adapter identification device 6510 and the controller 6528 may be able to wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0132] The handle 6504 may include a transmitter 6506 configured to transmit instrument data from the controller 6528 to other components of the system 6500 (e.g., LAN 6518, cloud 6520, console 6522, or portable device 6526). The transmitter 6506 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 6500. For example, the controller 6528 may transmit instrument data to the console 6528 that includes the serial number of an attached adapter (e.g., adapter 6508) attached to the handle 6504, the serial number of a loading unit attached to the adapter (e.g., loading unit 6514), and the serial number of a multi-fire fastener cartridge (e.g., multi-fire fastener cartridge) loaded into the loading unit. The console 6522 may then send back data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively, to the controller 6528. The controller 6528 can display a message on the local device display, or send a message via the transmitter 6506 to the console 6522 or portable device 6526, which can then display the message on the display 6524 or the portable device screen, respectively.

[0133] Figure 15A shows an exemplary flow for determining the operating mode and operating in the determined mode. The computer-implemented interactive surgical system, and / or its components and / or subsystems, may be configured to be updated. Such updates may include the inclusion of features and benefits that were not available to the user before the update. These updates can be established by any method of hardware, firmware, and software updates suitable for introducing functionality to the user. For example, the computer-implemented interactive surgical system, and / or its components and / or subsystems, may be updated using interchangeable / swappable (e.g., hot-swappable) hardware components, flashable firmware devices, and updatable software systems.

[0134] An update may be conditional on any preferred criteria or set of criteria. For example, an update may be conditional on one or more hardware capabilities of the system, such as processing power, bandwidth, or resolution. For example, an update may be conditional on one or more software aspects, such as the purchase of a certain software code. For example, an update may be conditional on a purchased service tier. A service tier may represent one or / or a set of features that a user is entitled to use in connection with a computer-implemented interactive surgical system. A service tier may be determined by a license code, e-commerce server authentication interaction, hardware key, username / password combination, biometric authentication interaction, public / private key exchange interaction, etc.

[0135] In 10704, system / device parameters may be identified. System / device parameters may be any element or set of elements on which an update is conditional. For example, a computer-implemented interactive surgical system may detect a certain bandwidth of communication between a modular device and a surgical hub. For example, a computer-implemented interactive surgical system may detect an indicator representing the purchase of a particular service tier.

[0136] In 10708, the operating mode may be determined based on identified system / device parameters. This determination may be made by a process that maps system / device parameters to operating modes. The process may be a manual process and / or an automated process. The process may be the result of local and / or remote calculations. For example, a client / server interaction can be used to determine the operating mode based on identified system / device parameters. For example, local software and / or locally embedded firmware may be used to determine the operating mode based on identified system / device parameters. For example, a hardware key, such as a secure microprocessor, can be used to determine the operating mode based on identified system / device parameters.

[0137] In 10710, operation may proceed according to the determined operating mode. For example, the system or device may proceed to operate in the default operating mode. For example, the system or device may proceed to operate in an alternative operating mode. The operating mode may be indicated by control hardware, firmware, and / or software already present in the system or device. The operating mode may also be indicated by newly installed / updated control hardware, firmware, and / or software.

[0138] Figure 15B shows an exemplary functional block diagram for changing the operating mode. The upgradeable element 10714 may include an initialization component 10716. The initialization component 10716 may include any hardware, firmware, and / or software suitable for determining the operating mode. For example, the initialization component 10716 may be part of the startup procedure for a system or device. The initialization component 10716 may be involved in interactions to determine the operating mode of the upgradeable element 10714. For example, the initialization component 10716 may interact with, for example, a user 10730, an external resource 10732, and / or a local resource 10718. For example, the initialization component 10716 may receive a license key from user 10730 to determine the operating mode. The initialization component 10716 may query an external resource 10732, such as a server, using the serial number of the upgradeable device 10714 to determine the operating mode. For example, the initialization component 10716 may query the local resource 10718, such as a local query to determine the amount of available bandwidth and / or a local query for a hardware key to determine the operating mode.

[0139] An upgradeable element 10714 may include one or more operational components 10720, 10722, 10726, 10728 and an operational pointer 10724. The initialization component 10716 may instruct the operational pointer 10724 to direct the operation of the upgradeable element 10741 to the operational components 10720, 10722, 10726, 10728 corresponding to the determined operational mode. The initialization component 10716 may instruct the operational pointer 10724 to direct the operation of the upgradeable element to the default operational component 10720. For example, the default operational component 10720 may be selected on the condition that no other alternative operational modes have been determined. For example, the default operational component 10720 may be selected on the condition that the initialization component is faulty and / or has a faulty interaction. The initialization component 10716 may instruct the operation pointer 10724 to instruct the resident operation component 10722 to operate the upgradeable component 10714. For example, certain functions may reside in the upgradeable component 10714 but require activation to operate. The initialization component 10716 may instruct the operation pointer 10724 to instruct the upgradeable component 10714 to install a new operation component 10728 and / or a newly installed operation component 10726. For example, new software and / or firmware may be downloaded. The new software and / or firmware may include code that enables the functions represented by the selected operating mode. For example, a new hardware component can be installed to enable the selected operating mode.

[0140] Figures 16A-16D and 17A-17F show various embodiments of an example of a visualization system 2108 that may be incorporated into a surgical system. The visualization system 2108 may include an imaging control unit 2002 and a hand unit 2020. The imaging control unit 2002 may include one or more illumination sources, power supplies for one or more illumination sources, one or more types of data communication interfaces (including USB, Ethernet, or wireless interfaces 2004), and one or more video outputs 2006. The imaging control unit 2002 may further include interfaces such as a USB interface 2010 configured to transmit integrated video and image acquisition data to a USB-enabled device. The imaging control unit 2002 may also include one or more computing components, including, but not limited to, a processor unit, a temporary memory unit, a non-temporary memory unit, an image processing unit, a bus structure for forming data links among computing components, and any interface (e.g., input and / or output) devices necessary to receive information from the imaging control unit and transmit information to components not included in the imaging control unit. The non-temporary memory, when executed by the processor unit, may further include instructions that can perform any number of operations based on data that may be received from computing devices not included in the hand unit 2020 and / or the imaging control unit.

[0141] The illumination source may include a white light source 2012 and one or more laser light sources. The imaging control unit 2002 may include one or more optical and / or electrical interfaces for optical and / or electrical communication with the hand unit 2020. The one or more laser light sources may, in non-limiting examples, include any one or more of a red laser light source, a green laser light source, a blue laser light source, an infrared laser light source, and an ultraviolet laser light source. In some non-limiting examples, the red laser light source may provide illumination with a peak wavelength that may be in the range of 635 nm to 660 nm, including both values. Non-limiting examples of the red laser peak wavelength may include about 635 nm, about 640 nm, about 645 nm, about 650 nm, about 655 nm, about 660 nm, or any value or range of values ​​in between. In some non-limiting examples, the green laser light source may provide illumination with a peak wavelength that may be in the range of 520 nm to 532 nm, including both values. Non-limiting examples of green laser peak wavelengths may include approximately 520 nm, approximately 522 nm, approximately 524 nm, approximately 526 nm, approximately 528 nm, approximately 530 nm, approximately 532 nm, or any value or range of values ​​in between. In some non-limiting examples, a blue laser source may provide illumination with a peak wavelength that may be in the range of 405 nm to 445 nm, including values ​​at both ends. Non-limiting examples of blue laser peak wavelengths may include approximately 405 nm, approximately 410 nm, approximately 415 nm, approximately 420 nm, approximately 425 nm, approximately 430 nm, approximately 435 nm, approximately 440 nm, approximately 445 nm, or any value or range of values ​​in between. In some non-limiting examples, an infrared laser source may provide illumination with a peak wavelength that may be in the range of 750 nm to 3000 nm, including values ​​at both ends. Non-limiting examples of infrared laser peak wavelengths may include approximately 750 nm, approximately 1000 nm, approximately 1250 nm, approximately 1500 nm, approximately 1750 nm, approximately 2000 nm, approximately 2250 nm, approximately 2500 nm, approximately 2750 nm, 3000 nm, or any value or range of values ​​in between. In some non-limiting examples, an ultraviolet laser light source may provide illumination with a peak wavelength that may be in the range of 200 nm to 360 nm, including values ​​at both ends.Non-limiting examples of ultraviolet laser peak wavelengths may include approximately 200 nm, approximately 220 nm, approximately 240 nm, approximately 260 nm, approximately 280 nm, approximately 300 nm, approximately 320 nm, approximately 340 nm, approximately 360 nm, or any value or range of values ​​in between.

[0142] In one non-limiting embodiment, the hand unit 2020 may include a body 2021, a camera scope cable 2015 attached to the body 2021, and an elongated camera probe 2024. The body 2021 of the hand unit 2020 may include hand unit control buttons 2022, or other control units that enable a medical professional to use the hand unit 2020 to control the operation of the hand unit 2020 or other components of the imaging control unit 2002, for example, a light source. The camera scope cable 2015 may include one or more conductors and one or more optical fibers. The camera scope cable 2015 may be terminated at the camera head connector 2008 at its proximal end, where the camera head connector 2008 is configured to mate with one or more optical and / or electrical interfaces of the imaging control unit 2002. The conductor may supply power to the hand unit 2020, which includes the main body 2021 and the elongated camera probe 2024, and / or to any electrical components inside the hand unit 2020, which includes the main body 2021 and / or the elongated camera probe 2024. The conductor may also function to provide bidirectional data communication between any one or more components between the hand unit 2020 and the imaging control unit 2002. One or more optical fibers may conduct illumination from one or more illumination sources in the imaging control unit 2002 through the hand unit body 2021 to the distal end of the elongated camera probe 2024. In some non-limiting embodiments, one or more optical fibers may also conduct light that has been reflected or refracted from the surgical site to one or more optical sensors disposed within the elongated camera probe 2024, the hand unit body 2021, and / or the imaging control unit 2002.

[0143] Figure 16B (plan view) shows in more detail several embodiments of the hand unit 2020 of the visualization system 2108. The hand unit body 2021 may be made of plastic material. The hand unit control buttons 2022 or other control units may have a rubber overmolding to protect the control units while allowing them to be operated by the surgeon. The camera scope cable 2015 may have an optical fiber integrated with a conductor, and the camera scope cable 2015 may have a protective and flexible overcoat such as PVC. In some non-limiting examples, the camera scope cable 2015 may be about 10 feet long for ease of use during surgical procedures. The length of the camera scope cable 2015 may range from about 5 feet to about 15 feet. Non-limiting examples of the length of the camera scope cable 2015 may be about 5 feet, about 6 feet, about 7 feet, about 8 feet, about 9 feet, about 10 feet, about 11 feet, about 12 feet, about 13 feet, about 14 feet, about 15 feet, or any length or range of lengths in between. The elongated camera probe 2024 may be made from a rigid material such as stainless steel. In some embodiments, the elongated camera probe 2024 may be joined to the hand unit body 2021 via a rotatable collar 2026. The rotatable collar 2026 may allow the elongated camera probe 2024 to rotate relative to the hand unit body 2021. In some embodiments, the elongated camera probe 2024 may be terminated at its distal end with an epoxy-sealed plastic window 2028.

[0144] The side plan view of the hand unit shown in Figure 16C shows that the optical sensor or image sensor 2030 may be located at the distal end 2032a of the elongated camera probe or within the hand unit body 2032b. In some alternative embodiments, the optical sensor or image sensor 2030 may be located within the imaging control unit 2002 along with additional optical elements. Figure 16C further shows an example of the optical sensor 2030 comprising a CMOS image sensor 2034 located within a mount 2036 having a radius of approximately 4 mm. Figure 16D shows an embodiment of the CMOS image sensor 2034, showing the active area 2038 of the image sensor. Although the CMOS image sensor in Figure 16C is depicted as being located within a mount 2036 having a radius of approximately 4 mm, it can be recognized that such a sensor and mount combination may be of any useful size, located within the elongated camera probe 2024, the hand unit body 2021, or the image control unit 2002. Some non-limiting examples of such alternative mounts include a 5.5mm mount 2136a, a 4mm mount 2136b, a 2.7mm mount 2136c, and a 2mm mount 2136d. It can also be recognized that the image sensor may include a CCD image sensor. The CMOS sensor or CCD sensor may include an array of individual light-sensing elements (pixels).

[0145] Figures 17A to 17F show various embodiments of several examples of lighting sources and their control that may be incorporated into the visualization system 2108.

[0146] Figure 17A shows an embodiment of a laser illumination system having multiple laser beams emitting multiple wavelengths of electromagnetic energy. As can be seen from the figure, the illumination system 2700 may include a red laser beam 2720, a green laser beam 2730, and a blue laser beam 2740, all of which are optically coupled together via an optical fiber 2755. As can be seen from the figure, each of the laser beams may have a corresponding photosensing element or electromagnetic sensor 2725, 2735, 2745, respectively, for sensing the output of a particular laser beam or wavelength.

[0147] Further disclosures relating to the laser illumination system shown in Figure 17A for use in the surgical visualization system 2108 can be found in U.S. Patent Application Publication No. 2014 / 0268860, filed on 15 March 2014, entitled "CONTROLLING THE INTEGRAL LIGHT ENERGY OF A LASER PULSE" (published as U.S. Patent No. 9,777,913 on 3 October 2017), the contents of which are incorporated herein by reference in their entirety for all purposes.

[0148] Figure 17B shows the operating cycle of the sensor used for rolling readout mode. The X direction corresponds to time, and the diagonal lines 2202 will be understood to indicate the activity of the internal pointer that reads each frame of data, time for each row. The same pointer is involved in resetting each column of pixels for the next exposure period. The net integral times of each column 2219a-c are equivalent, but are time-shifted relative to each other due to the rolling reset and readout process. Therefore, for any scenario where adjacent frames are required to represent light of different configurations, the only option for ensuring consistency across each column is to pulse the light between readout cycles 2230a-c. More specifically, the maximum available period corresponds to the sum of the blanking time and the time during which optical black or optical blind (OB) columns (2218, 2220) are served at the beginning or end of a frame.

[0149] Figure 17B shows the operating cycle of the sensor used in rolling readout mode or during sensor readout 2200. Frame readout may be represented by the vertical line 2210, which marks the start of the frame readout. The readout period is represented by the diagonal or slanted line 2202. The sensor can be read out in column units, with the top of the downward slanted edge being the top row of the sensor 2212 and the bottom of the downward slanted edge being the bottom row of the sensor 2214. The time between the last column readout and the next readout cycle may be called the blanking time 2216a-d. It should be understood that the blanking time 2216a-d may be the same or different between successful readout cycles. Note that some of the sensor pixel rows may be covered with a light shield (e.g., a metal coating or a substantially black layer of any other kind of material). These covered pixel rows may be referred to as optical black rows 2218 and 2220. The optical black columns 2218 and 2220 can be used as inputs for the correction algorithm.

[0150] As shown in Figure 17B, these optical black rows 2218 and 2220 may be located above the top of the pixel array, at the bottom of the pixel array, or both above and below the pixel array. In some embodiments, it may be desirable to control the amount of electromagnetic radiation, e.g., light that is exposed to the pixels and thereby integrated or accumulated by the pixels. Photons will be understood to be the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into charge or electric current. In some embodiments, integration time (2219a-c) can be initiated by resetting the pixels using an electron shutter or rolling shutter. The light can then be integrated until the next readout stage. In some embodiments, the position of the electron shutter may be moved between two readout cycles 2202 to control pixel saturation with respect to a given amount of light. In some alternative embodiments without an electron shutter, the integration time 2219a-c of incident light may be initiated during the first readout cycle 2202 and terminated in the next readout cycle 2202, which also defines the start of the next integration. In some alternative embodiments, the amount of light accumulated by each pixel may be controlled by the time 2230a-d during which the light was pulsed during the blanking time 2216a-d. This allows all rows to see the same light originating from the same light pulse 2230a-c. In other words, each row begins its integration in a first dark environment 2231, which may be in the back row 2220 of the optical black of the readout frame (m) with the maximum light pulse width, and then ends its integration in a second dark environment 2232, which may be in the front row 2218 of the optical black of the next subsequent readout frame (m+1) with the maximum light pulse width. Thus, the image generated from the light pulses 2230a-c may be available only during the readout of frame (m+1) without interference with either frame (m) or (m+2).

[0151] It should be noted that having optical pulses 2230a to c that are read only within one frame and do not interfere with adjacent frames corresponds to having the emission of a given optical pulse 2230a to c during the blanking time 2216. Since the optical black rows 2218 and 2220 are insensitive to light, the frame time (m) of the rear optical black row 2220 and the frame time (m+1) of the front optical black row 2218 can be added to the blanking time 2216 to determine the maximum range of the emission time of the optical pulse 2230.

[0152] In some embodiments, Figure 17B shows an example of a timing diagram for continuous frame acquisition by a conventional CMOS sensor. Such a CMOS sensor may incorporate a Bayer pattern of color filters, as shown in Figure 17C. It is recognized that the Bayer pattern provides luminance detail higher than chromaticity. Furthermore, it may be further recognized that the sensor has reduced spatial resolution because a total of four adjacent pixels are required to generate color information for the aggregated spatial portion of the image. In an alternative approach, the color image may be constructed by rapidly stroving the visualization region at high speed with various light sources (either lasers or light-emitting diodes) having different central light wavelengths.

[0153] The optical stroving system may be under the control of the camera system and may include a specially designed CMOS sensor with high-speed readout. The main benefit is that the sensor can achieve the same spatial resolution with significantly fewer pixels compared to conventional Bayer cameras or three-sensor cameras. Thus, the physical space occupied by the pixel array can be reduced. The actual pulse periods (2230a~c) may vary within the repeating pattern, as shown in Figure 17B. This is useful, for example, to allocate more time to components that require greater light energy or components with weaker light sources. The data may simply be buffered as appropriate within the signal processing chain, as long as the average capture frame rate is an integer multiple of the required final system frame rate.

[0154] The equipment for reducing the CMOS sensor chip area to an acceptable level by combining all of these methods is particularly attractive for small-diameter (approximately 3-10 mm) endoscopes. Specifically, it enables endoscope designs in which the sensor is located within a spatially constrained distal end, thereby significantly reducing the complexity and cost of the optical component while providing high-definition images. The result of this approach is that data must be temporally fused from three separate snapshots to reconstruct each final full-color image. Any movement in the scene relative to the optical frame of reference of the endoscope will generally degrade the perceived resolution because the edges of the object appear to be in slightly different positions within each captured component. This disclosure describes means for mitigating this problem by taking advantage of the fact that spatial resolution is much more important to luminance information than to chromaticity information.

[0155] The basis of this approach is that, instead of emitting monochromatic light in each frame, a combination of three wavelengths is used to provide all the luminance information within a single image. Chromaticity information is derived from separate frames with repeating patterns, such as Y-Cb-Y-Cr (Figure 17D). While a clever selection of pulse ratios can provide pure luminance data, the same cannot be said for chromaticity.

[0156] In one embodiment, as shown in Figure 17D, the endoscope system 2300a may include a pixel array 2302a having uniform pixels, and the system 2300a may operate to receive Y (luminance pulse) 2304a, Cb (ChromaBlue) 2306a, and Cr (ChromaRed) 2308a pulses.

[0157] To complete a full-color image, two components of chromaticity must also be provided. However, the same algorithm applied to luminance cannot be directly applied to chromaticity images because, as reflected in the fact that some of the RGB coefficients are negative, they are encoded. The solution is to add a sufficiently large amount of luminance so that all of the final pulse energies are positive. As long as the color fusion process in the ISP recognizes the composition of the chromaticity frames, they can be decoded by subtracting an appropriate amount of luminance from adjacent frames. The pulse energy ratios are as follows: Y = 0.183·R + 0.614·G + 0.062·B Cb=λ Y-0.101 R-0.339 G+0.439 B Cr=6·Y+0.439·R-0.399·G-0.040·B λ≧0.399 / 0.614=0.552 δ≧0.339 / 0.614=0.650

[0158] It has been found that when the λ coefficient is equal to 0.552, both the red and green components cancel each other out exactly, in which case the Cb information can be provided with pure blue light. Similarly, setting δ = 0.650 cancels out the blue and green components of Cr, resulting in pure red. This particular example is shown in Figure 17E, which also shows λ and δ,

[0159]

number

[0160] In the Y-Cb-Y-Cr pulsed scheme, the image data is already in the YCbCr space after color fusion. Therefore, in this case, it means performing operations based on the previous luminance and chromaticity before converting to linear RGB in order to perform color correction.

[0161] The color fusion process is simpler than demosaicing required by Bayer patterns (see Figure 17C) because there is no spatial interpolation. Frame buffering is required to have all the necessary information available for each pixel. In one common embodiment, Y-Cb-Y-Cr pattern data may be pipelined to generate one full-color image for every two raw capture images. This is achieved by using each chromaticity sample twice. Figure 17F shows a specific example of a 120Hz frame capture rate providing a final image at 60Hz.

[0162] Further disclosures relating to the control of the laser components of the illumination system shown in Figures 17B–17F for use in the surgical visualization system 108 can be found in U.S. Patent Application Publication No. 2014 / 0160318, filed on July 26, 2013, entitled “YCBCR PULSED ILLUMINATION SCHEME IN A LIGHT DEFICIENT ENVIRONMENT,” published on December 6, 2016, as U.S. Patent No. 9,516,239, and U.S. Patent Application Publication No. 2014 / 0160319, filed on July 26, 2013, entitled “CONTINUOUS VIDEO IN A LIGHT DEFICIENT ENVIRONMENT,” published on August 22, 2017, entitled U.S. Patent No. 9,743,016, the contents of which are incorporated herein by reference in their entirety.

[0163] Imaging of subsurface blood vessels During surgical procedures, it may be necessary for the surgeon to manipulate tissue to achieve the desired medical outcome. The surgeon's actions are limited by what is visually observable at the surgical site. Therefore, the surgeon may, for example, fail to recognize the arrangement of vascular structures beneath the tissue being manipulated during the procedure.

[0164] Because surgeons cannot visualize the vascular structure beneath the surgical site, they risk accidentally severing one or more important blood vessels during surgery.

[0165] Therefore, it is desirable to have a surgical visualization system that can acquire imaging data of the surgical site for presentation to the surgeon, and whose presentation can include information related to the presence of vascular structures located beneath the surface of the surgical site.

[0166] Some aspects of the present disclosure further provide control circuits configured to control the illumination of a surgical site using one or more illumination sources, such as laser light sources, and to receive imaging data from one or more image sensors. In some aspects, the present disclosure provides a non-temporary computer-readable medium for storing computer-readable instructions that, when executed, cause a device to detect blood vessels in tissue and determine the depth below the surface of the tissue.

[0167] In some embodiments, the surgical imaging system may include a plurality of illumination sources, each configured to emit light having a specific central wavelength; an optical sensor configured to receive a portion of the light reflected from a tissue sample when illuminated by one or more of the plurality of illumination sources; and a computing system. The computing system may be configured to receive data from the optical sensor when the tissue sample is illuminated by each of the plurality of illumination sources, to determine the depth location of structures within the tissue sample based on the data received by the optical sensor when the tissue sample is illuminated by each of the plurality of illumination sources, and to calculate visualization data relating to the structures and their depth locations. In some embodiments, the visualization data may have a data format that can be used by a display system, and the structures may include one or more vascular tissues.

[0168] Vascular imaging using NIR spectroscopy In one embodiment, a surgical imaging system can image one or more tissues within a surgical site at different times and depths, including an independent color cascade of illumination sources including visible light and light outside the visible range. The surgical imaging system can further detect or calculate the properties of light reflected and / or refracted from the surgical site. The properties of the light can be used to provide a composite image of the tissues within the surgical site and to provide analysis of underlying tissues that are not directly visible on the surface of the surgical site. The surgical imaging system can determine the tissue depth position without requiring a separate measuring device.

[0169] In one embodiment, the characteristics of light reflected and / or refracted from the surgical site may be the amount of absorbance of light at one or more wavelengths. The various chemical components of individual tissues may result in specific light absorption patterns that are wavelength-dependent.

[0170] In one embodiment, the illumination source may include a red laser source and a near-infrared laser source, and one or more tissues to be imaged may include vascular tissue such as veins or arteries. In some embodiments, a red laser source (within the visible range) may be used to image a portion of the underlying vascular tissue based on spectroscopy in the visible-red range. In some non-limiting examples, the red laser source may provide illumination having a peak wavelength that may be in the range of 635 nm to 660 nm, including both extreme values. Non-limiting examples of the red laser peak wavelength may include about 635 nm, about 640 nm, about 645 nm, about 650 nm, about 655 nm, about 660 nm, or any value or range of values ​​in between. In some other embodiments, a near-infrared laser source may be used to image the underlying vascular tissue based on near-infrared spectroscopy. In some non-limiting examples, the near-infrared laser source may have a wavelength that may be in the range of 750 nm to 3000 nm, including both extreme values. Non-limiting examples of infrared laser peak wavelengths may include approximately 750 nm, 1000 nm, 1250 nm, 1500 nm, 1750 nm, 2000 nm, 2250 nm, 2500 nm, 2750 nm, 3000 nm, or any value or range of values ​​in between. It can be recognized that the underlying vascular tissue may be probed using a combination of red light spectroscopy and infrared spectroscopy. In some examples, vascular tissue can be probed using a red laser source with a peak wavelength of approximately 660 nm and a near-infrared laser source with a peak wavelength of approximately 750 nm or approximately 850 nm.

[0171] Near-infrared spectroscopy (NIRS) is a non-invasive technique that enables the determination of tissue oxygenation based on the spectral quantification of oxygenated and deoxygenated hemoglobin within tissues. In some embodiments, NIRS can be used to directly image vascular tissue based on the difference in illumination absorbance between vascular and non-vascular tissues. Alternatively, vascular tissue can be indirectly visualized based on the difference in illumination absorbance of blood flow in the tissue before and after the application of physiological interventions such as arterial and venous occlusion.

[0172] Instruments for near-infrared (NIR) spectroscopy may be similar to those for UV-visible and mid-infrared ranges. Such spectroscopic instruments may include an illumination source, a detector, and a dispersion element for selecting a specific near-infrared wavelength for illuminating a tissue sample. In some embodiments, the light source may include an incandescent or quartz halogen light source. In some embodiments, the detector may include a semiconductor (e.g., InGaAs) photodiode or photoarray. In some embodiments, the dispersion element may include a prism or, more generally, a diffraction grating. Fourier transform NIR instruments using an interferometer are also common, particularly for wavelengths above about 1000 nm. Depending on the sample, the spectrum can be measured in either reflection mode or transmission mode.

[0173] Figure 18 schematically shows an example of apparatus 2400 similar to those used in NIR spectroscopy for UV-visible and mid-infrared light ranges. The light source 2402 can emit illumination 2404 over a broad spectral range that can strike a dispersing element 2406 (such as a prism or diffraction grating). The dispersing element 2406 may operate to select a narrow wavelength portion 2408 of the light emitted by the broad-spectrum light source 2402, and the selected portion 2408 of the light may illuminate the tissue 2410. The light reflected from the tissue 2412 may be directed to a detector 2416 (e.g., by a dichroic mirror 2414) so ​​that the intensity of the reflected light 2412 can be recorded. The wavelength of the light illuminating the tissue 2410 may be selected by the dispersing element 2406. In some embodiments, the tissue 2410 may be illuminated only by a single narrow wavelength portion 2408 selected by the dispersing element 2406 to form the light source 2402. In other embodiments, the tissue 2410 may be scanned over various narrow wavelength segments 2408 selected by the dispersive element 2406. In this way, the spectroscopic analysis of the tissue 2410 may be obtained over a range of NIR wavelengths.

[0174] Figure 19 shows a schematic example of instrumentation 2430 for measuring NIRS based on Fourier transform infrared imaging. In Figure 19, a laser source 2432 emitting light 2434 in the near-infrared range illuminates a tissue sample 2440. Light 2436 reflected by the tissue 2440 is reflected to a beam splitter 2446 by a mirror such as a dichroic mirror 2444. The beam splitter 2446 directs a portion 2448 of the light reflected by the tissue 2440 to a stationary mirror 2450 and a portion 2452 of the light 2436 reflected by the tissue 2440 to a moving mirror 2454. The moving mirror 2454 can vibrate in a fixed position based on a fixed piezoelectric transducer actuated by a sinusoidal voltage having a voltage frequency. The position of the moving mirror 2454 in space corresponds to the frequency of the sinusoidal operating voltage of the piezoelectric transducer. Light reflected from the moving mirror and the stationary mirror may be recombined (2458) by the beam splitter 2446 and directed to the detector 2456. The computational components can receive the signal output of the detector 2456 and perform the Fourier transform (time) of the received signal. Since the wavelength of the light received from the moving mirror 2454 changes over time relative to the wavelength of the light received from the stationary mirror 2450, the time-based Fourier transform of the recombined light corresponds to the wavelength-based Fourier transform of the recombined light 2458. In this way, the spectrum based on the wavelength of the light reflected from the tissue 2440 can be determined, and the spectral characteristics of the light 2436 reflected from the tissue 2440 can be obtained. Therefore, changes in the absorbance of the illumination in the spectral components from the light reflected from the tissue 2440 may indicate the presence or absence of tissue having specific light-absorbing properties (such as hemoglobin).

[0175] An alternative to near-infrared light for determining hemoglobin oxygenation is the use of monochromatic red light to determine the red light absorbance characteristics of hemoglobin. The red light absorbance characteristics of hemoglobin with a central wavelength of approximately 660 nm can indicate whether the hemoglobin is oxygenated (arterial blood) or deoxygenated (venous blood).

[0176] In some alternative surgical procedures, contrast agents can be used to improve the data collected for oxygenation and tissue oxygen consumption. In a non-limiting example, NIRS techniques may be used in combination with a bolus injection of a near-infrared contrast agent, such as indocyanine green (ICG), which has a peak absorbance at approximately 800 nm. ICG is used in some medical procedures to measure cerebral blood flow.

[0177] Vascular imaging using laser Doppler flow measurement In one embodiment, the characteristics of the light reflected and / or refracted from the surgical site may be the Doppler shift of the light wavelength from the illumination source.

[0178] Laser Doppler flow measurement can be used to visualize and characterize the flow of particles moving against an effectively stationary background. Therefore, laser light scattered by moving particles such as blood cells may have a different wavelength than the original illumination laser source. In contrast, laser light scattered by an effectively stationary background (e.g., vascular tissue) may have the same wavelength as the original illumination laser source. The change in wavelength of scattered light from blood cells can reflect both the direction of blood cell flow relative to the laser source and the blood cell velocity.

[0179] Figures 20A to 20C show the changes in the wavelength of light scattered from blood cells that can move away from the laser light source (Figure 20A) or towards the laser light source (Figure 20C).

[0180] In Figures 20A and 20C, the original illumination light 2502 with a relative central wavelength of 0 is shown. In Figure 20A, it can be observed that the light scattered from blood cells moving away from the laser source 2504 has a wavelength that is shifted to a wavelength that is some amount 2506 longer than the wavelength of the laser source (and therefore red-shifted). Also in Figure 20C, it can be observed that the light scattered from blood cells moving toward the laser source 2508 has a wavelength that is shifted to a wavelength that is some amount 2510 shorter than the wavelength of the laser source (and therefore blue-shifted). The amount of wavelength shift (e.g., 2506 or 2510) may depend on the velocity of the blood cells. In some embodiments, the amount of red shift (2506) for some blood cells may be approximately the same as the amount of blue shift (2510) for some other blood cells. Alternatively, the amount of red shift (2506) for some blood cells may differ from the amount of blue shift (2510) for some other blood cells. As shown in Figure 20A, the velocity of blood cells flowing away from the laser source may be lower than the velocity of blood cells flowing towards the laser source, as shown in Figure 26C, based on the relative magnitude of the wavelength shifts (2506 and 2510). In contrast, as shown in Figure 26B, light scattered from tissue not moving relative to the laser source (e.g., blood vessels 2512 or non-vascular tissue 2514) may not show any change in wavelength.

[0181] Figure 21 shows an embodiment of an instrument 2530 that may be used to detect the Doppler shift of laser light scattered from a portion of tissue 2540. Light 2534 emitted from laser 2532 may pass through beam splitter 2544. A portion of the laser light 2536 may be transmitted by beam splitter 2544 to illuminate tissue 2540. Another portion of the laser light may be reflected by beam splitter 2544 to collide with detector 2550 (2546). Backscattered light 2542 from tissue 2540 may be directed by beam splitter 2544 and may also collide with detector 2550. The combination of light 2534 emitted from laser 2532 and backscattered light 2542 from tissue 2540 may result in an interference pattern detected by detector 2550. The interference pattern received by detector 2550 may include interference fringes resulting from a combination of light 2534 emitted from laser 2532 and Doppler-shifted (and therefore wavelength-shifted) backscattered light 2452 from tissue 2540.

[0182] It can be recognized that the backscattered light 2542 from tissue 2540 may also include backscattered light from the boundary layer within tissue 2540 and / or wavelength-specific light absorption by the material within tissue 2540. As a result, the interference pattern observed by detector 2550 may incorporate interference fringe features from these additional optical effects, and therefore, unless properly analyzed, may result in calculations of Doppler shifts that are mixed in with them.

[0183] Figure 22 illustrates some of these additional optical effects. It is well known that light passing through a first optical medium having a first refractive index n1 can be reflected at the interface with a second optical medium having a second refractive index n2. Light transmitted through the second optical medium has a transmission angle to the interface that is different from the angle of incident light, based on the difference between refractive indices n1 and n2 (Snell's Law). Figure 22 illustrates the effect of Snell's Law on light striking the surface of a multicomponent tissue 2150, as may be presented in the field of surgery. The multicomponent tissue 2150 may consist of an outer tissue layer 2152 having a refractive index n1 and an implanting tissue such as a blood vessel having a blood vessel wall 2156. The blood vessel wall 2156 can be characterized by a refractive index n2. Blood can flow into the lumen of the blood vessel 2160. In some embodiments, it may be important during surgical procedures to determine the location of the blood vessel 2160 beneath the surface 2154 of the outer tissue layer 2152 and to characterize the blood flow using Doppler shift techniques.

[0184] The incident laser beam 2170a may be used to probe the blood vessel 2160 and may be directed onto the upper surface 2154 of the outer tissue layer 2152. A portion 2172 of the incident laser beam 2170a may be reflected at the upper surface 2154. Another portion 2170b of the incident laser beam 2170a may penetrate the outer tissue layer 2152. The reflected portion 2172 at the upper surface 2154 of the outer tissue layer 2152 has the same path length as the incident light 2170a and therefore has the same wavelength and phase as the incident light 2170a. However, the portion 2170b of light transmitted through the outer tissue layer 2152 has a different transmission angle than the incident angle of light hitting the tissue surface because the outer tissue layer 2152 has a refractive index n1 that is different from that of air.

[0185] If some of the light transmitted through the outer tissue layer 2152 collides with, for example, the second tissue surface 2158 of the blood vessel wall 2156, then portions 2174a and 2174b of the light are reflected back toward the light source 2170a of the incident light. Thus, the light 2174a reflected at the interface between the outer tissue layer 2152 and the blood vessel wall 2156 has the same wavelength as the incident light 2170a, but is phase-shifted due to the change in optical path length. Projecting the reflected light 2174a and 2174b along with the incident light on the sensor from the interface between the outer tissue layer 2152 and the blood vessel wall 2156 generates an interference pattern based on the phase difference between the two light sources.

[0186] Furthermore, a portion of the incident light 2170c can penetrate the blood vessel wall 2156 and into the blood vessel lumen 2160. This portion of the incident light 2170c can interact with moving blood cells in the blood vessel lumen 2160 and, as described above, can be reflected back towards the light source of impacting light having a wavelength Doppler shifted according to the velocity of the blood cells (2176a-c). The Doppler-shifted light 2176a-c reflected from the moving blood cells can be projected onto the sensor along with the incident light, resulting in an interference pattern with fringes based on the wavelength difference between the two light sources.

[0187] Figure 22 shows the optical path 2178 of light colliding with red blood cells in the vascular lumen 2160, assuming no change in refractive index between the emitted light and the light reflected by the moving blood cells. In this example, only the Doppler shift of the reflected light wavelength can be detected. However, the light reflected by the blood cells (2176a-c) may incorporate a phase change due to variations in tissue refractive index, in addition to the wavelength change due to the Doppler effect.

[0188] Therefore, when a light sensor receives incident light, light reflected from one or more tissue interfaces (2172 and 2174a, b), and Doppler-shifted light from blood cells (2176a-c), it can be understood that the interference pattern generated on the light sensor may include effects due to changes in refractive index within the tissue (phase changes) as well as effects due to changes in refractive index within the tissue. As a result, if the effects of changes in refractive index within the sample are not compensated for, Doppler analysis of light reflected by the tissue sample may produce erroneous results.

[0189] Figure 23 shows an example of the effect of light 2250 impacting a tissue sample on Doppler analysis to determine the depth and location of the underlying blood vessel. When no intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be primarily due to changes in wavelength reflected from moving blood cells. As a result, the spectrum 2252 derived from the interference pattern may generally reflect only the Doppler shift of the blood cells. However, when intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be due to a combination of changes in wavelength reflected from moving blood cells and phase shifts due to the refractive index of the intervening tissue. The spectrum 2254 derived from such an interference pattern may result in a calculation of a Doppler shift with added confusion due to the additional phase changes of the reflected light. In some embodiments, if information regarding the properties (thickness and refractive index) of the intervening tissue is known, the resulting spectrum 2256 may be corrected to provide a more accurate calculation of the wavelength changes.

[0190] It is recognized that the tissue penetration depth of light depends on the wavelength of light used. Therefore, the wavelength of the laser source light may be selected to detect particle motion (such as blood cells) at a specific range of tissue depths.

[0191] Figures 24A to 24C schematically illustrate means for detecting moving particles such as blood cells at various tissue depths based on laser wavelength. As shown in Figure 24A, the laser source 2340 can direct the incident beam of laser light 2342 onto the surface 2344 of the surgical site. Blood vessels 2346 (veins or arteries, etc.) may be located within the tissue 2348 at a certain depth δ from the tissue surface. The penetration depth 2350 of the laser into the tissue 2348 may depend at least partially on the laser wavelength. Thus, laser light with a wavelength in the red range of approximately 635 nm to approximately 660 nm can penetrate tissue 2351a to a depth of approximately 1 mm. Laser light with a wavelength in the green range of approximately 520 nm to approximately 532 nm can penetrate tissue 2351b to a depth of approximately 2 to 3 mm. Laser light with a wavelength in the blue range of approximately 405 nm to approximately 445 nm can penetrate tissue 2351c to a depth of approximately 4 mm or more. In the examples shown in Figures 30A to 30C, blood vessel 2346 may be located at a depth δ of approximately 2-3 mm below the tissue surface. Red laser light does not penetrate to this depth and therefore does not detect blood cells flowing within this vessel. However, both green and blue laser light can penetrate to this depth. Therefore, scattered green and blue laser light from blood cells within blood vessel 2346 may exhibit a Doppler shift in wavelength.

[0192] Figure 24B illustrates how the Doppler shift 2355 at the wavelength of the reflected laser light can manifest. Synchrotron radiation (or laser source light 2342) impacting the tissue surface 2344 may have a central wavelength 2352. For example, light from a green laser may have a central wavelength 2352 in the range of approximately 520 nm to approximately 532 nm. The reflected green light may have a central wavelength 2354 that is shifted to a longer wavelength (red shift) if the light is reflected from particles such as red blood cells moving away from the detector. The difference between the central wavelength 2352 and the central wavelength 2354 of the emitted laser light includes the Doppler shift 2355.

[0193] As described above in relation to Figures 22 and 23, the reflected laser light from the structure within the tissue 2348 may also exhibit a phase shift of the reflected light due to changes in refractive index resulting from changes in the tissue structure or composition. The synchrotron radiation (or laser source light 2342) striking the tissue surface 2344 may have a first phase characteristic 2356. The reflected laser light may have a second phase characteristic 2358. It can be recognized that blue laser light 2351c, which can penetrate the tissue to a depth of about 4 mm or more, may encounter a wider variety of tissue structures than red laser light (about 1 mm, 2351a) or green laser light (about 2-3 mm, 2351b). Therefore, as shown in Figure 30C, the phase shift 2358 of the reflected blue laser light may be significant, at least due to the depth of penetration.

[0194] Figure 24D shows an embodiment in which tissue is continuously illuminated by red 2360a, green 2360b, and blue 2360c laser light. In some embodiments, tissue may be continuously probed by red 2360a, green 2360b, and blue 2360c laser illumination. In some alternative examples, as shown in Figures 17D to 17F, one or more combinations of red 2360a, green 2360b, and blue 2360c laser light may be used to illuminate tissue according to a defined illumination sequence. Figure 24D shows the effect of such illumination on CMOS imaging sensors 2362a to d over time. Thus, at a first time t.sub.1, CMOS sensor 2362a may be illuminated by the red 2360a laser. At a second time t.sub.2, CMOS sensor 2362b may be illuminated by the green 2360b laser. At the third time t.sub.3, the CMOS sensor 2362c can be illuminated by a blue 2360c laser. The illumination cycle can then be repeated, starting from the fourth time t.sub.4, at which point the CMOS sensor 2362d can again be illuminated by a red 2360a laser. It can be recognized that continuous illumination of tissue with laser illumination at different wavelengths may enable Doppler analysis at various tissue depths over time. While the surgical site may be illuminated using red 2360a, green 2360b, and blue 2360c laser sources, it can be recognized that other wavelengths outside the visible light range (such as the infrared or ultraviolet region) can be used to illuminate the surgical site for Doppler analysis.

[0195] Figure 25 shows an example of the use of Doppler imaging to detect the presence of non-observable blood vessels at the surgical site 2600. In Figure 25, the surgeon may wish to excise a tumor 2602 located within the right upper posterior lobe 2604 of the lung. Because the lung is highly vascularized, care must be taken to identify only these vessels associated with the tissue and to seal only those vessels without impairing blood flow to the unaffected portion of the lung. In Figure 25, the surgeon has identified the margin 2606 of the tumor 2604. The surgeon may then cut the initial incision area 2608 in the margin area 2606, and exposed blood vessels 2610 may be observed for cutting and sealing. A Doppler imaging detector 2620 may be used to locate and identify non-observable blood vessels 2612 within the incision area. The imaging system may receive data from the Doppler imaging detector 2620 for analysis and display of data acquired from the surgical site 2600. In some embodiments, the imaging system may include a display that shows the surgical site 2600, including a visible image of the surgical site 2600, along with an image overlay of hidden blood vessels 2612 on the image of the surgical site 2600.

[0196] In the scenario described above with respect to Figure 25, the surgeon wants to sever the blood vessels supplying oxygen and nutrients to the tumor while preserving blood vessels associated with non-cancerous tissue. Furthermore, the blood vessels may be located at different depths within or around the surgical site. Therefore, the surgeon must identify the location (depth) of the blood vessels and determine whether they are suitable for resection. Figure 26 shows one method for identifying deep blood vessels based on the Doppler shift of light from blood cells flowing through them. As mentioned above, red laser light has a penetration depth of approximately 1 mm, and green laser light has a penetration depth of approximately 2-3 mm. However, blood vessels with a subsurface depth of 4 mm or more are outside the penetration depths at these wavelengths. However, blue laser light can detect such blood vessels based on their blood flow.

[0197] Figure 26 shows the Doppler shift of laser light reflected from a blood vessel at a specific depth below the surgical site. This site can be illuminated by red, green, and blue laser light. The central wavelength 2630 of the illumination light may be normalized to the relative center 3631. If the blood vessel is at a depth of 4 mm or more below the surface of the surgical site, neither the red nor the green laser light will be reflected by the blood vessel. As a result, the central wavelength 2632 of the reflected red light and the central wavelength 2634 of the reflected green light are not significantly different from the central wavelength 2630 of the illumination red or green light, respectively. However, if this site is illuminated by a blue laser light, the central wavelength 2638 of the reflected blue light 2636 is different from the central wavelength 2630 of the illumination blue light. In some cases, the amplitude of the reflected blue light 2636 may also be significantly reduced from the amplitude of the illumination blue light. Therefore, the surgeon can determine the presence of deep blood vessels along with their approximate depth, thereby avoiding deep blood vessels during surface tissue incision.

[0198] Figures 27 and 28 schematically illustrate the use of laser sources with different central wavelengths (colors) to determine the approximate depth of blood vessels beneath the surface of a surgical site. Figure 27 shows a first surgical site 2650 having a surface 2654 and blood vessels 2656 located beneath the surface 2654. In one method, blood vessels 2656 can be identified based on the Doppler shift of light striking the blood cell flow 2658 within the blood vessels 2656. The surgical site 2650 may also be illuminated by light from multiple lasers 2670, 2676, 2682, each laser characterized by emitting light at one of several different central wavelengths. As described above, illumination by the red laser 2670 can penetrate the tissue only about 1 mm. Therefore, if the blood vessels 2656 are located at a depth of less than 1 mm 2672 beneath the surface 2654, the red laser illumination will be reflected (2674), and the Doppler shift of the reflected red illumination 2674 can be determined. Furthermore, as mentioned above, illumination by the green laser 2676 can penetrate the tissue to a depth of approximately 2-3 mm. If the blood vessel 2656 is located at a depth of 2-3 mm 2678 below the surface 2654, the green laser illumination will be reflected (2680), while the red laser illumination 2670 will not be reflected, and the Doppler shift of the reflected green illumination 2680 can be determined. However, as shown in Figure 27, the blood vessel 2656 is located at a depth of 2684 at a depth of approximately 4 mm below the surface 2654. Therefore, neither the red laser illumination 2670 nor the green laser illumination 2676 is reflected. Instead, only the blue laser illumination is reflected (2686), and the Doppler shift of the reflected blue illumination 2686 may be determined.

[0199] In contrast to vessel 2656 shown in Figure 27, vessel 2656' shown in Figure 28 is located close to the surface of the tissue at the surgical site. Vessel 2656' can also be distinguished from vessel 2656 in that it is shown to have a much thicker wall 2657. Thus, vessel 2656 may be an example of a vein, while vessel 2656' may be an example of an artery, since arterial walls are known to be thicker than venous walls. In some cases, arterial walls can be about 1.3 mm thick. As mentioned above, red laser illumination 2670' can penetrate tissue to a depth of about 1 mm 2672'. Therefore, even if vessel 2656' is exposed at the surgical site (see 2610 in Figure 25), red laser light 2674' reflected from the surface of vessel 2656' may not be able to visualize the blood flow 2658' within vessel 2656' under Doppler analysis due to the thickness of the vessel wall 2657. However, as mentioned above, the green laser light 2676' impacting the tissue surface can penetrate to a depth of approximately 2-3 mm 2678'. Furthermore, the blue laser light 2682' impacting the tissue surface can penetrate to a depth of approximately 4 mm 2684'. Therefore, the green laser light may be reflected from the blood cells 2658' flowing within the blood vessels 2656' (2680'), and the blue laser light may be reflected from the blood cells 2658' flowing within the blood vessels 2656' (2686'). As a result, Doppler analysis of the reflected green light 2680' and the reflected blue light 2686' can provide information about blood flow in near-surface blood vessels, particularly the approximate depth of the vessels.

[0200] As described above, the depth of blood vessels beneath the surgical site may be probed based on wavelength-dependent Doppler imaging. The amount of blood flow through such vessels may be determined by speckle contrast (interference) analysis. The Doppler shift may indicate moving particles relative to a stationary light source. As described above, the Doppler wavelength shift can be an indicator of the velocity of particle motion. Individual particles, such as blood cells, may not be observable separately. However, the velocity of each blood cell generates a proportional Doppler shift. Interference patterns can be generated by the combination of backscattered light from multiple blood cells, resulting from the difference in the Doppler shifts of the backscattered light from each of the blood cells. Interference patterns can be an indicator of the number density of blood cells in the visualization frame. Interference patterns are sometimes called speckle contrast. Speckle contrast analysis can be calculated using a full-frame 300×300 CMOS imaging array, and the speckle contrast may be directly related to the amount of moving particles (e.g., blood cells) interacting with the laser light over a given exposure period.

[0201] The CMOS image sensor may be coupled to a digital signal processor (DSP). Each pixel of the sensor may be multiplexed and digitized. The Doppler shift of light can be analyzed by observing the light source laser beam compared to the Doppler-shifted light. A larger Doppler shift and speckle may be associated with a greater number of blood cells in the blood vessels and their velocity.

[0202] Figure 29 shows an embodiment of a composite visual display 2800 that may be presented to a surgeon during a surgical procedure. The composite visual display 2800 may be constructed by overlaying a white light image 2830 of the surgical site using a Doppler analysis image 2850.

[0203] In some embodiments, the white light image 2830 can make the surgical site 2832, one or more surgical incisions 2834, and tissue 2836 easily visible within the surgical incision 2834. The white light image 2830 may be generated by illuminating the surgical site 2832 with a white light source 2838 (2840) and receiving the reflected white light 2842 with a photodetector. The surface of the surgical site may be illuminated using the white light source 2838, but in one embodiment, as described above with respect to Figures 17C to 17F, the surface of the surgical site can be visualized using an appropriate combination of red 2854, green 2856, and blue 2858 laser light.

[0204] In some embodiments, the Doppler analysis image 2850 may include vessel depth information (from speckle analysis) along with blood flow information 2852. As described above, vessel depth and blood flow velocity can be obtained by illuminating the surgical site with laser light of multiple wavelengths and determining vessel depth and blood flow based on the known penetration depth of light of a particular wavelength. Generally, the surgical site 2832 may be illuminated with light emitted by one or more lasers such as a red laser 2854, a green laser 2856, and a blue laser 2858. The CMOS detector 2872 may receive light reflected back from the surgical site 2832 and the surrounding tissue (2862, 2866, 2870). The Doppler analysis image 2850 may be constructed based on an analysis of multiple pixel data from the CMOS detector 2872 (2874).

[0205] In one embodiment, a red laser 2854 may emit red laser illumination 2860 onto the surgical site 2832, and the reflected light 2862 can expose the surface or the smallest subsurface structure. In another embodiment, a green laser 2856 may emit green laser illumination 2864 onto the surgical site 2832, and the reflected light 2866 can reveal deeper subsurface characteristics. In yet another embodiment, a blue laser 2858 may emit blue laser illumination 2868 onto the surgical site 2832, and the reflected light 2870 can reveal, for example, blood flow in deeper vascular structures. In addition, speckle contrast analysis can provide the surgeon with information on the amount and velocity of blood flow through deeper vascular structures.

[0206] Although not shown in Figure 29, it can be understood that the imaging system may also illuminate the surgical site with light outside the visible range. Such light may include infrared and ultraviolet light. In some embodiments, the infrared or ultraviolet light source may include a broadband light source (such as a tungsten light source, a tungsten halogen light source, or a deuterium light source). In some other embodiments, the infrared or ultraviolet light source may include a narrowband light source (such as an IR diode laser, a UV gas laser, or a dye laser).

[0207] Figure 30 is a flowchart 2900 of a method for determining the depth of surface features within a tissue sample. The image acquisition system may illuminate the tissue with a first ray having a first center frequency (2910) and receive a first reflected light from the tissue illuminated by the first ray (2912). The image acquisition system may then calculate a first Doppler shift based on the first ray and the first reflected light (2914). The image acquisition system may then illuminate the tissue with a second ray having a second center frequency (2916) and receive a second reflected light from the tissue illuminated by the second ray (2918). The image acquisition system may then calculate a second Doppler shift based on the second ray and the second reflected light (2920). The image acquisition system may then calculate the depth of the tissue features based at least partially on the first center wavelength, the first Doppler shift, the second center wavelength, and the second Doppler shift (2922). In some embodiments, tissue features may include the presence of migrating particles, such as blood cells moving within blood vessels, as well as the direction and velocity of the flow of these migrating particles. It can be understood that this method may be extended to include illumination of the tissue with any one or more additional rays. Furthermore, the system can compute an image that includes a combination of an image of the tissue surface and an image of structures located within the tissue.

[0208] In some embodiments, multiple visual displays may be used. For example, a 3D display may provide a composite image showing a combined white light (or a suitable combination of red, green, and blue laser light) and a laser Doppler image. Additional displays may provide only a white light display or only a combined white light display and a NIRS display to visualize only the tissue's blood oxygenation response. However, the NIRS display may not be required for each cycle that allows the tissue response to occur.

[0209] Characterization of surface structure using multispectral OCT During surgical procedures, surgeons may use “smart” surgical devices for manipulating tissue. Such devices can be considered “smart” in that they include automated functions that instruct, control, and / or modify the operation of device-based parameters associated with their use. Parameters may include the type and / or composition of the tissue being manipulated. If the type and / or composition of the tissue being manipulated is unknown, the operation of the smart device may be inappropriate for the tissue being manipulated. As a result, the tissue may be damaged, or the manipulation of the tissue may be invalid due to improper settings of the smart device.

[0210] Surgeons may manually attempt to change the parameters of smart devices through trial and error, resulting in inefficient and lengthy surgical procedures.

[0211] Therefore, it is desirable to have a surgical visualization system that can probe the tissue structures beneath the surgical site to determine their structural and compositional properties, and provide such data to smart surgical instruments used in surgical procedures.

[0212] Some aspects of the present disclosure further provide control circuits configured to control illumination of a surgical site using one or more illumination sources, such as laser light sources, and to receive imaging data from one or more image sensors. In some aspects, the present disclosure provides a non-temporary computer-readable medium for storing computer-readable instructions, which, when executed, cause a device to characterize subsurface structures at a surgical site and determine the depth of subsurface structures in tissue.

[0213] In some embodiments, a surgical imaging system may comprise a plurality of illumination sources, each configured to emit light having a specific central wavelength; an optical sensor configured to receive a portion of the light reflected from a tissue sample when illuminated by one or more of the plurality of illumination sources; and a computing system. The computing system may be configured to receive data from the optical sensor when the tissue sample is illuminated by each of the plurality of illumination sources, to calculate structural data relating to the properties of structures within the tissue sample based on the data received by the optical sensor when the tissue sample is illuminated by each of the illumination sources, and to transmit structural data relating to the properties of structures to be received by a smart surgical device. In some embodiments, the structural properties are surface properties or structural composition.

[0214] In one embodiment, the surgical system may include multiple laser light sources capable of receiving laser light reflected from tissue. The light reflected from the tissue can be used by the system to calculate the surface properties of components placed within the tissue. The properties of the components placed within the tissue may include metrics related to the composition and / or surface roughness of the components.

[0215] In one embodiment, the surgical system may transmit data relating to the composition of the components and / or metrics relating to the surface irregularities of the components to a second instrument used on tissue to modify the control parameters of the second instrument.

[0216] In some embodiments, the second device may be a forward energy device, and modifications to the control parameters may include clamp pressure, operating power level, operating frequency, and transducer signal amplitude.

[0217] As described above, blood vessels can be detected beneath the surface of the surgical site based on the Doppler shift of light reflected by blood cells moving within the blood vessels.

[0218] Laser Doppler flow measurement can be used to visualize and characterize the flow of moving particles relative to an effectively stationary background. Therefore, laser light scattered by moving particles such as blood cells may have a different wavelength than the original illumination laser source. In contrast, laser light scattered by an effectively stationary background (e.g., vascular tissue) may have the same wavelength as the original illumination laser source. The change in wavelength of scattered light from blood cells can reflect both the direction of blood cell flow relative to the laser source and the blood cell velocity. As previously disclosed, Figures 20A–20C show the change in wavelength of scattered light from blood cells moving away from the laser light source (Figure 20A) or toward the laser light source (Figure 20C).

[0219] In Figures 20A and 20C, the original illumination light 2502 with a relative central wavelength of 0 is shown. In Figure 20A, it can be observed that the light scattered from blood cells moving away from the laser source 2504 has a wavelength that is shifted to a wavelength that is some amount 2506 longer than the wavelength of the laser source (and is therefore red-shifted). Also in Figure 20C, it can be observed that the light scattered from blood cells moving toward the laser source 2508 has a wavelength that is shifted to a wavelength that is some amount 2510 shorter than the wavelength of the laser source (and is therefore blue-shifted). The amount of wavelength shift (e.g., 2506 or 2510) may depend on the velocity of the blood cells. In some embodiments, the amount of red shift (2506) for some blood cells may be approximately the same as the amount of blue shift (2510) for some other blood cells. Alternatively, the amount of red shift (2506) for some blood cells may differ from the amount of blue shift (2510) for some other blood cells. As shown in Figure 24A, the velocity of blood cells flowing away from the laser source may be lower than the velocity of blood cells flowing towards the laser source, as shown in Figure 20C, based on the relative magnitude of the wavelength shifts (2506 and 2510). In contrast, as shown in Figure 20B, light scattered from tissue not moving relative to the laser source (e.g., blood vessels 2512 or non-vascular tissue 2514) may not show any change in wavelength.

[0220] As previously disclosed, Figure 21 shows an embodiment of an instrument 2530 that may be used to detect the Doppler shift of laser light scattered from a portion of tissue 2540. Light 2534 emitted from laser 2532 may pass through beam splitter 2544. A portion of the laser light 2536 may be transmitted by beam splitter 2544 and illuminate tissue 2540. Another portion of the laser light may be reflected by beam splitter 2544 (2546) and strike detector 2550. Backscattered light 2542 from tissue 2540 may be directed by beam splitter 2544 and may also strike detector 2550. The combination of light 2534 emitted from laser 2532 and backscattered light 2542 from tissue 2540 may result in an interference pattern detected by detector 2550. The interference pattern received by detector 2550 may include interference fringes resulting from a combination of light 2534 emitted from laser 2532 and Doppler-shifted (and therefore wavelength-shifted) backscattered light 2452 from tissue 2540.

[0221] It can be recognized that the backscattered light 2542 from tissue 2540 may also include backscattered light from the boundary layer within tissue 2540 and / or wavelength-specific light absorption by the material within tissue 2540. As a result, the interference pattern observed by detector 2550 may incorporate interference fringe features from these additional optical effects, and therefore, unless properly analyzed, may result in calculations of Doppler shifts that are mixed in with them.

[0222] It can be recognized that light reflected from tissue may also include backscattered light from boundary layers within the tissue and / or wavelength-specific light absorption by materials within the tissue. As a result, the interference pattern observed by the detector may incorporate fringe features that, unless properly analyzed, could be misinterpreted in Doppler shift calculations.

[0223] As previously disclosed, Figure 22 illustrates some of these additional optical effects. It is well known that light passing through a first optical medium having a first refractive index n1 can be reflected at the interface with a second optical medium having a second refractive index n2. Light transmitted through the second optical medium has a transmission angle to the interface that is different from the angle of incident light, based on the difference between refractive indices n1 and n2 (Snell's Law). Figure 20 illustrates the effect of Snell's Law on light striking the surface of a multicomponent tissue 2150, as may be presented in the surgical field. The multicomponent tissue 2150 may consist of an outer tissue layer 2152 having a refractive index n1 and an implanting tissue such as a blood vessel having a blood vessel wall 2156. The blood vessel wall 2156 can be characterized by a refractive index n2. Blood can flow into the lumen of the blood vessel 2160. In some embodiments, it may be important during surgical procedures to determine the location of the blood vessel 2160 beneath the surface 2154 of the outer tissue layer 2152 and to characterize the blood flow using Doppler shift techniques.

[0224] The incident laser beam 2170a may be used to probe the blood vessel 2160 and may be directed onto the upper surface 2154 of the outer tissue layer 2152. A portion 2172 of the incident laser beam 2170a may be reflected at the upper surface 2154. Another portion 2170b of the incident laser beam 2170a may penetrate into the outer tissue layer 2152. The reflected portion 2172 at the upper surface 2154 of the outer tissue layer 2152 has the same path length as the incident light 2170a and therefore has the same wavelength and phase as the incident light 2170a. However, the portion 2170b of light transmitted into the outer tissue layer 2152 has a different transmission angle than the incident angle of light impacting the tissue surface because the outer tissue layer 2152 has a refractive index n1 that is different from that of air.

[0225] When some of the light transmitted through the outer tissue layer 2152 collides with, for example, the second tissue surface 2158 of the blood vessel wall 2156, portions 2174a and 2174b of the light are reflected back toward the light source 2170a of the incident light. Therefore, the light 2174a reflected at the interface between the outer tissue layer 2152 and the blood vessel wall 2156 has the same wavelength as the incident light 2170a, but is phase-shifted due to the change in optical path length. When the light 2174a and 2174b reflected from the interface between the outer tissue layer 2152 and the blood vessel wall 2156 is projected onto the sensor along with the incident light, an interference pattern based on the phase difference between the two light sources is generated.

[0226] Furthermore, a portion of the incident light 2170c can penetrate the vessel wall 2156 and into the vessel lumen 2160. This portion of the incident light 2170c can interact with moving blood cells in the vessel lumen 2160 and, as described above, can be reflected back towards the light source of impacting light having a wavelength Doppler shifted according to the velocity of the blood cells (2176a-c). The Doppler-shifted light 2176a-c reflected from the moving blood cells can be projected onto the sensor along with the incident light, resulting in an interference pattern with fringes based on the wavelength difference between the two light sources.

[0227] Figure 22 shows the optical path 2178 of light colliding with red blood cells in the vascular lumen 2160, assuming no change in refractive index between the emitted light and the light reflected by the moving blood cells. In this example, only the Doppler shift of the reflected light wavelength can be detected. However, the light reflected by the blood cells (2176a-c) may incorporate a phase change due to variations in tissue refractive index, in addition to the wavelength change due to the Doppler effect.

[0228] Therefore, when a light sensor receives incident light, light reflected from one or more tissue interfaces (2172 and 2174a, b), and Doppler-shifted light from blood cells (2176a-c), it can be understood that the interference pattern generated on the light sensor may include effects due to changes in refractive index within the tissue (phase changes) as well as effects due to changes in refractive index within the tissue. As a result, if the effects of changes in refractive index within the sample are not compensated for, the Doppler analysis of light reflected by the tissue sample may produce erroneous results.

[0229] As previously disclosed, Figure 23 shows an example of the effect of light 2250 impacting a tissue sample on Doppler analysis to determine the depth and location of underlying blood vessels. When no intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be primarily due to changes in wavelength reflected from migrating blood cells. As a result, the spectrum 2252 derived from the interference pattern may generally reflect only the Doppler shift of the blood cells. However, when intervening tissue is present between the blood vessel and the tissue surface, the interference pattern detected by the sensor may be due to a combination of changes in wavelength reflected from migrating blood cells and phase shifts due to the refractive index of the intervening tissue. The spectrum 2254 derived from such an interference pattern may result in a calculation of a Doppler shift with added leeway due to the additional phase changes of the reflected light. In some embodiments, if information regarding the properties (thickness and refractive index) of the intervening tissue is known, the resulting spectrum 2256 may be corrected to provide a more accurate calculation of the wavelength changes.

[0230] It can be recognized that the phase shift of reflected light from tissue, regardless of the Doppler effect, may provide additional information about the underlying tissue structure.

[0231] Surgical visualization systems using the imaging techniques disclosed herein can benefit from very high sampling rates and very high display frequencies. The sampling rate may be related to the ability of the underlying device performing the sampling. General-purpose computing systems with software may be related to a first range of achievable sampling rates. Pure hardware implementations (e.g., dedicated application integrated circuits, ASICs) may be related to a second range of achievable sampling rates. The second range associated with pure hardware implementations is generally higher (e.g., much higher) than the first range associated with general-purpose computing software implementations.

[0232] Surgical visualization systems using imaging techniques disclosed herein may benefit from adaptable and / or updatable imaging algorithms (e.g., transformation and imaging processes). General-purpose computing systems with software may be associated with a high degree of adaptability and / or updatableness. Pure hardware implementations (e.g., dedicated application-specific integrated circuits, ASICs) may generally be associated with a lower degree of adaptability and / or updatableness than general-purpose computing systems with software. This may be partly due to the general ease with which software can be adapted and / or updated (which may include compiling and loading different software and / or updating module components) compared to pure hardware implementations (where new hardware components are physically designed, built, added, and / or replaced).

[0233] Surgical visualization systems using the imaging techniques disclosed herein can benefit from solutions that balance the higher sampling rates associated with hardware-based implementations with the adaptability and / or updatable nature of software systems. Such surgical visualization systems may employ a mixture of hardware and software solutions. For example, a surgical visualization system can use various hardware implementation transformations with software selectors. A surgical visualization system may also employ a field-programmable gate array (FPGA). An FPGA may include a hardware device that may contain one or more logic elements. These logic elements may be composed of bitstreams to implement various functions. For example, logic elements may be configured to perform specific individual logic functions, or they may be configured to perform them in a specific order and interconnection. Once configured, an FPGA can perform its functions using hardware logic elements without further configuration. Also, once configured, an FPGA can be reconfigured with different bitstreams to implement different functions. Similarly, once reconfigured, an FPGA can perform these different functions using hardware logic elements.

[0234] Figure 31 shows an exemplary surgical visualization system 10000. The surgical visualization system 10000 may be used to analyze at least a portion of the surgical field. For example, the surgical visualization system 10000 may be used to analyze tissue 10002 within at least a portion of the surgical field. The surgical visualization system 10000 may include a field-programmable gate array (FPGA) 10004, a processor (e.g., a processor 10006 local to the FPGA 10004), memory 10008, a laser light illumination source 10010, an optical sensor 10012, a display 10014, and / or a processor 10016 remote from the FPGA. The surgical visualization system 10000 may include, for example, the components and functions described in relation to Figures 16A to 16D.

[0235] System 10000 may use FPGA 10004 to transform reflected laser light through frequency conversion, for example, to identify the Doppler shift of the light and determine moving particles. This transformed data may be displayed (for example, in real time). The data may be displayed, for example, as a graphic and / or metric 10020 representing the number of particles moving per second. System 10000 may include communication between a local processor 10006 to FPGA 10004 and a remote processor 10016 from FPGA 10004. For example, the remote processor 10016 from FPGA 10004 can aggregate the data (for example, data for multiple seconds). The system may also be able to display this aggregated data. For example, it may be displayed as a graphic and / or metric 10026 representing the movement trend. This graphic and / or metric 10026 may be overlaid on the real-time data. Such trend information can be used to identify occlusion, to identify the vascular sealing / clamping efficiency of instruments, to identify the vascular tree overview, and even to identify the magnitude of motion oscillations over time. FPGA10004 may be configured to be on-the-fly updateable, for example, updateable with different (e.g., more advanced) transformations. These updates may be obtained from a local or remote communication server. These updates may change the transformation analysis, for example, from refractive index (e.g., analysis of cellular irregularity) to blood flow, multiple simultaneous depth analysis, etc.

[0236] FPGA updates may include transformations that implement various imaging options for the user. These imaging options may include standard composite visible light, tissue refractive index, Doppler shift, motion artifact correction, improved dynamic range, improved local clarity, superresolution, NIR fluorescence, multispectral imaging, confocal laser microscopy, optical coherence tomography, Raman spectroscopy, photoacoustic imaging, or any combination thereof. The imaging options may include any of the options presented below. U.S. Patent Application No. 15 / 940,742, entitled “DUAL CMOS ARRAY IMAGING,” filed March 29, 2018; U.S. Patent Application No. 13 / 952,564, entitled “WIDE DYNAMIC RANGE USING MONOCHROMATIC SENSOR,” filed July 26, 2013; U.S. Patent Application No. 14 / 214,311, entitled “SUPER RESOLUTION AND COLOR MOTION ARTIFACT CORRECTION IN A PULSED COLOR IMAGING SYSTEM,” filed March 14, 2014; and U.S. Patent Application No. 13 / 952,550, entitled “CAMERA SYSTEM WITH MINIMAL AREA MONOLITHIC CMOS IMAGE SENSOR,” filed July 26, 2013. Each of these is incorporated herein by reference in its entirety. Doppler wavelength shift can be used, for example, to identify the number, size, speed, and / or direction of moving particles. Doppler wavelength shift may be used in conjunction with multiple laser wavelengths, for example, to correlate tissue depth and moving particles. Tissue refractive index can be used, for example, to identify irregularities or diversity in the surface and subsurface aspects of tissue. In surgical practice, it can be useful in identifying tumor margins, infections, damaged surface tissue, adhesions, changes in tissue composition, etc. NIR fluorescence may involve techniques in which a systemically injected drug is preferentially absorbed by the target tissue. When illuminated with light of the appropriate wavelength, the injected drug fluoresces and can be imaged through an NIR-compatible scope / camera.Hyperspectral imaging and / or multispectral imaging may involve illuminating and evaluating tissue across many wavelengths throughout the electromagnetic spectrum to provide real-time images. This can be used to distinguish between target tissues. It can also enable imaging depths of, for example, 0–10 mm. Confocal laser endomicroscopy (CLE) uses light to capture high-resolution, cellular-level images without penetrating the tissue. CLE can provide real-time pathological images of tissue. It is a technique that uses light to capture 3D images with micrometer resolution from within the tissue. Optical coherence tomography (OCT) may use NIR light. OCT can enable imaging of tissue at depths of, for example, 1–2 mm. Raman spectroscopy may involve measuring the photon shift caused by monochromatic laser illumination of tissue. Raman spectroscopy can be used to identify specific molecules. Photoacoustic imaging may involve exposing tissue to laser pulses such that some of the energy causes thermoelastic expansion and ultrasonic emission. The resulting ultrasound can be detected and analyzed to form an image.

[0237] These updates can be performed automatically based on user input or system compatibility checks. These real-time, aggregated, and updatable features of System 10000 may be selectively enabled based on any aspect of the system configuration, such as system capacity, power availability, free memory access, communication capacity, software level, tiered purchase level, etc.

[0238] The laser light illumination source 10010 may include any illumination source of laser light suitable for analyzing human tissue. The laser light illumination source 10010 may include devices such as a source laser emitter, as illustrated in Figures 17A to 17F. The laser light illumination source 10010 may use one or more wavelengths of laser light to illuminate the tissue 10002. For example, the laser light illumination source 10010 may use a combination of red-blue-green-ultraviolet 1-ultraviolet 2-infrared. For example, the above combination with a sampling and operating rate of 360-480 Hz allows each light source to have multiple frames at a combined frame rate of 60 Hz for the end user. Combinations of laser light wavelengths with independent light sources can increase the resolution from a single array and allow penetration to various depths.

[0239] The tissue 10002 may be, for example, human tissue within a portion of the surgical field. The laser light may be reflected from the tissue 10002, resulting in reflected laser light. The reflected laser light may be received by the optical sensor 10012. The optical sensor 10012 may be configured to receive reflected laser light from at least a portion of the surgical field. The optical sensor 10012 may be configured to receive laser light from the entire surgical field. The optical sensor may be configured to receive reflected laser light from a selectable portion of the surgical field. For example, a user such as a surgeon may instruct the optical sensor and the optical laser light source and / or laser light source to analyze a specific portion of the surgical field.

[0240] The optical sensor 10012 can be any device suitable for sensing reflected laser light and outputting corresponding information. For example, the optical sensor 10012 can detect one or more characteristics of the reflected laser light, such as amplitude, frequency, wavelength, Doppler shift, and / or other time-domain or frequency-domain qualities. The laser optical sensor 10012 may include, for example, devices such as the optical sensors disclosed in relation to Figures 16A to 16D.

[0241] The laser light sensor 10012 may include one or more sensor modules 10013. The sensor modules 10013 may be configured to measure a wide range of wavelengths. The sensor modules 10013 may be tuned and / or filtered to measure specific wavelengths, for example. The sensor modules 10013 may include, for example, individual sensors, a collection of sensors, a sensor array, a combination of sensor arrays, etc. For example, the sensor modules 10013 may include semiconductor components such as photodiodes, CMOS (complementary metal-oxide-semiconductor) image sensors, and CCD (charge-coupled device) image sensors.

[0242] The laser light sensor 10012 may include a dual CMOS array. FIG. 31B shows an exemplary laser light sensor 10030. The laser light sensor 10030 may include two sensor modules 10032, 10034. The sensor modules 10032, 10034 may be implemented as a dual parallel CMOS array. For example, the laser light sensor 10030 may be incorporated into the form factor of a surgical scope 10031 (e.g., a 7 mm diameter surgical scope) having two sensor modules 10032, 10034 (e.g., two parallel 4 mm sensors). The laser light sensor 10030 may be configured to enable a shift between imaging modes. The modes may include, for example, 3D stereoscopic imaging and 2D simultaneous imaging (e.g., visual imaging together with imaging for refractive analysis and / or Doppler analysis). The modes may include, for example, imaging with a narrower or wider visualization range. The modes may include, for example, imaging with a lower or higher resolution and / or artifact correction. The sensor modules 10032, 10034 may include different types of sensors. For example, the first sensor module 10032 may be a CMOS device. The second sensor module 10034 may be a different CMOS device. The differences in the CMOS devices may allow for a greater diversity of light collection capabilities. For example, different CMOS devices may allow for a wider light contrast and / or better light collection. For example, the first sensor array 10032 may have more pixel detectors than the second sensor array 10034. The surgical scope 10031 may include one or more light sources 10036, such as, for example, a laser light illumination source.

[0243] Figure 31C is a graphical representation of exemplary operation of a pixel array for multiple frames. A sensor module (e.g., a CMOS sensor module) may incorporate patterns and / or techniques for light sensing. Light sensing techniques associated with the operation of the sensor module may incorporate filtering. Light sensing techniques associated with the sensor module may incorporate stroving of a light source. Examples of these techniques may include, for example, the techniques disclosed herein in relation to Figures 17C and 17D. Using a stroving light source pattern associated with the sensor module, reflected light may be measured and information indicating the reflected light may be generated. The pixel array may be captured by rapidly stroving a visualization area at high speed with various light sources (either lasers or light-emitting diodes) having various central light wavelengths.

[0244] Strobing can cause the sensor to capture each pixel array associated with a corresponding wavelength. For example, in the first pattern 10038, wavelengths of light in red, green, and blue, as well as infrared (e.g., near-infrared), can be strobed. Such strobing can cause the sensor to capture, for example, a first pixel array 10040 associated with a red wavelength, a second pixel array 10042 associated with a green wavelength, a third pixel array 10044 associated with a blue wavelength, a fourth pixel array 4046 associated with a green wavelength, a fifth pixel array 10048 associated with an infrared (e.g., near-infrared) wavelength, a sixth pixel array 10050 associated with a green wavelength, and a seventh pixel array 10052 associated with a blue wavelength. For example, in the second pattern 10054, wavelengths of light in red, green, and blue, as well as infrared (e.g., near-infrared), can be strobed. Such strobing can cause the sensor to capture, for example, an eighth pixel array 10056 associated with a red wavelength, a ninth pixel array 10058 associated with a green wavelength, a tenth pixel array 10060 associated with a blue wavelength, an eleventh pixel array 10062 associated with a green wavelength, a twelfth pixel array 10064 associated with an ultraviolet wavelength, a thirteenth pixel array 10066 associated with a green wavelength, and a fourteenth pixel array 10068 associated with a blue wavelength.

[0245] For example, patterns such as the first pattern 10038 and the second pattern 10054 may be associated with one or more sensor modules. For example, patterns such as the first pattern 10038 and the second pattern 10054 may be associated with operating modes as disclosed herein. For example, patterns such as the first pattern 10038 and the second pattern 10054 may operate sequentially. For example, patterns such as the first pattern 10038 and the second pattern 10054 can be operated in parallel (for example, using appropriate blanking). For example, patterns such as the first pattern 10038 and the second pattern 10054 may each be associated with their respective corresponding sensor modules. For example, patterns such as the first pattern 10038 and the second pattern 10054 may be jointly associated with a sensor module.

[0246] As shown in Figure 31A, information collected by the optical sensor 10012 can be transmitted to FPGA 10004. FPGA 10004 may include an optional updatable gate array device suitable for analyzing data from the optical sensor 10012. FPGA 10004 may include one or more logic elements 10018. Logic elements 10018 may be configured to perform transformations on incoming information. FPGA 10004 may include outputs suitable for passing analyzed and / or processed data representing the organization from FPGA 10016 and / or display 10014 to a remote processor.

[0247] For example, logic element 10018 of FPGA 10004 may provide information that can be passed to display 10014 and displayed as real-time data or metric 10020 representing a transformation of reflected laser light information received by optical sensor 10012. The transformation may include any mathematical and / or logical operations to convert the data received from optical sensor 10012 into information indicating partial motion. For example, the transformation may include a fast Fourier transform (FFT).

[0248] The logic elements 10018 of the FPGA 10004 can, for example, provide real-time data or metrics 10020 directly to the display 10014 and / or in cooperation with a local processor 10006 for the field-programmable gate array. The real-time data and / or metrics 10020 may include, for example, a representation of particle movement such as the number of particles per second. The real-time data and / or metrics 10020 may be displayed on the display 10014. The real-time data and / or metrics 10020 may be displayed overlaid on a visualization of the organization 10002.

[0249] For example, logic elements 10018 of FPGA 10004 may provide information that can be passed from FPGA 10004 to a remote processor 10016 for aggregation and / or processing. The remote processor 10016 may provide aggregation and analysis of this data. For example, the remote processor 10016 may provide moving averages and other aggregation techniques. The remote processor 10016 may unpack time-aggregated data with variable time granularity. For example, the remote processor 10016 may aggregate several seconds' worth of data from the field-programmable gate array 10004. The remote processor 10016 may include other algorithms 10022 suitable for aggregating and analyzing the data, such as least-squares regression techniques, polynomial fitting techniques, and other statistical methods such as mean, arithmetic mean, mode, maximum, minimum, variance and / or similar. A remote processor 10016 of FPGA 10004 may include correlation algorithms that correlate data received from the optical sensor 10012 and / or data transformed by FPGA 10004 with other aspects of surgery, including other aggregated data such as situational awareness data, treatment status, medical information, patient outcomes, and adverse events such as bleeding events. The remote processor 10016 of FPGA 10004 may include specific artificial intelligence and / or machine learning-based algorithms. For example, previously acquired data may be used as a training set for one or more artificial intelligence and / or machine learning algorithms to provide further correlations between various surgical events and inputs received from the optical sensor 10012 and inputs transformed by FPGA 10004. Information obtained from the aggregation and analysis algorithms may be transmitted to a display 10014 for display to the user (e.g., transmitted in cooperation with a local processor 10006 to FPGA 10004).

[0250] The display 10014 may include any device suitable for displaying information to the user. The display 10014 may include, for example, the monitor 135 associated with Figure 3. For example, the display 10014 may include a conventional computer monitor. For example, the display 10014 may include any device suitable for displaying images and / or text data to the user. For example, the display 10014 may display image data 10024 received from the optical sensor 10012 and / or other image sensors to depict a visual representation of the organization 10002. The display 10014 may also be suitable for providing the user with contextual information including one or more displayed data elements. The data elements may include numerical or graphical representations of data and / or metrics. For example, a metric may include one or more numbers accompanied by a graphical representation of units. For example, the display 10014 may display a real-time metric 10020, such as the number of particles per second detected according to the output of the FPGA 10004. The display 10014 may show a processed metric 10026 (for example, the rate of change in the number of particles per second, measured over a period of time) from the aggregation and analysis algorithms of the processor 10016, which is located remotely from the FPGA 10006.

[0251] The processor 10006, which is locally included with FPGA 10004, can include any device suitable for control processing of the surgical visualization system 10000. For example, the processor 10006, which is local to FPGA, may include microprocessors, microcontrollers, FPGAs, and application-specific integrated circuits (ASICs), system-on-chip (SOICs), digital signal processing (DSP) platforms, real-time computing systems, and the like.

[0252] The processor 10006, local to FPGA 10004, may provide control operations for any of the lower-level components of the surgical visualization system 10000. For example, the processor 10006, local to FPGA 10004, may control the operation of the laser light illumination source 10010. The processor 10006, local to FPGA 10004, may, for example, provide timing for various laser light sequences. The processor 10006, local to FPGA 10004, may, for example, provide frequency and / or amplitude modulation of the laser light illumination source. The processor 10006, local to FPGA 10004, may, for example, instruct the laser light illumination source to illuminate in any of the techniques disclosed in Figures 17A to 17F.

[0253] A processor 10006 local to FPGA 10004 may be suitable for controlling the operation of the light sensor 10012. For example, the processor 10006 local to FPGA 10004 may instruct the light sensor 10012 to provide a specific shuttering sequence, for example, by turning a particular light sensor on or off at a specific time. The processor 10006 local to FPGA 10004 may instruct a particular configuration of the light sensor 10012, for example, local exposure, contrast, resolution, bandwidth, field of view, and imaging processing.

[0254] A processor 10006 local to FPGA 10004 may provide internal networking functions to direct data flow between components of the surgical visualization system. For example, the processor 10006 local to FPGA 10004 may direct data received from the optical sensor 10012 to FPGA 10004. The processor 10006 local to FPGA 10004 may provide and / or instruct a switching fabric to enable proper data transfer from the optical sensor 10012 to one or more logic elements 10018 of FPGA 10004.

[0255] A processor 10006 local to FPGA 10004 can control all or part of the operation of the display 10014. For example, a processor 10006 local to FPGA 10004 can provide instructions to display specific image data 10024, processed data and / or metrics 10026, and / or real-time data and / or metrics 10020 on the display 10014.

[0256] A processor 10006 local to FPGA 10004 may receive information from a user interface (not shown). For example, the processor 10006 local to FPGA 10004 may receive a specific selection of a region of interest on image data 10024. For illustrative purposes, if a surgeon is interested in the flow of particles in a particular region of the surgical field, the surgeon may use a user interface (e.g., keyboard and mouse) to select a region of interest on the display, and the processor 10006 local to FPGA 10004 will respond accordingly. For example, the above response may be performed by causing the surgical visualization system to determine and display one or more metrics associated with the selection made by the surgeon.

[0257] A processor 10006 local to FPGA 10004 and / or a processor 10016 remote from FPGA 10004 can operate individually or in cooperation to enable configuration changes to FPGA 10004. For example, FPGA 10004 may include a first array of logic elements for performing a first data transformation. FPGA 10004 may be configured to transition from the first array of logic elements to a second array of logic elements to perform a second data transformation. For example, a processor 10006 local to FPGA 10004 and / or a processor 10016 remote from FPGA 10004 may be preferred to adjust, reconfigure, and / or rearrange the arrangement or configuration of logic elements 10018 of FPGA 10004 so that logic elements 10018 perform a second transformation. The second transformation may differ from the first transformation. The second transformation may be a variation of the first transformation. To illustrate this feature, the first example transformation may include a 32-point Cooly-Tukey Radix-2 implemented Fast Fourier Transform (FFT) using 11-bit signed integer inputs, while the second transformation may include a 1024-point Cooly-Tukey Radix-2 implemented FFT using 12-bit signed integer inputs.

[0258] Data representing various configurations of logic elements 10028 that implement different transformations may be available to a surgical visualization system. For example, a processor 10016 located remotely from FPGA 10004 may store one or more configurations of logic elements 10028 in a database. These configurations 10028 may be updated from time to time. These configurations 10028 may represent various transformations. These configurations 10028 may represent transformations that require different levels of hardware and processing resources. For example, they may include transformations that can be implemented by less powerful FPGAs and / or transformations that can be implemented by more powerful FPGAs. The configuration information 10028 may include configurations for transformations associated with various procedures and / or tissues. For example, the configuration information 10028 may include newly developed transformations and / or transformations developed in accordance with the analysis of aggregated data over time. To illustrate this aspect, in one example, a particular transformation may be determined to be a better predictor of bleeding events in a particular surgical procedure, and such correlations may be used to further refine the transformation, and further to encourage its use when similar patient and / or procedure data indicate it.

[0259] The upgradeability of the transformations may be associated with the purchased functional hierarchy (e.g., the purchased software hierarchy). For example, the purchased functional hierarchy may enable FPGA10004 to be updatable and / or make specific transformations available to the surgical visualization system 10000. The purchased functional hierarchy may be associated, for example, with a hospital, operating room, surgeon, procedure, set of instruments, and / or specific instruments. For illustrative purposes, the surgical visualization system 10000 may be installed in a hospital for use with default transformations. Default transformations may include generalized transformations suitable for many procedures. If an upgraded functional hierarchy is purchased, FPGA10004 may be reconfigured to implement alternative transformations, which may be more customized, for example, to specific procedures, tissue types, or surgeon preferences.

[0260] Adaptive FPGA updates can enable variable overlays. Such overlays may include data and / or metrics from alternative source datasets. These datasets can be used to contextualize real-time particle movement and aggregated trend data. For example, environmental parameters may be controlled to affect blood flow and / or inflammation at a local surgical site. By monitoring fluid flow, a processor remote from the FPGA can recommend (or, for example, automatically change) room and / or patient settings. These setting changes can optimize the surgical site and / or improve device performance. For example, by monitoring blood flow, a user can receive visual feedback before actually performing an action (e.g., stapling and / or sealing) to understand the consequences. Settings such as rising or falling body temperature, raising / lowering bed angle, and compression cuff pressure and placement can be used, along with visual feedback, to direct blood toward or away from the area being monitored.

[0261] Memory 10008 may include any device suitable for storing and providing stored data. Memory may include read-only memory (ROM) and / or random-access memory (RAM). Memory 10008 may include, for example, electrically erasable programmable read-only memory (EEPROM). Memory 10008 may be suitable for, for example, an embedded system. Memory 10008 is suitable for, for example, storing any intermediate data products in the operation of a surgical visualization system. Memory 10008 may be suitable for storing configuration information of a surgical visualization system, including one or more command parameters and / or configuration information for the logic elements. Memory 10008 may be suitable for storing system parameters. Memory 10008 may be suitable for providing one or more buffers, registers, and / or temporary storage of information.

[0262] Figure 32 shows an exemplary method for determining the operating mode. In 10200, real-time data may be collected. For example, a surgical visualization system may collect real-time data associated with tissue. For example, a laser light illumination source may illuminate tissue and produce reflected laser light, which may be sensed by an optical sensor and transformed according to a transformation implemented in an array of logic elements within a field-programmable gate array (FPGA). This collected real-time data may be presented to the user. This collected real-time data may be processed and / or stored and / or aggregated by, for example, a processor local to the field-programmable gate array and / or a processor remote from the field-programmable gate array.

[0263] In 10202, control parameters and / or inputs may be considered for logical processing. For example, this consideration of control parameters and / or inputs may be used to determine whether the operation continues in the default operating mode and / or an alternative operating mode. For example, there may be a determination of system lockout status regarding local processing and tendencies based on system parameters.

[0264] User inputs and / or control parameters can include any number of parameters or any information suitable for helping to determine whether to operate in a default mode or an alternative mode. For example, data exchange with a locally arranged control system can be used as a control parameter. For example, a local control system that communicates bidirectionally with a remote system can be used. For example, the control parameter can include either a processing capability, a memory capability or a bandwidth. The control parameter can include the purchase of software layers. The input can include an input from a user such as a surgeon for selecting an alternative conversion rather than a default conversion. For example, the input can be, for example, a user input for selecting a part of the surgical field for a specific analysis. The control parameter and / or the input can include control parameters and inputs suitable for indicating the aggregation of data and / or enabling the analysis of the aggregated data.

[0265] The determination of whether to operate in the default mode or the alternative mode can include displaying the maximum capacity of the data to the user. The determination of whether to operate in the default mode or the alternative mode can include a notification and confirmation interaction with the user via the display and the user interface. According to the determination of whether to operate in the default mode or the alternative mode, the operation can continue in the default mode at 10204 or in the alternative mode at 10206. For example, the operation in the default operation mode can include collecting and processing real-time data according to the default conversion. Also, the operation in the alternative operation mode can include, for example, operating according to a conversion or a second conversion or an alternative conversion that replaces the collection of real-time data.

[0266] In a surgical visualization system with a light-generating and imaging sensor array, the detected light can be converted into information such as moving particle size, velocity, and volume. The results of the conversion can be displayed on a monitor. Default and / or alternative conversions may include various program parameters. The outputs from default and / or alternative conversions may be coupled to external processing to determine and aggregate data trends. Whether to operate in default or alternative operating modes may include selections for displaying particle data, trend data, hierarchical data, etc., and this selection may depend on system control parameters.

[0267] Figure 33 illustrates an exemplary method for displaying real-time and trend information to the user. The real-time conversion of the Doppler shift of the optical wavelength can be exported to a processor component. The processor component may be capable of storing a dataset of a certain number of preceding seconds. These datasets can be used as a reference for aggregating motion data into trend data. Alternatively, the trend data may be overlaid on the display to show both real-time motion and the trend of motion.

[0268] The migration trend may be compared to, for example, historical data (e.g., local historical data from the preceding few minutes and / or hours within the same procedure, longer-term historical data, etc.). The migration trend may also be compared to, for example, data from local sources and / or external sources. The comparison can provide context for the trend, e.g., the trend relative to a baseline. For example, the comparison may be made from the same patient at different points in time. For example, the comparison may be made from one or more similar patients (e.g., patients with similar relevant characteristics). The comparison may be used to inform surgeon decisions.

[0269] In 10300, real-time data can be collected. Laser light can be shone onto the tissue in the surgical field and reflected back towards the optical sensor. Real-time data may include data received by the optical sensor. Real-time data may include a representation of the frequency and / or wavelength of the reflected light.

[0270] Moving particles in the surgical field can cause a Doppler shift at the wavelength of reflected light. In 10302, real-time data can be transformed by a transformation to evaluate the Doppler shift. The resulting information can represent aspects of the moving particle, such as speed, velocity, and volume. This resulting information can be displayed to the user in 10304.

[0271] Furthermore, the maximum capacity of the data and / or system may be displayed to the user. Then, in 10306, the resulting information and / or real-time data may be aggregated and / or further analyzed. For example, it may be processed using contextual recognition. For example, this may enable the separation and / or identification of blood flow, interstitial fluid, smoke, particulate matter, fog, aerosols, etc. It may also enable the display of selected data without noise from other data types. For example, the user's selection of highlighted particle tracking may be further processed and analyzed to focus the display on desired real-time data, resulting information, etc. For example, the user may select the type of data to display, particle size, volume, growth rate, velocity of particle groups, and / or movement of tagged groups over time. The resulting information and / or real-time data may be aggregated and / or further analyzed to determine, for example, trends over time, conversion to various forms of rate of change (e.g., acceleration), and calibration and / or adjustment for temperature, blown gas type, laser source, combined laser data sets, etc. Aggregation and analysis may be performed simultaneously with the display of real-time information. The aggregation and analysis of information regarding moving particles may be performed at some point after the display of real-time data regarding moving particles. The aggregation and analysis of information regarding moving particles may be performed without the display of real-time information regarding moving particles. The aggregation and analysis of information regarding moving particles may include any number of algorithms suitable for analyzing the visualization data and the inventors' analysis.

[0272] In 10308, information obtained from aggregation and further analysis (e.g., trend information) can be displayed to the user. The trend information may be combined with a graphical trend animation. The trend information can be presented as a metric. The trend information can be overlaid on raw moving particle data.

[0273] Figure 34 depicts an exemplary user interface displaying real-time and / or trending information. The first user interface 10402 includes image data. This image data may represent an image portion 2810 of the surgical field. The image portion 2810 may present a magnified view of the vascular tree 2814 so that the surgeon can focus on dissecting only the target vessel 2815. To excise the target vessel 2815, the surgeon may use a smart RF cauterization device 2816. The image data may be generated by a CMOS image sensor and / or an optical sensor. The data may also be collected by broadband light and / or laser light that strikes this tissue, is received by the optical sensor, and processed in real time by a transformer. The output of this transformer may be a metric 10404 and / or other representation of the number of particles per second moving within a portion of the field of view. For example, this metric may represent particles such as smoke, liquid, blood cells, etc. The metric 10404 may be displayed on the first user interface 10402.

[0274] User interface element 10405 may be displayed to the user. For example, user interface element 10405 may include a text box indicating whether the surgeon wants to perform local and / or remote processing for further analysis of the data. Certain conditions may be required to be met in order to activate such processing. For example, activation may be conditional on the purchase of a software tier. For example, activation may be conditional on bandwidth and / or processing capacity.

[0275] With this operation in mind, trend information 10406 ​​may be displayed on a second user interface 10408. The second user interface 10408 may be displayed on a screen. For example, trend data may include a metric for particles per second and / or information graphics or other visualizations, such as charts, icons, graphs, etc.

[0276] For example, real-time metrics 10404, such as the number of particles per second, and trend information 10406, such as particle acceleration, may be included on a second user interface. These information elements may be displayed to the user. For example, real-time metrics 10404 and trend information 10406 ​​may be superimposed on image data. Such real-time metrics 10404, such as the number of particles per second, and / or trend information 10406, such as particle acceleration, may be useful to a surgeon performing resection of blood vessels 2815.

[0277] Figure 35 shows an exemplary upgrade framework for a surgical visualization system. The framework includes a 2x2 grid. The left axis represents the inputs. The lower access represents the transformations and / or algorithms. When performing an update to a surgical visualization system, the update may include changes to the inputs, such as changing the wavelength, pattern, or intensity of light. Changes to the inputs may include, for example, changing from a single wavelength to a multispectral input. When performing an update to a surgical visualization system, the update may include changes to the transformations and / or algorithms. Transformations may include re-tuning the transformations for processing efficiency, responsiveness, energy use, bandwidth, etc.

[0278] As shown in the diagram, updates can take the form of any box within the grid. Updates may include changes to the input while keeping the transformation and / or algorithm the same. Updates may include changes to the transformation and / or algorithm while keeping the input the same. Updates may include changes to the transformation and / or algorithm, and changes to the input.

[0279] Figure 36 shows an exemplary method for reconfiguring the FPGA. In 10602, a predetermined FPGA transformation may be used to transform the multispectral imaging data. In 10604, the information generated from this transformation can be aggregated and / or further analyzed. Aggregation and further analysis can identify alternative transformations that are more suitable for a particular purpose. In 10606, the system may request and / or receive inputs (e.g., control parameters) associated with updates to the transformation. If such inputs and / or control parameters indicate that no updates are needed, the system may continue to use the existing transformation in 10608. If the system is upgradeable, the system may acquire an alternative configuration in 10610. In 10612, the logic elements in the FPGA may be reconfigured according to the alternative configuration to reflect the updated transformation. The system may resume multispectral imaging using the updated FPGA transformation.

[0280] The following list of embodiments forms part of the description. 1. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is: A laser light illumination source configured to illuminate at least a portion of the surgical field with laser light, A light sensor configured to receive reflected laser light, A field-programmable gate array configured to convert information indicating reflected laser light into information indicating moving particles within at least a portion of the surgical field, A display configured to show an image including at least a portion of the surgical field, A processor configured to control a display to overlay and display a first metric and a second metric onto an image, The first metric represents the current state of the moving particle in at least a portion of the surgical field. The second metric is a surgical visualization system that represents the aggregation state of moving particles in at least a portion of the surgical field. 2. The surgical visualization system according to Embodiment 1, which includes data trends for the aggregation state of moving particles in at least a portion of the surgical field. 3. The surgical visualization system according to Embodiment 1, wherein the second metric represents aggregation over several seconds and is shown as a trend. 4. The surgical visualization system according to Embodiment 1, wherein the second metric is suitable for identifying any of the following: occlusion, instrument vascular sealing / clamping efficiency, vascular tree overview, and magnitude of motion vibration over time. 5. The surgical visualization system according to Embodiment 1, wherein the second metric includes the acceleration of moving particles in at least a portion of the surgical field. 6. The surgical visualization system according to Embodiment 1, wherein the processor is configured to control the display to show a graphically oriented animation superimposed on the image. 7. The surgical visualization system according to Embodiment 1, further comprising a user interface for receiving a user selection of the type of unit to be used to display a second metric. 8. The surgical visualization system according to Embodiment 1, further comprising a user interface for receiving user selection of tagged particle groups, wherein a second metric represents the aggregation state of the tagged particle groups. 9. The surgical visualization system according to Embodiment 1, comprising a first processor associated with an image acquisition module, the system further comprising a second processor associated with an external processing resource, information indicating moving particles within at least a portion of the surgical field is transmitted to the second processor for aggregation analysis, and information indicating a second metric is transmitted from the second processor to the first processor. 10. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is A field-programmable gate array configured to convert information indicating reflected laser light into information indicating moving particles within at least a portion of the surgical field, A first processor, housed together with a field-programmable gate array and configured to generate a first metric representing the current state of a moving particle in at least a portion of the surgical field, A surgical visualization system comprising: a second processor located remotely from a field-programmable gate array, configured to receive information indicating moving particles within at least a portion of the surgical field, and to generate a second metric representing the aggregation state of the moving particles within at least a portion of the surgical field. 11. The surgical visualization system according to Embodiment 10, further comprising a display, wherein a second processor transmits a second metric to a first processor, and the first processor instructs the display to display the first metric and the second metric superimposed on an image including at least a portion of the surgical field. 12. The surgical visualization system according to Embodiment 10, further comprising a laser light illumination source configured to illuminate at least a portion of the surgical field with laser light, and an optical sensor configured to receive reflected laser light. 13. The surgical visualization system according to Example 10, including data trends, of the aggregation state of moving particles in at least a portion of the surgical field. 14. The surgical visualization system according to Embodiment 10, wherein the second metric represents aggregation over several seconds and is shown as a trend. 15. The surgical visualization system according to Embodiment 10, wherein the second metric is suitable for identifying any of the following: occlusion, instrument vascular sealing / clamping efficiency, vascular tree overview, and magnitude of motion vibration over time. 16. The surgical visualization system according to Embodiment 10, wherein the second metric includes the acceleration of moving particles in at least a portion of the surgical field. 17. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is A display configured to show an image including at least a portion of the surgical field, A processor configured to control a display to overlay and display a first metric and a second metric onto an image, The first metric represents the current state of the moving particle in at least a portion of the surgical field. The second metric is a surgical visualization system that represents the aggregation state of moving particles in at least a portion of the surgical field. 18. The surgical visualization system according to Embodiment 17, which includes data trends for the aggregation state of moving particles in at least a portion of the surgical field. 19. The surgical visualization system according to Embodiment 17, wherein the second metric represents aggregation over several seconds and is shown as a trend. 20. The surgical visualization system according to claim 17, further comprising a user interface for receiving user selection of tagged particle groups, wherein a second metric represents the aggregation state of the tagged particle groups.

[0281] [Implementation Method] (1) A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is A laser light illumination source configured to illuminate at least a portion of the surgical field with laser light, A light sensor configured to receive reflected laser light, A field-programmable gate array configured to convert information representing the reflected laser light into information representing moving particles within at least a portion of the surgical field, A display configured to display an image including at least a portion of the surgical field, A processor configured to control the display to overlay and display a first metric and a second metric onto the image, The first metric represents the current state of the moving particle in at least a portion of the surgical field, The second metric represents the aggregation state of moving particles in at least a portion of the surgical field, in a surgical visualization system. (2) The surgical visualization system according to Embodiment 1, wherein the aggregation state of moving particles in at least a portion of the surgical field includes data trends. (3) The surgical visualization system according to Embodiment 1, wherein the second metric represents aggregation over several seconds and is shown as a trend. (4) The surgical visualization system according to Embodiment 1, wherein the second metric includes the acceleration of a moving particle in at least a portion of the surgical field. (5) The surgical visualization system according to any one of embodiments 1 to 4, wherein the second metric is suitable for identifying any of occlusion, the vascular sealing / clamping efficiency of the instrument, the vascular tree overview, and the magnitude of vibration of motion over time.

[0282] (6) The surgical visualization system according to any one of embodiments 1 to 5, wherein the processor is configured to control the display to show a graphical animation superimposed on the image. (7) A surgical visualization system according to any one of embodiments 1 to 6, further comprising a user interface for receiving a user selection of the type of unit to be used to display the second metric. (8) A surgical visualization system according to any one of embodiments 1 to 7, further comprising a user interface for receiving a user selection of tagged particle groups, wherein the second metric represents the aggregation state of the tagged particle groups. (9) A surgical visualization system according to any one of embodiments 1 to 8, wherein the processor comprises a first processor associated with an image acquisition module, the system further comprises a second processor associated with an external processing resource, the information indicating moving particles in at least a portion of the surgical field is transmitted to the second processor for aggregation analysis, and the information indicating a second metric is transmitted from the second processor to the first processor. (10) A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is A field-programmable gate array configured to convert information indicating reflected laser light into information indicating moving particles within at least a portion of the surgical field, A first processor, housed together with the field-programmable gate array and configured to generate a first metric representing the current state of the moving particles in at least a portion of the surgical field, A surgical visualization system comprising: a second processor located remotely from the field-programmable gate array, configured to receive the information indicating moving particles within at least a portion of the surgical field, and to generate a second metric representing the aggregation state of the moving particles within at least a portion of the surgical field.

[0283] (11) The surgical visualization system according to Embodiment 10, further comprising a display, wherein the second processor transmits the second metric to the first processor, and the first processor instructs the display to display the first metric and the second metric superimposed on an image including at least a portion of the surgical field. (12) The surgical visualization system according to embodiment 10 or 11, further comprising a laser light illumination source configured to illuminate at least a portion of the surgical field with laser light, and a light sensor configured to receive the reflected laser light. (13) The aggregation state of moving particles in at least a portion of the surgical field is a surgical visualization system according to any one of embodiments 10 to 12, including data trends. (14) The surgical visualization system according to any of embodiments 10 to 12, wherein the second metric represents aggregation over several seconds and is shown as a trend. (15) The surgical visualization system according to any one of embodiments 10 to 12, wherein the second metric includes the acceleration of a moving particle in at least a portion of the surgical field.

[0284] (16) The surgical visualization system according to any of embodiments 10 to 15, wherein the second metric is suitable for identifying any of occlusion, the vascular sealing / clamping efficiency of the instrument, the vascular tree overview, and the magnitude of vibration of motion over time. (17) A surgical visualization system for analyzing at least a portion of the surgical field, wherein the system is A display configured to display an image including at least a portion of the surgical field, A processor configured to control the display to overlay and display a first metric and a second metric onto the image, The first metric represents the current state of the moving particle in at least a portion of the surgical field, The second metric represents the aggregation state of moving particles in at least a portion of the surgical field, in a surgical visualization system. (18) The surgical visualization system according to Embodiment 17, wherein the aggregation state of moving particles in at least a portion of the surgical field includes data trends. (19) The surgical visualization system according to Embodiment 17, wherein the second metric represents aggregation over several seconds and is shown as a trend. (20) A surgical visualization system according to any one of embodiments 17 to 19, further comprising a user interface for receiving a user selection of tagged particle groups, wherein the second metric represents the aggregation state of the tagged particle groups.

[0285] (21) The system according to any one of embodiments 1 to 10 and 12 to 16, wherein the optical sensor provides the information indicating the reflected laser light. (22) The system according to any one of embodiments 1 to 16, wherein the first metric represents the information indicating a moving particle, provided by the field-programmable gate array. (23) The system according to any one of embodiments 1 to 22, wherein the information indicating the reflected laser light includes one or more of the following: amplitude, frequency, wavelength, Doppler shift, and / or other time-domain or frequency-domain qualities. (24) The system according to any one of Embodiments 1 to 23, wherein the information indicating the moving particles includes one or more of the number of moving particles per unit time, particle speed, particle velocity, and / or volume. (25) The system according to any one of embodiments 1 to 24, wherein the second metric is calculated by aggregating the information indicating the moving particle over time and performing other statistical methods such as least-squares regression, polynomial fitting, mean, arithmetic mean, mode, maximum, minimum, variance and / or similar, or by calculating a value representing acceleration.

Claims

1. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the surgical visualization system is A laser light illumination source configured to illuminate at least a portion of the surgical field with laser light, A light sensor configured to receive reflected laser light, A field-programmable gate array configured to convert information representing the reflected laser light into information representing moving particles within at least a portion of the surgical field, A display configured to display an image including at least a portion of the surgical field, It is a processor, It was decided to operate in the first mode, Based on the decision to operate in the first mode, the display is controlled to show the first metric. Receive input, Based on the input received, it is decided to operate in a second mode. Based on the decision to operate in the second mode, The system comprises a processor configured to control the display to overlay and display the first metric and the second metric onto the image, The first metric represents the current state, including the state of the moving particles in at least a portion of the surgical field, based on the current number of particles per second. A surgical visualization system in which the second metric represents the aggregation state, including the state based on the rate of change of the number of moving particles per second in at least a portion of the surgical field.

2. The surgical visualization system according to claim 1, wherein the aggregation state of moving particles in at least a portion of the surgical field includes data trends.

3. The surgical visualization system according to claim 1, wherein the second metric represents aggregation over several seconds and is shown as a trend.

4. The surgical visualization system according to claim 1, wherein the second metric includes the acceleration of moving particles in at least a portion of the surgical field.

5. The surgical visualization system according to any one of claims 1 to 4, wherein the processor is configured to control the display to show a graphical animation superimposed on the image.

6. The surgical visualization system according to any one of claims 1 to 5, further comprising a user interface for receiving a user selection of the type of unit to be used to display the second metric.

7. The surgical visualization system according to any one of claims 1 to 6, further comprising a user interface for receiving user selection of tagged particle groups, wherein the second metric represents the aggregation state of the tagged particle groups.

8. The surgical visualization system according to any one of claims 1 to 7, wherein the processor comprises a first processor associated with an image acquisition module, the surgical visualization system further comprises a second processor associated with an external processing resource, the information indicating moving particles in at least a portion of the surgical field is transmitted to the second processor for aggregation analysis, and the information indicating a second metric is transmitted from the second processor to the first processor.

9. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the surgical visualization system is A field-programmable gate array configured to convert information indicating reflected laser light into information indicating moving particles within at least a portion of the surgical field, A first processor, housed together with the field-programmable gate array, is configured to generate a first metric representing the current state, including a state based on the number of moving particles per second in at least a portion of the surgical field. A second processor, located remotely from the field-programmable gate array, is configured to receive the information indicating moving particles within at least a portion of the surgical field and to generate a second metric representing the aggregation state of the moving particles within at least a portion of the surgical field, including a state based on the rate of change of the number of particles per second. The second processor, It was decided to operate in the first mode, Based on the decision to operate in the first mode, the display is controlled to show the first metric on the image. Receive input, Based on the input received, it is decided to operate in a second mode. Based on the decision to operate in the second mode, The display is configured to control the display so as to overlay the first metric and the second metric onto the image. Surgical visualization system.

10. The surgical visualization system according to claim 9, further comprising a display, wherein the second processor transmits the second metric to the first processor, and the first processor instructs the display to display the first metric and the second metric superimposed on an image including at least a portion of the surgical field.

11. The surgical visualization system according to claim 9 or 10, further comprising: a laser light illumination source configured to illuminate at least a portion of the surgical field with laser light; and a light sensor configured to receive the reflected laser light.

12. The surgical visualization system according to any one of claims 9 to 11, wherein the aggregation state of moving particles in at least a portion of the surgical field includes data trends.

13. The surgical visualization system according to any one of claims 9 to 11, wherein the second metric represents aggregation over several seconds and is shown as a trend.

14. The surgical visualization system according to any one of claims 9 to 11, wherein the second metric includes the acceleration of moving particles in at least a portion of the surgical field.

15. A surgical visualization system for analyzing at least a portion of the surgical field, wherein the surgical visualization system is A display configured to display an image including at least a portion of the surgical field, It is a processor, It was decided to operate in the first mode, Based on the decision to operate in the first mode, the display is controlled to show the first metric. Receive input, Based on the input received, it is decided to operate in a second mode. Based on the decision to operate in the second mode, The system comprises a processor configured to control the display to overlay and display the first metric and the second metric onto the image, The first metric represents the current state, including a state based on the number of moving particles per second in at least a portion of the surgical field. A surgical visualization system in which the second metric represents the aggregation state, including the state based on the rate of change of the number of moving particles per second in at least a portion of the surgical field.

16. The surgical visualization system according to claim 15, wherein the aggregation state of moving particles in at least a portion of the surgical field includes data trends.

17. The surgical visualization system according to claim 15, wherein the second metric represents aggregation over several seconds and is shown as a trend.

18. The surgical visualization system according to any one of claims 15 to 17, further comprising a user interface for receiving user selection of tagged particle groups, wherein the second metric represents the aggregation state of the tagged particle groups.

19. The surgical visualization system according to any one of claims 1 to 8 and 11, wherein the optical sensor provides the information indicating the reflected laser light.

20. The surgical visualization system according to any one of claims 1 to 14, wherein the first metric represents the information indicating a moving particle, provided by the field-programmable gate array.

21. The surgical visualization system according to any one of claims 1 to 14, wherein the information indicating the reflected laser light includes one or more of the following: amplitude, frequency, wavelength, Doppler shift, and / or other time-domain or frequency-domain qualities.

22. The surgical visualization system according to any one of claims 1 to 14, wherein the information indicating moving particles includes one or more of the number of moving particles per unit time, particle speed, particle velocity, and / or volume.

23. The surgical visualization system according to any one of claims 1 to 14, wherein the second metric is calculated by aggregating the information indicating moving particles over time and performing other statistical methods such as least-squares regression, polynomial fitting, mean, arithmetic mean, mode, maximum, minimum, variance and / or similar, or by calculating a value representing acceleration.

Images