Ad-hoc synchronization of data from a multi-link adjustment sensing system.

Abstract

Examples herein may include a computer-implemented method for synchronizing data from a multi-link coordinated sensing system. The method may include receiving measurements at a first time. The measurements may be associated with the sensing system. The measurements may be associated with a communication interface. The method may include obtaining a latency value. The latency value may be associated with the surgical sensing system. The latency value may be associated with the communication interface. The method may include applying a time code to the received measurements. The time code may be applied based on the first time and the obtained latency value. The method may include sequencing the output based on the received measurements and the time code.

Description

Technical Field

[0005] ,

[0001] (Cross - reference to related applications) This application is related to the following applications filed simultaneously, the contents of each of which are incorporated herein by reference. · US Patent Application titled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS", Attorney Docket No. END9290USNP1.

Background Art

[0002] Modern surgical environments may include systems (e.g., sensing systems) that sense and / or monitor aspects of a patient's surgery. The systems may capture, for example, surgical - related information such as biomarkers, surgical tool parameters, etc.

[0003] These sensing systems may operate with a certain level of independence. For example, a surgical environment may include multiple independent sensing systems each providing its own independent data stream.

[0004] The technical task of collecting and / or using multiple independent data streams is a difficult task. The independent nature of the data streams can complicate their integration and / or combined use. The amount of data and processing can overwhelm systems in a surgical environment. Such problems can prevent medical professionals from being able to appropriately view, interpret, and ultimately act based on this surgical - related information. ​​​​​​​​One embodiment of the present invention provides a device for ordering received measurements, comprising a processor. The processor may be configured to receive measurements at a first time. The measurements may be associated with a surgical sensing system and a communication interface. The processor may be further configured to acquire latency values ​​associated with the surgical sensing system and the communication interface. The processor may be further configured to apply time codes to the received measurements based on the first time and the acquired latency values. The processor may be configured to order outputs based on the received measurements and time codes.

[0006] One embodiment of the present invention provides a method for ordering measurements received by a device. The method may include receiving measurements at a first time. The measurements may be associated with a surgical sensing system and a communication interface. The method may further include obtaining latency values ​​associated with the surgical sensing system and the communication interface. The method may further include applying a time code to the received measurements based on the first time and the obtained latency values. The method may further include ordering outputs based on the received measurements and the time code.

[0007] One embodiment of the present invention provides a device for ordering two or more received measurements, comprising a processor. The processor may be configured to receive a first measurement at a first time. The first measurement may be associated with a first surgical sensing system and a communication interface. The processor may be further configured to receive a second measurement at a second time. The second measurement may be associated with a second surgical sensing system and a communication interface. The processor may be further configured to acquire a first latency value and a second latency value associated with the first surgical sensing system and the second surgical sensing system, as well as the communication interface. The processor may be further configured to apply a first time code and a second time code to the first measurement and the second measurement based on the first time and the second time, as well as the first latency value and the second latency value. The processor may be further configured to order outputs based on the first measurement and the second measurement, as well as the first time code and the second time code.

[0008] Medical professionals may be interested in analyzing received data in relation to other data. Measurements of biomarkers, the environment, and surgical instruments can provide context to one another. Therefore, it is important for medical professionals to know whether measurements from surgical sensing systems are being taken simultaneously or at different times offset from each other. For example, clock drift can occur when the real-time clock of a sensing system becomes asynchronous with the device from which it transmits data, or a changing data rate or processing delay can asynchronously affect measurements, or a stream of measurements or data. Medical professionals can determine contextual information from multiple sensing systems if the data received from each sensing system is synchronized with each other. The apparatus and method of the present invention enable the use of timecodes to compensate for measurement latency, thereby allowing measurements from various surgical sensing systems to be precisely ordered and synchronized with other data and precise contextual information derived by the medical professional.

[0009] Received measurements may be ordered. Measurements may be associated with a surgical sensing system. Measurements may be associated with a communication interface. Measurements may be associated with both a surgical sensing system and a communication interface. Measurements may be received first. Latency values ​​may be received. Latency values ​​may be associated with a surgical sensing system. Latency values ​​may be associated with a communication interface. Latency values ​​may be associated with both a surgical sensing system and a communication interface. Time codes may be applied. Time codes may be applied to received measurements. Time codes may be applied based on a first time. Time codes may be applied based on acquired latency values. Time codes may be applied based on a first time and acquired latency values. Outputs may be ordered. Outputs may be ordered based on received measurements. Outputs may be ordered based on applied time codes. Outputs may be ordered based on received measurements and applied time codes.

[0010] In one embodiment, the surgical sensing system may include one or more sensors. The surgical sensing system may include one or more sensors configured to sense at least one biomarker. The surgical sensing system may include one or more sensors configured to sense at least one environment. The surgical sensing system may include one or more sensors configured to sense one or more of the biomarker and / or environment. The surgical sensing system may be configured to sense at least one surgical instrument parameter. [Brief explanation of the drawing]

[0011] [Figure 1A] This is a block diagram of a computer-implemented patient and surgeon monitoring system. [Figure 1B] This is a block diagram illustrating exemplary relationships between sensing systems, biomarkers, and physiological systems. [Figure 2A] This figure shows an example of a surgeon monitoring system in a surgical operating room. [Figure 2B] This figure shows an example of a patient monitoring system (for example, a controlled patient monitoring system). [Figure 2C] This figure shows an example of a patient monitoring system (for example, an uncontrolled patient monitoring system). [Figure 3] This figure shows an exemplary surgical hub paired with various systems. [Figure 4] This figure shows a surgical data network having a set of communication surgical hubs configured to connect to a set of sensing systems, an environmental sensing system, a set of devices, and so on. [Figure 5] This figure shows an exemplary computer-implemented bidirectional surgical system, which may be part of a surgeon monitoring system. [Figure 6A] This diagram shows a surgical hub comprising multiple modules connected to a modular control tower. [Figure 6B] This figure shows an example of a controlled patient monitoring system. [Figure 6C] This figure shows an example of an uncontrolled patient monitoring system. [Figure 7A] This is a logic diagram of a control system for surgical instruments or tools. [Figure 7B] This figure shows an exemplary sensing system having a sensor unit and a data processing and communication unit. [Figure 7C] This figure shows an exemplary sensing system having a sensor unit and a data processing and communication unit. [Figure 7D] This figure shows an exemplary sensing system having a sensor unit and a data processing and communication unit. [Figure 8] This diagram shows an exemplary timeline of a surgical procedure illustrating the adjustment of operating parameters of surgical equipment based on the surgeon's biomarker level. [Figure 9] This is a block diagram of a computer-implemented bidirectional surgeon / patient monitoring system. [Figure 10]A diagram showing an exemplary surgical system including a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter. [Figure 11A] A diagram showing an example of a sensing system that can be used to monitor a surgeon biomarker or a patient biomarker. [Figure 11B] A diagram showing an example of a sensing system that can be used to monitor a surgeon biomarker or a patient biomarker. [Figure 11C] A diagram showing an example of a sensing system that can be used to monitor a surgeon biomarker or a patient biomarker. [Figure 11D] A diagram showing an example of a sensing system that can be used to monitor a surgeon biomarker or a patient biomarker. [Figure 12] A block diagram of a patient monitoring system or a surgeon monitoring system. [Figure 13] A diagram showing an exemplary display including a visual representation of sensed measurements. [Figure 14A] Functional block diagrams of an exemplary surgical data ordering system and an exemplary processing unit, respectively. [Figure 14B] Functional block diagrams of an exemplary surgical data ordering system and an exemplary processing unit, respectively. [Figure 15] A diagram showing a timeline illustrating an exemplary method of obtaining a latency value associated with a surgical sensing system. [Figure 16] A diagram showing a timeline illustrating an exemplary method of obtaining a latency value associated with a surgical sensing system. [Figure 17] A diagram showing a timeline illustrating an exemplary method of selecting sensed measurements. [Figure 18] A diagram depicting an exemplary master time log having surgical sensing system data. [Figure 19] A diagram showing an exemplary method for ordering surgical sensing system data. < [Modes for carrying out the invention]

[0012] Figure 1A is a block diagram of a computer-implemented patient and surgeon monitoring system 20000. The patient and surgeon monitoring system 20000 may include one or more surgeon monitoring systems 20002 and one or more patient monitoring systems (e.g., one or more controlled patient monitoring systems 20003 and one or more uncontrolled patient monitoring systems 20004). Each surgeon monitoring system 20002 may include a computer-implemented bidirectional surgical system. Each surgeon monitoring system 20002 may include at least one of a surgical hub 20006 that communicates with a cloud computing system 20008, for example, as shown in Figure 2A. Each patient monitoring system may include at least one of a surgical hub 20006 or a computing device 20016 that communicates with a cloud computing system 20008, for example, as further shown in Figures 2B and 2C. The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Each of the surgeon monitoring system 20002, the controlled patient monitoring system 20003, or the uncontrolled patient monitoring system 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent devices 20014, a human interface system 20012, etc. The human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 may include one or more surgeon sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include one or more devices used to measure one or more environmental attributes, for example, as further described in Figure 2A. The robotic system 20013 (same as 20034 in Figure 2A) may include, for example, multiple devices used to perform surgical procedures, as further described in Figure 2A.

[0013] The surgical hub 20006 may have collaborative interaction with one or more means for displaying images from a laparoscope, as well as information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 may interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 may be configured to collect measurement data from one or more sensing systems 20011 and to transmit notification or control messages to one or more sensing systems 20011. The surgical hub 20006 may transmit and / or receive information, including notification information, to and from a human interface system 20012. The human interface system 20012 may include one or more human interface devices (HIDs). The surgical hub 20006 may transmit and / or receive notification or control information to audio equipment, displays, and / or control information to various devices communicating with the surgical hub.

[0014] Figure 1B is a block diagram of an exemplary relationship between a sensing system 20001, a biomarker 20005, and a physiological system 20007. This relationship may be used in a computer-implemented patient and surgeon monitoring system 20000, and in the systems, apparatus, and methods disclosed herein. For example, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more surgeon sensing systems and one or more patient sensing systems) and an environmental sensing system 20015, as discussed in Figure 1A. One or more sensing systems 20001 may measure data relating to various biomarkers 20005. One or more sensing systems 20001 may measure the biomarkers 20005 using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. One or more sensors may measure the biomarker 20005 as described herein, using one or more sensing techniques such as photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedimentary, potentiometric measurement, and current measurement.

[0015] Biomarkers 20005 measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, hydrogen blood potential, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, mental state, sweat, circulating tumor cells, autonomic nervous system tone, circadian rhythm, and / or menstrual cycle.

[0016] Biomarkers 20005 may relate to physiological systems 20007, including, but not limited to, behavioral and psychological, cardiovascular, renal, cutaneous, nervous, gastrointestinal, respiratory, endocrine, immune, oncological, musculoskeletal, and / or reproductive systems. Information from biomarkers may be determined and / or used, for example, by a computer-implemented patient and surgeon monitoring system 20000. Information from biomarkers may be determined and / or used by the computer-implemented patient and surgeon monitoring system 20000 to, for example, improve the system and / or improve patient outcomes.

[0017] Figure 2A shows an example of a surgeon monitoring system 20002 in a surgical operating room. As shown in Figure 2A, a patient is operated on by one or more healthcare professionals (HCPs). The HCPs are monitored by one or more surgeon sensing systems 20020 worn by the HCPs. The HCPs and the environment surrounding them may also be monitored by one or more environmental sensing systems, including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors that may be deployed in the operating room. The surgeon sensing systems 20020 and the environmental sensing systems communicate with a surgical hub 20006 and, further, with one or more cloud servers 20009 of a cloud computing system 20008, as shown in Figure 1. The environmental sensing systems may be used to measure one or more environmental attributes, such as the HCP's position in the operating room, HCP movement, ambient noise in the operating room, and temperature / humidity in the operating room.

[0018] As shown in Figure 2A, the main display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned in the sterile field so that they are visible to the operator on the operating table 20024. In addition, a visualization / notification tower 20026 is positioned outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human-interactive device (HID) 20027 and a second non-sterile HID 20029 facing opposite directions from each other. The HIDs may be displays or displays having touchscreens that allow humans to interface directly with the HIDs. A human interface system guided by the surgical hub 20006 may be configured to utilize the HIDs 20027, 20029, and 20023 to coordinate the flow of information to operators inside and outside the sterile field. In one embodiment, the surgical hub 20006 may cause the HIDs (e.g., the main HID 20023) to display notifications and / or information regarding the patient and / or surgical steps. In one embodiment, the surgical hub 20006 may prompt and / or receive input from a person in a sterile or non-sterile area. In one embodiment, the surgical hub 20006 may display snapshots of the surgical site recorded by the imaging device 20030 on non-sterile HID 20027 or 20029 while maintaining live video of the surgical site on the main HID 20023. The snapshots on the non-sterile display 20027 or 20029 may, for example, allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0019] In one embodiment, the surgical hub 20006 may be configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 20026 to a main display 20023 in the sterile field, which can then be viewed by a sterile operator on the operating table. In one embodiment, the input may take the form of modifications to snapshots displayed on non-sterile displays 20027 or 20029, which can then be sent to the main display 20023 by the surgical hub 20006.

[0020] Referring to Figure 2A, surgical instrument 20031 is used as part of a surgeon monitoring system 20002 during a surgical procedure. Hub 20006 may also be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator in visualization tower 20026 is sent by Hub 20006 to a surgical instrument display in the sterile field, which can be viewed by the operator of surgical instrument 20031. Examples of surgical instruments suitable for use with surgical system 20002 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application No. 16 / 209,385, “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY” No. 2019-0200844(A1), filed December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0021] Figure 2A shows an example of a surgical system 20002 used to perform surgery on a patient lying on an operating table 20024 in an operating room 20035. A robotic system 20034 may be used as part of the surgical system 20002 in a surgical procedure. The robotic system 20034 may include a surgeon's console 20036, a patient-side cart 20032 (surgical robot), and a surgical robot hub 20033. While the surgeon views the surgical site through the surgeon's console 20036, the patient-side cart 20032 can manipulate at least one detachably connected surgical tool 20037 through a minimally invasive incision in the patient's body. Images of the surgical site may be acquired by a medical imaging device 20030, which can be operated by the patient-side cart 20032 to change the orientation of the imaging device 20030. The robot hub 20033 can be used to process images of the surgical site, which can then be displayed to the surgeon through the surgeon's console 20036.

[0022] Other types of robotic systems can be readily adapted for use with surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed December 4, 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0023] Various examples of cloud-based analytical methods implemented by Cloud Computing System 20008 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the disclosure of which is incorporated herein by reference in its entirety.

[0024] In various embodiments, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0025] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0026] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light or simply light. The human eye typically responds to wavelengths in air between approximately 380 nm and 750 nm.

[0027] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0028] In various embodiments, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0029] The imaging device may employ a multispectral monitor to distinguish between topography and underlying structures. A multispectral image captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging can extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is necessary in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including imaging devices and their accessories and components. It will be understood that the sterile field may be considered a specific area deemed free of microorganisms, such as within a tray or on a sterile towel, or it may be considered the area immediately surrounding a patient ready for surgery. The sterile field may include cleaned team members in appropriate clothing, as well as all equipment and restraints within that area.

[0030] The wearable sensing system 20011 shown in Figure 1 may include one or more sensing systems, for example, a surgical sensing system 20020 as shown in Figure 2A. The surgical sensing system 20020 may include sensing systems for monitoring and detecting a set of physical and / or physiological conditions of a healthcare provider (HCP). The HCP is generally one or more healthcare professionals assisting a surgeon or other healthcare service provider. In one embodiment, the sensing system 20020 may measure a set of biomarkers to monitor the HCP's heart rate. In another embodiment, the sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movements and / or tremors and determine the magnitude and frequency of the tremors. The sensing system 20020 may transmit the measured data associated with the set of biomarkers and the data associated with the surgeon's physical condition to the surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing device may include a camera 20021 for detecting the position of the HCP's hand / body. The environmental sensing device may also include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include, for example, a thermometer for measuring temperature and a hygrometer for measuring ambient humidity in the operating room. The surgical hub 20006 may, either independently or in communication with a cloud computing system, use the surgeon biomarker measurement data and / or environmental sensing information to, for example, modify the control algorithm for handheld instruments or the average delay of a robotic interface to minimize tremor. In one embodiment, the surgeon sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and transmit the measurement data associated with the surgeon biomarkers to the surgical hub 20006.The surgical sensing system 20020 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth®, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), and Wi-Fi. Surgical biomarkers may include one or more of the following: stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels related to the movements of the surgeon or patient, the surgeon and / or staff, and the attention levels of the surgeon and / or staff.

[0031] The surgical hub 20006 can adaptively control one or more surgical instruments 20031 using surgeon biomarker measurement data associated with the HCP. For example, the surgical hub 20006 may transmit a control program to the surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may transmit a control program based on contextual awareness and / or circumstances relating to the importance or criticality of the task. The control program may instruct the instrument to modify its behavior to provide more control when control is needed.

[0032] Figure 2B shows an example of a patient monitoring system 20003 (e.g., a controlled patient monitoring system). As shown in Figure 2B, a patient in a controlled environment (e.g., a recovery room) may be monitored by multiple sensing systems (e.g., patient sensing system 20041). Patient sensing system 20041 (e.g., a headband) may be used to measure electroencephalography (EEG) to measure the electrical activity of the patient's brain. Patient sensing system 20042 may be used to measure various biomarkers of the patient, including, for example, heart rate, VO2 levels, etc. Patient sensing system 20043 (e.g., a flexible patch attached to the patient's skin) may be used to measure sweat lactate and / or potassium levels by analyzing small amounts of sweat captured from the skin surface using microfluidic channels. Patient sensing system 20044 (e.g., a wristband or watch) may be used to measure blood pressure, heart rate, heart rate variability, VO2 levels, etc., using various techniques as described herein. Patient sensing systems 20045 (e.g., finger rings) may be used to measure peripheral temperature, heart rate, heart rate variability, VO2 levels, etc., using various techniques as described herein. Patient sensing systems 20041-20045 may communicate with the surgical hub 20006 using a radio frequency (RF) link. Patient sensing systems 20041-20045 may use one or more RF protocols such as Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), Thread, Wi-Fi, etc., for communication with the surgical hub 20006.

[0033] Sensing systems 20041-20045 communicate with the surgical hub 20006 and may further communicate with the remote server 20009 of the remote cloud computing system 20008. The surgical hub 20006 also communicates with HID20046. HID20046 may display measurement data associated with one or more patient biomarkers. For example, HID20046 may display blood pressure, oxygen saturation, respiratory rate, etc. HID20046 may display notifications for the patient or HCP that provide information about the patient, such as information about recovery milestones or complications. In one embodiment, information about recovery milestones or complications may be associated with any surgical procedures the patient may have undergone. In one embodiment, HID20046 may display commands for the patient to perform activities. For example, HID20046 may display inhalation commands and exhalation commands. In one embodiment, HID20046 may be part of the sensing system.

[0034] As illustrated in Figure 2B, the patient and the environment surrounding the patient may be monitored by one or more environmental sensing systems 20015, including, for example, microphones (for detecting ambient noise associated with or around the patient), temperature / humidity sensors, and cameras for detecting the patient's breathing patterns. The environmental sensing systems 20015 communicate with the surgical hub 20006 and further communicate with the remote server 20009 of the remote cloud computing system 20008.

[0035] In one embodiment, the patient sensing system 20044 may receive notification information from the surgical hub 20006 for display on the display unit or HID of the patient sensing system 20044. The notification information may include notifications regarding recovery milestones or, for example, notifications regarding complications in the case of postoperative recovery. In one embodiment, the notification information may include a manageable severity level associated with the notification. The patient sensing system 20044 may display the notification and the manageable severity level to the patient. The patient sensing system may alert the patient using haptic feedback. Visual and / or haptic notifications may be accompanied by audible notifications prompting the patient to pay attention to the visual notification provided on the sensing system's display unit.

[0036] Figure 2C shows an example of a patient monitoring system (e.g., an uncontrolled patient monitoring system 20004). As shown in Figure 2C, a patient in an uncontrolled environment (e.g., the patient's residence) is monitored by multiple patient sensing systems 20041–20045. Patient sensing systems 20041–20045 may measure and / or monitor measurement data associated with one or more patient biomarkers. For example, patient sensing system 20041, a headband, may be used to measure electroencephalography (EEG). Other patient sensing systems 20042, 20043, 20044, and 20045 are examples in which various patient biomarkers are monitored, measured, and / or reported, as shown in Figure 2B. One or more of the patient sensing systems 20041–20045 may transmit measurement data associated with the monitored patient biomarker to a computing device 20047 and further communicate with a remote server 20009 of a remote cloud computing system 20008. Patient detection systems 20041-20045 may use radio frequency (RF) links to communicate with computing devices 20047 (e.g., smartphones, tablets, etc.). Patient detection systems 20041-20045 may use one or more RF protocols such as Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), Thread, and Wi-Fi for communication with computing devices 20047. In one embodiment, patient detection systems 20041-20045 may be connected to computing devices 20047 via a wireless router, wireless hub, or wireless bridge.

[0037] Computing device 20047 may communicate with remote server 20009, which is part of cloud computing system 20008. In one embodiment, computing device 20047 may communicate with remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one embodiment, patient sensing system may communicate directly with remote server 20009. Computing device 20047 or sensing system may communicate with remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0038] In one embodiment, the computing device 20047 may display information associated with patient biomarkers. For example, the computing device 20047 may display blood pressure, oxygen saturation, respiratory rate, etc. The computing device 20047 may also display notifications for the patient or HCP that provide information about the patient, such as recovery milestones or complications.

[0039] In one embodiment, the computing device 20047 and / or the patient sensing system 20044 may receive notification information from the surgical hub 20006 for display on the display unit of the computing device 20047 and / or the patient sensing system 20044. The notification information may include notifications regarding recovery milestones or, for example, notifications regarding complications in the case of postoperative recovery. The notification information may also include a manageable severity level associated with the notification. The computing device 20047 and / or the sensing system 20044 may display the notifications and manageable severity levels to the patient. The patient sensing system may alert the patient using haptic feedback. Visual and / or haptic notifications may be accompanied by audible notifications prompting the patient to pay attention to visual notifications provided on the display unit of the sensing system.

[0040] Figure 3 shows an exemplary surgical monitoring system 20002 having a surgical hub 20006 paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain embodiments, as shown in Figure 3, the hub 20006 further includes a fume extraction module 20054 and / or a suction / irrigation module 20055. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, suction of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Dealing with this problem during surgery can result in the loss of valuable time. Untangling lines may require disconnecting them from their corresponding modules, which may necessitate resetting the modules. The hub modular enclosure 20060 provides an integrated environment for managing power lines, data lines, and fluid lines, reducing the frequency of such line entanglements. Aspects of this disclosure present a surgical hub 20006 for use in surgical procedures involving the application of energy to tissue at the surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combination generator module slidably received within a docking station of the hub enclosure 20060. The docking station includes data and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed within a single unit.In one embodiment, the combination generator module also includes a fume extraction component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume extraction component configured to exhaust smoke, fluid and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the fume extraction component. In one embodiment, the fluid line may be a first fluid line, and a second fluid line may extend from the remote surgical site to a suction and irrigation module 20055 slidably received within a hub enclosure 20060. In one embodiment, the hub enclosure 20060 may include a fluid interface. Some specific surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of the present disclosure present a solution in which a hub modular enclosure 20060 is configured to house various generators and facilitate bidirectional communication between them. One of the advantages of the hub modular enclosure 20060 is that it allows for the rapid removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the electrical engagement with the first power and data contacts.In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, distinct from a first energy, and a second docking station having a second docking port including a second data contact and a second power contact, wherein the second energy generator module is slidably movable to electrically engage with the power contact and the data contact, and the second energy generator module is slidably movable to disengage from the electrical engagement with the second power contact and the second data contact. In addition, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port, configured to facilitate communication between the first energy generator module and the second energy generator module. Referring to Figure 3, an aspect of the present disclosure relating to a hub modular enclosure 20060 enabling modular integration of a generator module 20050, a fume exhaust module 20054, and a suction / irrigation module 20055 is presented. The hub modular enclosure 20060 further facilitates bidirectional communication between modules 20059, 20054, and 20055. The generator module 20050 may be a generator module 20050 having integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the hub modular enclosure 20060. The generator module 20050 may be configured to connect to a unipolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the hub modular enclosure 20060. The hub modular enclosure 20060 can be configured to facilitate the insertion of multiple generators and bidirectional communication between generators docked to the hub modular enclosure 20060, so that multiple generators function as a single generator.

[0041] Figure 4 shows a surgical data network, according to at least one aspect of the present disclosure, having a set of communication hubs configured to connect a set of sensing systems, an environmental sensing system, and other modular devices to a cloud, located in one or more operating rooms, patient recovery rooms, or rooms within a medical facility specifically equipped for surgery.

[0042] As shown in Figure 4, the surgical hub system 20060 may include a modular communication hub 20065 configured to connect modular devices located within a medical facility to a cloud-based system (e.g., a cloud computing system 20064 which may include a remote server 20067 connected to remote storage 20068). The modular communication hub 20065 and devices may be connected in a room within the medical facility specifically equipped for surgical procedures. In one embodiment, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicates with a network router 20066. The modular communication hub 20065 may also be connected to a local computer system 20063 to provide local computer processing and data manipulation. The surgical data network associated with the surgical hub system 20060 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to travel from one device (or segment) to another device (or segment) and to cloud computing resources. The intelligent surgical data network enables traffic to pass through the monitored surgical data network and includes additional mechanisms that constitute each port within the network hub 20061 or network switch 20062. The intelligent surgical data network may be referred to as a manageable hub or switch. The switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0043] Modular devices 1a-1n, located in the operating room, may be connected to a modular communication hub 20065. A network hub 20061 and / or a network switch 20062 may be connected to a network router 20066 to connect devices 1a-1n to a cloud computing system 20064 or a local computer system 20063. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and operation. Data associated with devices 1a-1n may also be transferred to a local computer system 20063 for local data processing and operation. Modular devices 2a-2m, located in the same operating room, may also be connected to a network switch 20062. The network switch 20062 may be connected to a network hub 20061 and / or a network router 20066 to connect devices 2a-2m to the cloud 20064. Data associated with devices 2a-2m may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. The data associated with devices 2a-2m may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0044] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing systems 20069 may include a surgeon sensing system and / or a patient sensing system. One or more sensing systems 20069 may communicate with the computer system 20063 or cloud server 20067 of the surgical hub system 20060 directly via one of the network routers 20066, or via a network hub 20061 or network switch 20062 that communicates with the network router 20066.

[0045] The sensing system 20069 may be connected to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. Data associated with the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0046] As shown in Figure 4, the surgical hub system 20060 can be expanded by interconnecting multiple network hubs 20061 and / or multiple network switches 20062 with multiple network routers 20066. The modular communication hub 20065 can be housed in a modular control tower configured to accommodate multiple devices 1a-1n / 2a-2m. The local computer system 20063 may also be housed in the modular control tower. The modular communication hub 20065 can be connected to a display 20068 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during surgical procedures. In various embodiments, the apparatus 1a-1n / 2a-2m may include various modules, such as imaging modules connected to an endoscope, generator modules connected to energy-based surgical devices, smoke extraction modules, suction / irrigation modules, communication modules, processor modules, storage arrays, surgical devices connected to displays, and / or non-contact sensor modules, among other modular devices that can be connected to the modular communication hub 20065 of the surgical data network.

[0047] In one embodiment, the surgical hub system 20060 shown in Figure 4 may comprise a combination of a network hub, network switch, and network router connecting devices 1a-1n / 2a-2m or sensing systems 20069 to a cloud-based system 20064. One or more of the devices 1a-1n / 2a-2m or sensing systems 20069 connected to the network hub 20061 or network switch 20062 may collect data or measurement data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” can be used as a metaphor for “Internet,” but the term is not limited to that. Therefore, the term “cloud computing” may be used herein to refer to “one type of internet-based computing,” in which various services such as servers, storage, and applications are delivered via the Internet to a modular communication hub 20065 and / or computer system 20063 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 20065 and / or computer system 20063. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. The cloud computing service may perform numerous calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to connect to a computer that communicates with cloud computing resources and storage.

[0048] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and improved patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud computing system 20064 or the local computer system 20063, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcomes of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be performed on tissue-specific sites and conditions. Such data analysis may further involve prognostic analysis processing, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0049] By applying cloud computing data processing techniques to measurement data collected by sensing systems 20069, surgical data networks can result in improved surgical outcomes, improved recovery outcomes, reduced costs, and improved patient satisfaction. At least some of the sensing systems 20069 may be used to assess the physiological state of a surgeon operating on a patient, a patient preparing for surgery, or a patient recovering after surgery. Cloud-based computing systems 20064 may be used to monitor biomarkers associated with a surgeon or patient in real time, generate surgical plans based at least on measurement data collected before surgery, supply control signals to surgical instruments during surgery, and notify patients of complications during the postoperative period.

[0050] Operating room devices 1a-1n may be connected to the modular communication hub 20065 via a wired or wireless channel, depending on the configuration of devices 1a-1n with respect to the network hub 20061. In one embodiment, the network hub 20061 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same operating room network. The network hub 20061 may collect data in packet form and transmit them to the router in half-duplex mode. The network hub 20061 also cannot store media access control (MAC / Internet Protocol, IP) for transferring device data. Only one of devices 1a-1n may transmit data through the network hub 20061 at a time. Network hub 20061 lacks routing tables or intelligence regarding the destination of information and broadcasts all network data to each connection and to the remote server 20067 of the cloud computing system 20064. While network hub 20061 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and may cause bottlenecks.

[0051] Operating room devices 2a-2m may be connected to network switch 20062 via a wired or wireless channel. Network switch 20062 operates within the data link layer of the OSI model. Network switch 20062 may also be a multicast device for connecting devices 2a-2m located in the same operating room to a network. Network switch 20062 transmits data in the form of frames to network router 20066 and may operate in full-duplex mode. Multiple devices 2a-2m may transmit data simultaneously through network switch 20062. Network switch 20062 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0052] Network hub 20061 and / or network switch 20062 may be coupled to network router 20066 to connect to cloud computing system 20064. Network router 20066 operates within the network layer of the OSI model. Network router 20066 creates routes for sending data packets received from network hub 20061 and / or network switch 20062 to cloud-based computing resources for further processing and manipulation of data collected by one or all of the devices 1a-1n / 2a-2m and wearable sensing system 20011. Network router 20066 may be used to connect two or more different networks located in different locations, for example, different networks located in different operating rooms of the same medical facility, or different networks located in different operating rooms of different medical facilities. Network router 20066 transmits data in packet form to cloud computing system 20064 and may operate in full-duplex mode. Multiple devices can transmit data simultaneously. Network router 20066 may use IP addresses to transfer data.

[0053] In one embodiment, the network hub 20061 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 20061 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0054] In the embodiment, the operating room devices 1a-1n / 2a-2m and / or sensing system 20069 may communicate with a modular communication hub 20065 via the Bluetooth wireless technology standard to exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and to build a personal area network (PAN). The operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 may communicate with the modular communication hub 20065 via several wireless or wired communication standards or protocols, including but not limited to Bluetooth, Low-Energy Bluetooth, Near Field Communication (NFC), Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long-Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G and later. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to short-range wireless communication such as Wi-Fi, Bluetooth, Low-Energy Bluetooth, and Bluetooth Smart, while the second communication module may be dedicated to long-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, and TDMA.

[0055] The modular communication hub 20065 functions as a central connection point for one or more of the operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 and can handle a data type known as a frame. A frame can carry data generated by the devices 1a-1n / 2a-2m and / or sensing systems 20069. When a frame is received by the modular communication hub 20065, it is amplified and transmitted to the network router 20066, which can then forward this data to the cloud computing system 20064 or the local computer system 20063 by using a number of wireless or wired communication standards or protocols as described herein.

[0056] The modular communication hub 20065 may be used as a standalone device or connected to compatible network hubs 20061 and network switches 20062 to form a larger network. Because the modular communication hub 20065 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room equipment 1a-1n / 2a-2m.

[0057] Figure 5 shows a computer-implemented bidirectional surgical system 20070, which may be part of the surgeon monitoring system 20002. The computer-implemented bidirectional surgical system 20070 is similar in many ways to the surgeon sensing system 20002. For example, the computer-implemented bidirectional surgical system 20070 may include one or more surgical subsystems 20072 that are similar in many ways to the surgeon monitoring system 20002. Each sub-surgical system 20072 includes at least one surgical hub 20076 that communicates with a cloud computing system 20064, which may include a remote server 20077 and remote storage 20078. In one embodiment, the computer-implemented bidirectional surgical system 20070 may include a modular control tower 20085 connected to multiple operating room devices, such as sensing systems (e.g., surgeon sensing system 20002 and / or patient sensing system 20003), intelligent surgical instruments, robots, and other computerized devices placed in the operating room. As shown in Figure 6A, the modular control tower 20085 may include a modular communication hub 20065 connected to the local computing system 20063.

[0058] As shown in the embodiment of Figure 5, the modular control tower 20085 can be connected to an imaging module 20088 which can be connected to an endoscope 20087, a generator module 20090 which can be connected to an energy device 20089, a fume exhaust module 20091, a suction / irrigation module 20092, a communication module 20097, a processor module 20093, a storage array 20094, a smart device / instrument 20095 which can be optionally connected to displays 20086 and 20084, respectively, and a non-contact sensor module 20096. The modular control tower 20085 can also communicate with one or more sensing systems 20069 and an environmental sensing system 20015. The sensing system 20069 can be connected to the modular control tower 20085 directly via a router or via the communication module 20097. The operating room equipment can be connected to cloud computing resources and data storage via the modular control tower 20085. The robotic surgery hub 20082 may also be connected to the modular control tower 20085 and cloud computing resources. In particular, the apparatus / instrument 20095 or 20084 and the human interface system 20080 may be connected to the modular control tower 20085 via wired or wireless communication standards or protocols as described herein. The human interface system 20080 may include a display subsystem and a notification subsystem. The modular control tower 20085 may be connected to a hub display 20081 (e.g., a monitor, screen) to display and overlay images received from the imaging module 20088, the apparatus / instrument display 20086, and / or other human interface systems 20080. The hub display 20081 may also display data received from devices connected to the modular control tower 20085 along with the images and overlay images.

[0059] Figure 6A shows a surgical hub 20076 comprising multiple modules connected to a modular control tower 20085. As shown in Figure 6A, the surgical hub 20076 may be connected to a generator module 20090, a fume exhaust module 20091, a suction / irrigation module 20092, and a communication module 20097. The modular control tower 20085 may comprise a modular communication hub 20065, such as a network connectivity device, and a computer system 20063, for example, to perform local wireless connectivity with sensing systems, local processing, complication monitoring, visualization, and imaging. As shown in Figure 6A, the modular communication hub 20065 may be connected in a configuration (e.g., a tiered configuration) to extend a number of modules (e.g., devices) and a number of sensing systems 20069 that can be connected to the modular communication hub 20065, and to transfer data associated with modules and / or measurement data associated with sensing systems 20069 to the computer system 20063, cloud computing resources, or both. As shown in Figure 6A, each of the network hubs / switches 20061 / 20062 within the modular communication hub 20065 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to processor 20102 to provide communication connectivity to cloud computing resources and local display 20108. At least one of the network / hub switches 20061 / 20062 within the modular communication hub 20065 may have at least one wireless interface to provide communication connectivity between sensing system 20069 and / or device 20095 and cloud computing system 20064. Communication to cloud computing system 20064 may be carried out via either a wired or wireless communication channel.

[0060] The surgical hub 20076 may utilize non-contact sensor module 20096 to measure the dimensions of the operating room and generate a map of the operating room using either an ultrasonic non-contact measuring device or a laser-type non-contact measuring device. In U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM,” which is incorporated herein by reference in its entirety, the sensor module is configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. However, as described in the same document under the heading “Surgical Hub Spatial Awareness Within an Operating Room,” an ultrasonic-based non-contact sensor module may scan the operating room by transmitting bursts of ultrasound and receiving echoes as the ultrasound bursts reflect off the outer walls of the operating room. A laser-based non-contact sensor module may scan the operating room, for example, by transmitting laser light pulses, receiving the laser light pulses reflected off the outer walls of the operating room, comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0061] Computer system 20063 may comprise a processor 20102 and a network interface 20100. The processor 20102 may be connected via a system bus to a communication module 20103, storage 20104, memory 20105, non-volatile memory 20106, and an input / output (I / O) interface 20107. The system bus can be any of several types of bus structures, including a memory bus or memory controller, peripheral bus or external bus, and / or local bus, using any various available bus architectures. Examples of such architectures include, but are not limited to, a 9-bit bus, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0062] The processor 20102 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0063] In one embodiment, the 20102 processor may include a safety controller comprising two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured specifically for IEC61508 and ISO26262 safety limit applications, in particular, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0064] System memory can be categorized into volatile and non-volatile memory. The Basic Input / Output System (BIOS), which includes basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. Examples of non-volatile memory include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Examples of volatile memory include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0065] Computer System 20063 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage devices may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of disk storage devices to the system bus.

[0066] It should be understood that computer system 20063 may include software that acts as an intermediary between the described user and basic computer resources in a suitable operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0067] The user can input commands or information to the computer system 20063 via input devices connected to the I / O interface 20107. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor 20102 via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system 20063 and output information from the computer system 20063 to the output device. Output adapters may be provided to indicate that some output devices, such as monitors, displays, speakers, and printers, may be present, among others, which may require special adapters. Examples of output adapters include video and sound cards that provide a means of connection between the output device and the system bus, but these are illustrative and not limiting. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0068] Computer System 20063 may operate in a networked environment using one or more remote computers, such as cloud computers, or logical connections to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication interface. Network interfaces may encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include Fiber Optic Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL).

[0069] In various embodiments, the computer system 20063 in Figures 4, 6A, and 6B, the imaging module 20088 and / or the human interface system 20080 and / or the processor module 20093 in Figures 5 and 6A may comprise an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0070] The communication connection section may also refer to the hardware / software used to connect the network interface to the bus. For illustrative purposes, the communication connection section is shown as being inside the computer system 20063, but it may also be outside the computer system 20063. For illustrative purposes only, hardware / software required for connecting to the network interface may include internal and external technologies such as modems including typical telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0071] Figure 6B shows an example of a wearable monitoring system, such as a controlled patient monitoring system. The controlled patient monitoring system may be a sensing system used to monitor a set of patient biomarkers while the patient is in a medical facility. The controlled patient monitoring system may be deployed for pre-surgical patient monitoring while the patient is preparing for surgery, intra-surgical monitoring while the patient is undergoing surgery, or post-surgical monitoring, for example, while the patient is recovering. As shown in Figure 6B, the controlled patient monitoring system may include a surgical hub system 20076 which may include one or more routers 20066 of a modular communication hub 20065 and a computer system 20063. The routers 20065 may include wireless routers, wired switches, wired routers, wired or wireless network hubs, etc. In one embodiment, the routers 20065 may be part of the infrastructure. Computing system 20063 may provide local processing for monitoring various biomarkers associated with the patient or surgeon, and a notification mechanism for informing the patient and / or healthcare provider (HCP) that milestones (e.g., recovery milestones) have been met or that complications have been detected. Computing system 20063 of surgical hub system 20076 may also be used to generate notifications, for example, severity levels associated with a notification that complications have been detected.

[0072] The computing system 20063 in Figures 4 and 6B, the computing device 20200 in Figure 6C, and the hub / computing device 20243 in Figures 7B, 7C, or 7D may be a surgical computing system or hub device, a laptop, tablet, smartphone, etc.

[0073] As shown in Figure 6B, a set of sensing systems 20069 and / or environmental sensing systems 20015 (as described in Figure 2A) may be connected to the surgical hub system 20076 via a router 20065. The router 20065 may also provide a direct communication connection between the sensing systems 20069 and the cloud computing system 20064, for example, without involving the local computer system 20063 of the surgical hub system 20076. Communication from the surgical hub system 20076 to the cloud 20064 may be carried out via either a wired or wireless communication channel.

[0074] As shown in Figure 6B, the computer system 20063 may include a processor 20102 and a network interface 20100. The processor 20102 may be connected via a system bus to a radio frequency (RF) interface or communication module 20103, storage 20104, memory 20105, non-volatile memory 20106, and input / output interface 20107, as shown in Figure 6A. The computer system 20063 may be connected to a local display unit 20108. In some embodiments, the display unit 20108 may be replaced by a HID. Further details regarding the hardware and software components of the computer system are provided in Figure 6A.

[0075] As shown in Figure 6B, the sensing system 20069 may include a processor 20110. The processor 20110 may be connected via a system bus to a radio frequency (RF) interface 20114, storage 20113, memory (e.g., non-volatile memory) 20112, and an I / O interface 20111. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, as described herein. The processor 20110 may be any single-core or multi-core processor as described herein.

[0076] It should be understood that the sensing system 20069 may include software that acts as an intermediary between the sensing system user and the described computer resources in a suitable operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented on various operating systems or combinations of operating systems.

[0077] The sensing system 20069 may be connected to the human interface system 20115. The human interface system 20115 may be a touchscreen display. The human interface system 20115 may include a human interface display for displaying information associated with surgeon biomarkers and / or patient biomarkers, and may display notifications to the patient or surgeon, such as prompting for user actions by the patient or surgeon, or indicating information about recovery milestones or complications. The human interface system 20115 may be used to receive input from the patient or surgeon. Other human interface systems may be connected to the sensing system 20069 via the I / O interface 20111. For example, the human interface device 20115 may include a device for providing haptic feedback as a mechanism to prompt the user to pay attention to notifications that may be displayed on the display unit.

[0078] The sensing system 20069 may operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may include personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described with respect to computer systems. Remote computers may be logically connected to computer systems via network interfaces. Network interfaces may include communication networks such as local area networks (LANs), wide area networks (WANs), and / or mobile networks. LAN technologies include fiber optic distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, and Wi-Fi / IEEE 802.11. WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL). A mobile network may include communication links based on one or more mobile communication protocols such as GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0079] Figure 6C shows, for example, an exemplary uncontrolled patient monitoring system when the patient is away from the medical facility. An uncontrolled patient monitoring system may be used for pre-operative patient monitoring when the patient is preparing for surgery but is away from the medical facility, or for example, in post-operative monitoring when the patient is away from the medical facility recovering.

[0080] As shown in Figure 6C, one or more sensing systems 20069 communicate with a computing device 20200, such as a personal computer, laptop, tablet, or smartphone. The computing system 20200 may provide processing for monitoring various biomarkers associated with the patient, notification mechanisms indicating that milestones (e.g., recovery milestones) have been met, or that complications have been detected. The computing system 20200 may also provide instructions for the user of the sensing system to follow. Communication between the sensing system 20069 and the computing device 20200 may be established directly using a wireless protocol as described herein, or via a wireless router / hub 20211.

[0081] As shown in Figure 6C, the sensing system 20069 may be connected to the computing device 20200 via a router 20211. The router 20211 may include a wireless router, a wired switch, a wired router, a wired or wireless network hub, etc. The router 20211 may, for example, enable a direct communication connection between the sensing system 20069 and the cloud server 20064 without involving the local computing device 20200. The computing device 20200 may communicate with the cloud server 20064. For example, the computing device 20200 may communicate with the cloud 20064 via a wired or wireless communication channel. In one embodiment, the sensing system 20069 may communicate directly with the cloud over a cellular network, for example, via a cellular base station 20210.

[0082] As shown in Figure 6C, the computing device 20200 may include a processor 20203 and a network or RF interface 20201. The processor 20203 may be connected via a system bus to storage 20202, memory 20212, non-volatile memory 20213, and input / output interface 20204, as shown in Figures 6A and 6B. Details of the hardware and software components of the computer system are provided in Figure 6A. The computing device 20200 may also include a set of sensors, e.g., sensor #1 20205, sensor #2 20206, sensor #n 20207. These sensors may be part of the computing device 20200 and may be used to measure one or more attributes associated with a patient. The attributes may provide context for biomarker measurements performed by one of the sensing systems 20069. For example, sensor #1 may be an accelerometer that can be used to measure acceleration forces to sense motion or vibration associated with a patient. In one embodiment, sensors 20205 to 20207 may include one or more of the following: a pressure sensor, an altimeter, a thermometer, a lidar, etc.

[0083] As shown in Figure 6B, the sensing system 20069 may include a processor, a radio frequency interface, storage, memory or non-volatile memory, and an input / output interface via a system bus, as described in Figure 6A. The sensing system may include a sensor unit and a processing and communication unit, as described in Figures 7B-7D. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, as described herein. The processor may be any single-core or multi-core processor, as described herein.

[0084] The sensing system 20069 may communicate with the human interface system 20215, which may be a touchscreen display. The human interface system 20215 may be used to display information associated with patient biomarkers, to display prompts for user actions by the patient, or to display notifications to the patient indicating information about recovery milestones or complications. The human interface system 20215 may be used to receive input from the patient. Other human interface systems may be connected to the sensing system 20069 via I / O interfaces. For example, a human interface system may include a device for providing haptic feedback as a mechanism to prompt the user to pay attention to notifications that may be displayed on the display unit. The sensing system 20069 may operate in a networked environment using one or more remote computers, such as cloud computers, or a logical connection to a local computer, as shown in Figure 6B.

[0085] Figure 7A shows a logic diagram of a control system 20220 for a surgical instrument or surgical tool according to one or more embodiments of the present disclosure. The surgical instrument or surgical tool may be configurable. The surgical instrument may include surgical equipment that is on hand and specific to the procedure, such as an imaging device, a surgical stapler, an energy device, an endocutter device, etc. For example, the surgical instrument may include any of the following: a powered stapler, a powered stapler generator, an energy device, a high-energy device, a high-energy jaw device, an endocutter clamp, an energy device generator, an intraoperative imaging system, a fume ventilator, a suction irrigation device, a pneumoperitoneum system, etc. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221 having a processor 20222 and memory 20223. For example, one or more of the sensors 20225, 20226, and 20227 provide real-time feedback to the processor 20222. Motor 20230, driven by motor driver 20229, drives the I-beam knife element by operably connecting a longitudinally movable displacement member. A tracking system 20228 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to a processor 20222, which may be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the launch member, launch bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control I-beam firing, occluder movement, shaft rotation, and joint movement. A display 20224 may display various operating states of the instrument and may include touchscreen functionality for data input. Information displayed on display 20224 may be overlaid with images acquired via the endoscopic imaging module.

[0086] In one embodiment, the microcontroller 20221 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 20221 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes on-chip memory of 256KB of single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance above 40MHz, 32KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs having 12 analog input channels.

[0087] In one embodiment, the microcontroller 20221 may include a safety controller comprising two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0088] The microcontroller 20221 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 20221 may include a processor 20222 and memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor driver 20229 may be the A3941 available from Allegro Microsystems, Inc. Other motor drivers can be readily substituted for use in the tracking system 20228 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” which is incorporated herein by reference in its entirety.

[0089] The microcontroller 20221 can be programmed to provide precise control over the velocity and position of the displacement member and joint motion system. The microcontroller 20221 can be configured to calculate the response within its software. The calculated response can be compared with the measured response of the actual system to obtain an "observed" response, which is used to determine the actual feedback. The observed response may be a suitable adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and this can detect external influences on the system.

[0090] In some examples, the motor 20230 may be controlled by a motor driver 20229 and utilized by a surgical instrument or tool launching system. In various forms, the motor 20230 may be a brushed DC-driven motor having a maximum rotational speed of approximately 25,000 RPM. In some examples, the motor 20230 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may comprise, for example, an H-bridge driver with field-effect transistors (FETs). The motor 20230 may be powered by a power supply assembly removably mounted to a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may comprise a battery that may include a number of battery cells connected in series, which can be used as a power source for supplying power to a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cell may be a lithium-ion battery that can be connected to and detached from a power supply assembly.

[0091] The motor drive unit 20229 may be the A3941, available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The drive unit 20229 may include a built-in charge pump regulator that supplies full (>10V) gate drive to a battery voltage down to 7V, allowing the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to supply the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side or low-side FET. The power FET may be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be readily substituted for use in the 20228 tracking system with an absolute positioning system.

[0092] The tracking system 20228 may comprise a controlled motor drive circuit configuration comprising a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for the absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element which may be displaced. In one aspect, the longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 20225 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include contact-type or non-contact-type displacement sensors. The linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable magnetoresistive transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0093] The electric motor 20230 may include a rotary shaft operably connected to a set of drive teeth on a displacement member, or to a gear assembly mounted in a meshing engagement with a rack of drive teeth. A sensor element may be operably connected to a gear assembly such that one rotation of the position sensor element 20225 corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply provides power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0094] One rotation of the sensor element associated with position sensor 20225 can correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor configuration may be connected via gear reduction, resulting in the position sensor 20225 completing one or more rotations relative to the full stroke of the displacement member. The position sensor 20225 may complete multiple rotations relative to the full stroke of the displacement member.

[0095] To provide a unique position signal for two or more rotations of the position sensor 20225, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches can be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 20225 is supplied to the microcontroller 20221. The sensor configuration of the position sensor 20225 may include an array of analog rotation sensors such as a magnetic sensor, a potentiometer, or an analog Hall effect element that outputs a unique combination of position signals or values.

[0096] The position sensor 20225 may comprise any number of magnetic sensing elements, such as magnetic sensors classified according to whether or not they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors may encompass numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0097] In one embodiment, the position sensor 20225 of the tracking system 20228, which includes an absolute positioning system, may include a magnetic rotation absolute positioning system. The position sensor 20225 may be implemented as the AS5055EQFT single-chip magnetic rotation position sensor, available from Austria Microsystems, AG. The position sensor 20225 is connected to the microcontroller 20221 to realize the absolute positioning system. The position sensor 20225 is a low-voltage, low-power component and may include four Hall effect elements in the area of ​​the position sensor 20225, which may be positioned above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement a simple and efficient algorithm for computing hyperbolic and trigonometric functions that require only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 20221 via a standard serial communication interface, such as a serial peripheral interface (SPI) interface. The position sensor 20225 can offer 12-bit or 14-bit resolution. The position sensor 20225 may also be an AS5055 chip, available in a small QFN 16-pin 4x4x0.85mm package.

[0098] The tracking system 20228, which includes an absolute positioning system, may also include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 20225, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. To predict what the state and output of the physical system will be by knowing the input, the calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, and inductive resistance.

[0099] The absolute positioning system can provide the absolute position of the displacement member when the device is powered on, without requiring a conventional rotary encoder to estimate the position of the device actuator, drive bar, knife, etc., which simply counts the number of steps taken forward or backward by the motor 20230, and does not require the displacement member to be moved back or forward to a reset (zero or home) position.

[0100] For example, sensor 20226, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as the amplitude of strain exerted on the anvil during clamping, for example, indicating the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to processor 20222. Instead of, or in addition to, sensor 20226, sensor 20227, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. For example, sensor 20227, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and make deformable contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, current sensor 20231 may be used to measure the current consumed by motor 20230. The force required to propel the launching member forward may, for example, correspond to the current consumed by the motor 20230. The measured force can be converted into a digital signal and provided to the processor 20222.

[0101] In one embodiment, a strain gauge sensor 20226 may be used to measure the force applied to the tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 20226, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 20226 may measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain can be converted into a digital signal and supplied to the processor 20222 of the microcontroller 20221. A load sensor 20227 may measure the force used to operate a knife element, for example, to cut tissue trapped between an anvil and a staple cartridge. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor may also be converted into a digital signal and supplied to the processor 20222.

[0102] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 20226 and 20227 respectively, may be used by the microcontroller 20221 to characterize the selected position of the launching member and / or the corresponding values ​​of the launching member's velocity. In one embodiment, memory 20223 may store techniques, equations, and / or lookup tables that may be used by the microcontroller 20221 during evaluation.

[0103] The surgical instrument or tool control system 20220 may also include wired or wireless communication circuits for communicating with the modular communication hub 20065, as shown in Figures 5 and 6A.

[0104] Figure 7B shows an exemplary sensing system 20069. The sensing system may be a surgical sensing system or a patient sensing system. Sensing system 20069 may include a sensor unit 20235 that communicates with a data processing and communication unit 20236 and a human interface system 20242. Data processing and communication unit 20236 may include an analog-to-digital converter 20237, a data processing unit 20238, a storage unit 20239, an input / output interface 20241, and a transceiver 20240. Sensing system 20069 can communicate with a surgical hub or computing device 20243, which then communicates with a cloud computing system 20244. Cloud computing system 20244 may include a cloud storage system 20078 and one or more cloud servers 20077.

[0105] The sensor unit 20235 may include one or more in vitro or intra vivo sensors to measure one or more biomarkers. Biomarkers may include, for example, blood pH, hydration status, oxygen saturation, core body temperature, heart rate, heart rate variability, sweat rate, skin conductance, blood pressure, light exposure, ambient temperature, respiratory rate, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, tissue perfusion pressure, bacteria in the airways, alcohol consumption, lactic acid (sweat), peripheral temperature, positivity and optimism, adrenaline (sweat), cortisol (sweat), edema, mycotoxins, VO2 max, preoperative pain, airborne chemicals, circulating tumor cells, stress and anxiety, confusion and delirium, physical activity, autonomic tension, circadian rhythm, menstrual cycle, sleep, etc. These biomarkers may be measured using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The sensor may measure biomarkers as described herein using one or more sensing techniques such as photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedimentary, potentiometric measurement, and current measurement.

[0106] As shown in Figure 7B, the sensor in sensor unit 20235 may measure physiological signals (e.g., voltage, current, PPG signal, etc.) associated with the biomarker to be measured. The physiological signals to be measured may depend on the sensing technique used, as described herein. Sensor unit 20235 of sensing system 20069 may communicate with data processing and communication unit 20236. In one embodiment, sensor unit 20235 may communicate with data processing and communication unit 20236 using a wireless interface. Data processing and communication unit 20236 may include an analog-to-digital converter (ADC) 20237, a data processing unit 20238, storage 20239, an I / O interface 20241, and an RF transceiver 20240. Data processing unit 20238 may include a processor and a memory unit.

[0107] The sensor unit 20235 may transmit the measured physiological signal to the ADC 20237 of the data processing and communication unit 20236. In one embodiment, the measured physiological signal may pass through one or more filters (e.g., an RC low-pass filter) before being transmitted to the ADC. The ADC may convert the measured physiological signal into measurement data associated with the biomarker. The ADC may pass the measurement data to the data processing unit 20238 for processing. In one embodiment, the data processing unit 20238 may transmit the measurement data associated with the biomarker to a surgical hub or computing device 20243, and then to a cloud computing system 20244 for further processing. The data processing unit may transmit the measurement data to the surgical hub or computing device 20243 using one of the radio protocols as described herein. In one embodiment, the data processing unit 20238 may first process the raw measurement data received from the sensor unit and then transmit the processed measurement data to the surgical hub or computing device 20243.

[0108] In one embodiment, the data processing and communication unit 20236 of the sensing system 20069 may receive thresholds associated with biomarkers for monitoring from the surgical hub, computing device 20243, or directly from the cloud server 20077 of the cloud computing system 20244. The data processing unit 20236 may compare the measured data associated with the monitored biomarker with the corresponding thresholds received from the surgical hub, computing device 20243, or cloud server 20077. The data processing and communication unit 20236 may send a notification message to HID20242 indicating that the measured data value has exceeded the threshold. The notification message may include the measured data associated with the monitored biomarker. The data processing and computing unit 20236 may send the notification via transmission to the surgical hub or computing device 20243 using one of the RF protocols, such as Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), or Wi-Fi. The data processing unit 20238 may send notifications (e.g., notifications about HCPs) directly to a cloud server via transmission to a cellular transmit / receive point (TRP) or base station, using one or more cellular protocols such as GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G. In one embodiment, the sensing unit may communicate with a hub / computing device via a router, as shown in Figures 6A-6C.

[0109] Figure 7C shows an exemplary sensing system 20069 (e.g., a surgical sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20245, a data processing and communication unit 20246, and a human interface device 20242. The sensor unit 20245 may include a sensor 20247 and an analog-to-digital converter (ADC) 20248. The ADC 20248 in the sensor unit 20245 can convert physiological signals measured by the sensor 20247 into measurement data associated with a biomarker. The sensor unit 20245 may transmit the measurement data to the data processing and communication unit 20246 for further processing. In one embodiment, the sensor unit 20245 may transmit the measurement data to the data processing and communication unit 20246 using an inter-integrated circuit (I2C) interface.

[0110] The data processing and communication unit 20246 includes a data processing unit 20249, a storage unit 20250, and an RF transceiver 20251. The sensing system communicates with a surgical hub or computing device 20243, which may then communicate with a cloud computing system 20244. The cloud computing system 20244 may include a remote server 20077 and associated remote storage 20078. The sensor unit 20245 may include one or more in vitro or in vivo sensors to measure one or more biomarkers, as described herein.

[0111] After processing the measurement data received from the sensor unit 20245, the data processing and communication unit 20246 may further process the measurement data and / or transmit it to the smart hub or computing device 20243, as shown in Figure 7B. In one embodiment, the data processing and communication unit 20246 may transmit the measurement data received from the sensor unit 20245 to a remote server 20077 of the cloud computing system 20244 for further processing and / or monitoring.

[0112] Figure 7D shows an exemplary sensing system 20069 (e.g., a surgeon sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20252, a data processing and communication unit 20253, and a human interface system 20261. Sensor unit 20252 may include a plurality of sensors 20254, 20255-20256 to measure one or more physiological signals associated with a patient or surgeon biomarker, and / or one or more physical state signals associated with the physical state of the patient or surgeon. Sensor unit 20252 may also include one or more analog-to-digital converters (ADCs) 20257. The list of biomarkers may include biomarkers such as those disclosed herein. The ADC 20257 in sensor unit 20252 may convert each of the physiological signals and / or physical state signals measured by sensors 20254-20256 into their respective measured data. Sensor unit 20252 may transmit measurement data associated with one or more biomarkers, as well as the physical condition of the patient or surgeon, to data processing and communication unit 20253 for further processing. Sensor unit 20252 may transmit measurement data to data processing and communication unit 20253 individually for each of sensors 1 20254 to N 20256, or in combination for all sensors. In one embodiment, sensor unit 20252 may transmit measurement data to data processing and communication unit 20253 via an I2C interface.

[0113] The data processing and communication unit 20253 may include a data processing unit 20258, a storage unit 20259, and an RF transceiver 20260. The sensing system 20069 can communicate with a surgical hub or computing device 20243, and then communicate with a cloud computing system 20244 comprising at least one remote server 20077 and at least one storage unit 20078. The sensor unit 20252 may include one or more in vitro or in vivo sensors for measuring one or more biomarkers, as described herein.

[0114] Figure 8 shows an example of using surgical task situation awareness and measurement data from one or more surgeon sensing systems to coordinate surgical instrument control. Figure 8 shows an exemplary surgical timeline 20265 and contextual information that the surgical hub may derive from data received from one or more surgical instruments, one or more surgeon sensing systems, and / or one or more environmental sensing systems at each step of the surgical procedure. Instruments that can be controlled by the surgical hub may include high-energy instruments, endocutter clamps, etc. The surgeon sensing system may include sensing systems for measuring one or more biomarkers associated with the surgeon, e.g., heart rate, sweat composition, respiratory rate, etc. The environmental sensing system may include systems for measuring one or more environmental attributes, e.g., cameras for detecting the surgeon's position / movement / breathing patterns, spatial microphones for measuring ambient noise and / or the tone of the healthcare provider's voice in the operating room, ambient temperature / humidity, etc.

[0115] In the following description of the timeline 20265 shown in Figure 8, please also refer to Figure 5. Figure 5 provides various components used in surgical procedures. Timeline 20265 shows steps that may be performed individually and / or collectively by nurses, surgeons, and other healthcare professionals during the course of an exemplary colorectal surgery. In a colorectal surgical procedure, the situation-aware surgical hub 20076 may receive data from various data sources throughout the course of the surgery, including data generated each time a healthcare provider (HCP) uses a modular device / instrument 20095 paired with the surgical hub 20076. The surgical hub may receive this data from the paired modular device 20095. The surgical hub may receive measured data from the sensing system 20069. The surgical hub may continuously derive the surgeon's stress level and inferences (i.e., contextual information) regarding the procedure in progress as new data is received, using data from modular device / instrument 20095 and / or measurement data from sensing system 20069, so that the surgeon's stress level for the steps of the procedure being performed is obtained. The situation awareness system of the surgical hub 20076 may perform one or more of the following: record data regarding the procedure to generate a report; verify the steps being performed by the healthcare professional; provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen); adjust the modular device based on the situation (e.g., activate a monitor, adjust the FOV of a medical imaging device, change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument); or perform any other such actions described herein. In one embodiment, these steps may be performed by a remote server 20077 of cloud system 20064, which communicates with the surgical hub 20076.

[0116] As a first step (not shown in Figure 8 for brevity), hospital staff may search the hospital's EMR database for the patient's EMR. Based on the patient data selected in the EMR, the surgical hub 20076 may determine that the procedure to be performed is a thoracic surgery. Staff may scan the medical supplies coming in for the procedure. The surgical hub 20076 may cross-reference the scanned supplies with a list of supplies that can be used in various types of procedures and confirm that the combination of supplies matches a thoracic procedure. The surgical hub 20076 may pair each of the sensing systems 20069 fitted by various HCPs.

[0117] Once all devices are prepared and preoperative preparations are complete, the surgical team may begin by making an incision and positioning the trocar. The surgical team may perform access and preparation by dissecting any adhesions and identifying the inferior mesenteric artery (IMA) branch. The surgical hub 20076 may infer that the surgeon is in the process of dissecting adhesions, at least based on data that can be received from an RF or ultrasound generator indicating that an energy instrument is being emitted. The surgical hub 20076 may cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (e.g., after the completion of the surgical steps described above) corresponds to the dissection step.

[0118] Following dissection, the HCP may proceed to the surgical ligation step (e.g., indicated by A1). As shown in Figure 8, the HCP may begin by ligating the IMA. The surgical hub 20076 may receive data from the high-energy jaw device and / or endocutter indicating that the instrument is being fired, thus allowing it to infer that the surgeon is ligating the arteries and veins. The surgical hub may also receive measured data from one of the HCP's sensing systems indicating a high stress level in the HCP (e.g., indicated by the B1 mark on the time axis). For example, a high stress level may be indicated by a change in the HCP's heart rate from a baseline. The surgical hub 20076 may derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps retrieved in the process (e.g., as shown by A2 and A3), as in the previous step. During the high-stress period, the surgical hub 20076 may monitor the high-energy jaw trigger ratio, as well as the endocutter clamp and firing rate. In one embodiment, the surgical hub 20076 may transmit auxiliary control signals to the high-energy jaw device and / or end-cutter device to control the device in operation. The surgical hub may transmit auxiliary signals based on the stress level of the HCP operating the surgical device and / or the situational awareness known to the surgical hub. For example, the surgical hub 20076 may transmit control auxiliary signals to the high-energy device or end-cutter clamp as shown by A2 and A3 in Figure 8.

[0119] The HCP may proceed to the next step of releasing the upper sigmoid colon, followed by the descending colon, rectum, and sigmoid colon. The surgical hub 20076 may continue to monitor the HCP's high-stress markers (e.g., as indicated by D1, E1a, E1b, F1). During periods of high stress, the surgical hub 20076 may transmit auxiliary signals to the high-energy jaw device and / or endocutter device, as shown in Figure 8.

[0120] After mobilizing the colon, the HCP may proceed to the segmental resection portion of the surgery. For example, the surgical hub 20076 may infer that the HCP is transversely incising the intestine and removing the sigmoid colon, based on data from surgical stapling and cutting instruments, including data from its cartridge. The cartridge data may correspond, for example, to the size or type of staples fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data may indicate the type of tissue being stapled and / or transversely incised. Note that, since various instruments are well suited to specific tasks, surgeons should periodically switch between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasound) instruments depending on the step in the procedure. Thus, the sequence in which stapling / cutting instruments and surgical energy instruments are used may indicate which step of the procedure the surgeon is performing.

[0121] The surgical hub may determine and transmit control signals to the surgical device based on the stress level of the HCP. For example, during period G1b, control signal G2b may be transmitted to the endocutter clamp. Once the sigmoid colon is removed, the incision may be closed and the postoperative portion of the procedure may begin. The patient may be awakened from anesthesia. The surgical hub 20076 may infer that the patient is awake from anesthesia based on one or more sensing systems attached to the patient.

[0122] Figure 9 is a block diagram of a computer-implemented bidirectional surgical system with surgeon / patient monitoring according to at least one aspect of the present disclosure. In one aspect, the computer-implemented bidirectional surgical system may be configured to monitor surgeon biomarkers and / or patient biomarkers using one or more sensing systems 20069. Surgeon biomarkers and / or patient biomarkers may be measured before, after, and / or during surgery. In one aspect, the computer-implemented bidirectional surgical system may be configured to monitor and analyze data on the operation of various surgical systems 20069, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented bidirectional surgical system may include a cloud-based analysis system. The cloud-based analysis system may include one or more analysis servers.

[0123] As shown in Figure 9, the cloud-based monitoring and analysis system may comprise a plurality of sensing systems 20268 (which may be the same as or similar to sensing system 20069), surgical instruments 20266 (which may be the same as or similar to instruments 20031), a plurality of surgical hubs 20270 (which may be the same as or similar to hub 20006), and a surgical data network 20269 (which may be the same as or similar to the surgical data network described in Figure 4) for connecting the surgical hubs 20270 to a cloud 20271 (which may be the same as or similar to cloud computing system 20064). Each of the plurality of surgical hubs 20270 may be communicatively connected to one or more surgical instruments 20266. Each of the plurality of surgical hubs 20270 may also be communicatively connected to one or more sensing systems 20268 and to the cloud 20271 of the computer-implemented bidirectional surgical system via the network 20269. The surgical hub 20270 and the sensing system 20268 may be connected in a communicative manner using a wireless protocol as described herein. The cloud system 20271 may be a remote centralized source of hardware and software for storing, processing, manipulating, and communicating measurement data from the sensing system 20268 and data generated based on the operation of various surgical systems 20268.

[0124] As shown in Figure 9, access to the cloud system 20271 is achieved via network 20269, which may be the Internet or another suitable computer network. A surgical hub 20270, which can be connected to the cloud system 20271, may be considered the client side of the cloud computing system (e.g., a cloud-based analytics system). Surgical instruments 20266 may be paired with the surgical hub 20270 for the control and execution of various surgical procedures or actions, as described herein. Sensing systems 20268 may be paired with the surgical hub 20270 for surgeon monitoring during surgery, pre-operative patient monitoring, intra-operative patient monitoring, or post-operative patient monitoring of surgeon-related biomarkers, in order to track and / or measure various milestones and / or detect various complications. Environmental sensing systems 20267 may be paired with the surgical hub 20270 to measure environmental attributes associated with the surgeon or patient for surgeon monitoring, pre-operative patient monitoring, intra-operative patient monitoring, or post-operative patient monitoring.

[0125] The surgical instruments 20266, the environmental sensing system 20267, and the sensing system 20268 may be equipped with wired or wireless transceivers for data transmission to and from their corresponding surgical hubs 20270 (which may also be equipped with transceivers). One or more combinations of the surgical instruments 20266, the sensing system 20268, or the surgical hubs 20270 may indicate a specific location within a medical facility (e.g., a hospital), such as an operating room, an intensive care unit (ICU), or a recovery room, for the purpose of providing medical surgery, pre-operative preparation, and / or post-operative recovery. For example, the memory of the surgical hub 20270 may store location data.

[0126] As shown in Figure 9, the cloud system 20271 may include one or more central servers 20272 (which may be the same as or similar to the remote servers 20067), a surgical hub application server 20276, a data analysis module 20277, and an input / output ("I / O") interface 20278. The central servers 20272 of the cloud system 20271 can collectively manage the cloud computing system, including monitoring requests from client surgical hubs 20270 and managing the processing power of the cloud system 20271 to perform those requests. Each of the central servers 20272 may comprise one or more processors 20273 coupled to a preferred memory device 20274, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage devices. Memory device 20274 is equipped with machine-executable instructions, which, when executed, can cause processor 20273 to execute data analysis module 20277 for cloud-based data analysis, real-time monitoring of measurement data received from sensing system 20268, and for operations, recommendations, and other operations as described herein. Processor 20273 may execute data analysis module 20277 independently or in conjunction with hub applications that run independently by hub 20270. Central server 20272 may also be equipped with a database 20275 of aggregated medical data that may reside in memory 20274.

[0127] Based on connections to various surgical hubs 20270 via network 20269, cloud 20271 can aggregate data from specific data generated by various surgical instruments 20266, and / or monitor real-time data from sensing systems 20268 and surgical hubs 20270 associated with surgical instruments 20266 and / or sensing systems 20268. Such aggregated data from surgical instruments 20266, and / or measurement data from sensing systems 20268, can be stored in the aggregated medical database 20275 of cloud 20271. In particular, cloud 20271 can advantageously track real-time measurement data from sensing systems 20268, and / or perform data analysis and actions on the measurement data and / or aggregated data to gain insights, and / or perform functions that individual hubs 20270 could not achieve on their own. For this purpose, cloud 20271 and surgical hubs 20270 are communicably linked to send and receive information, as shown in Figure 9. The I / O interface 20278 connects to multiple surgical hubs 20270 via network 20269. In this way, the I / O interface 20278 can be configured to transfer information between the surgical hubs 20270 and the aggregated medical data database 20275. Thus, the I / O interface 20278 can facilitate read / write operations of a cloud-based analysis system. Such read / write operations may be performed in response to requests from the hubs 20270. These requests may be sent to the surgical hubs 20270 via a hub application. The I / O interface 20278 may include one or more high-speed data ports, including a Universal Serial Bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 20271 to the surgical hubs 20270. The hub application server 20276 of the cloud 20271 may be configured to host and supply shared functions to a software application (e.g., a hub application) run by the surgical hubs 20270.For example, the hub application server 20276 may manage requests made by the hub application through hub 20270, control access to the aggregated medical data database 20275, and perform load balancing.

[0128] The cloud computing system configurations described in this disclosure may be designed to address various problems arising in the context of medical procedures (e.g., preoperative, intraoperative, and postoperative monitoring) and treatments performed using medical devices such as surgical instruments 20266, 20031. In particular, surgical instrument 20266 may be a digital surgical device configured to interact with cloud 20271 in order to implement techniques for improving surgical outcomes. Sensing system 20268 may be a system having one or more sensors configured to measure one or more biomarkers associated with a surgeon performing a medical procedure and / or a patient in whom a medical surgery is scheduled, is being performed, or has been performed. Various surgical instruments 20266, sensing system 20268, and / or surgical hub 20270 may include a human interface system (e.g., having a touch-controlled user interface) so that a clinician and / or patient can control the manner of interaction between surgical instrument 20266 or sensing system 20268 and cloud 20271. Other suitable user interfaces for control, such as auditory-controlled user interfaces, may also be used.

[0129] The cloud computing system configurations described herein may be designed to address various issues arising in situations where sensing system 20268 is used to monitor one or more biomarkers associated with a healthcare professional (HCP) or patient during pre-operative, intra-operative, and post-operative procedures. Sensing system 20268 may be a surgeon sensing system or a patient sensing system configured to interact with a surgical hub 20270 and / or cloud system 20271 to implement techniques for monitoring surgeon biomarkers and / or patient biomarkers. Various sensing systems 20268 and / or surgical hub 20270 may include a touch-controlled human interface system so that the HCP or patient can control the manner of interaction between sensing system 20268 and surgical hub 20270 and / or cloud system 20271. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0130] Figure 10 shows an exemplary surgical system 20280 according to the present disclosure, which may include a surgical instrument 20282 capable of communicating with a console 20294 or a portable device 20296 via a wired or wireless connection through a local area network 20292 or a cloud network 20293. In various embodiments, the console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 so that the adapter 20285 transmits force from the drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) located inside for measuring the force acting on the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may also be a bio-implant, i.e., a multi-fire loading unit (MFLU), which allows a clinician to fire multiple fasteners multiple times without having to remove the loading unit 20287 from the surgical site in order to reload the loading unit 20287.

[0131] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) that can be fired two or more times before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener as it is ejected from the multi-fire fastener cartridge.

[0132] The handle 20297 may include a motor connected to the drive shaft so as to act on the rotation of the drive shaft. The handle 20297 may include a control interface for selectively operating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces, which can be engaged by a clinician to start the motor.

[0133] The control interface of the handle 20297 communicates with the controller 20298 of the handle 20297 to selectively activate the motor and act on the rotation of the drive shaft. The controller 20298 may be located within the handle 20297 and may be configured to receive input from the control interface and adapter data from adapter 20285 or loading unit data from loading unit 20287. The controller 20298 may analyze the input from the control interface and the data received from adapter 20285 and / or loading unit 20287 in order to selectively activate the motor. The handle 20297 may also include a display that can be viewed by a clinician while the handle 20297 is in use. The display may be configured to show portions of the adapter or loading unit data before, during, or after firing the instrument 20282.

[0134] Adapter 20285 may include an internally located adapter identification device 20284, and loading unit 20287 may include an internally located loading unit identification device 20288. Adapter identification device 20284 communicates with controller 20298, and loading unit identification device 20288 may communicate with controller 20298. It will be understood that loading unit identification device 20288 may communicate with adapter identification device 20284, which relays or passes through communication from loading unit identification device 20288 to controller 20298.

[0135] Adapter 20285 may also include a plurality of sensors 20286 (one shown) positioned around it to detect various states of Adapter 20285 or the environment (e.g., whether Adapter 20285 is connected to a loading unit, whether Adapter 20285 is connected to a handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside Adapter 20285, the number of times Adapter 20285 has fired, the peak force of Adapter 20285 during firing, the total amount of force applied to Adapter 20285, the peak recoil force of Adapter 20285, the number of pauses of Adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to Adapter Identifier 20284 in the form of data signals. The data signals from the plurality of sensors 20286 may be stored in Adapter Identifier 20284 or used to update Adapter data stored in Adapter Identifier 20284. The data signals from the plurality of sensors 20286 may be analog or digital. Multiple sensors 20286 may include force gauges for measuring the force exerted on the loading unit 20287 during firing.

[0136] The handle 20297 and adapter 20285 may be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals between them). Additionally or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductively). It is also intended that the adapter identification device 20284 and the controller 20298 may be able to wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0137] Handle 20297 may include transceiver 20283 configured to transmit instrument data from controller 20298 to other components of system 20280 (e.g., LAN 20292, cloud 20293, console 20294, or portable device 20296). Controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to surgical hub 20270, as illustrated in Figure 9. Transceiver 20283 may receive data from surgical hub 20270 (e.g., cartridge data, loading unit data, adapter data, or other notifications). Transceiver 20283 may receive data from other components of system 20280 (e.g., cartridge data, loading unit data, or adapter data). For example, controller 20298 may transmit instrument data to console 20294, including the serial number of a mounting adapter attached to handle 20297 (e.g., adapter 20285), the serial number of a loading unit attached to adapter 20285 (e.g., loading unit 20287), and the serial number of a multi-shot fastener cartridge loaded into the loading unit. Console 20294 may then reply to controller 20298 with data associated with the mounted cartridge, loading unit, and adapter, respectively (e.g., cartridge data, loading unit data, or adapter data). Controller 20298 may display the message on the local instrument display, or transmit the message via transceiver 20283 to console 20294 or portable device 20296, which will then display the message on display 20295 or the portable device screen, respectively.

[0138] Figures 11A to 11D illustrate embodiments of wearable sensing systems, such as surgical sensing systems or patient sensing systems. Figure 11A is an example of a glasses-based sensing system 20300 that may be based on an electrochemical sensing platform. The sensing system 20300 may monitor sweat electrolytes and / or metabolites (e.g., in real time) using multiple sensors 20304 and 20305 that are in contact with the skin of the surgeon or patient. For example, the sensing system 20300 may use an ampereometry-based biosensor 20304 and / or a potentiometry-based biosensor 20305 integrated into the nose bridge pad of the glasses 20302 to measure current and / or voltage.

[0139] The current-measuring biosensor 20304 can be used to measure sweat lactate levels (e.g., in mmol / L units). Lactate is a product of lactic acidosis, which can occur due to decreased tissue oxygenation that may be caused by sepsis or hemorrhage. The patient's lactate level (e.g., >2 mmol / L) may be used to monitor the development of sepsis, for example, during postoperative monitoring. The potentiometric biosensor 20305 can be used to measure potassium levels in a patient's sweat. A voltage follower circuit with an operational amplifier can be used to measure the potential signal between a reference electrode and a working electrode. The output of the voltage follower circuit can be filtered and converted to a digital value using an ADC.

[0140] Current measuring sensor 20304 and potentiometer measuring sensor 20305 may be connected to circuit 20303 located on each arm of the eyeglasses. Electrochemical sensors may be used for simultaneous real-time monitoring of sweat lactic acid and potassium levels. Electrochemical sensors may be screen-printed on stickers and placed on each side of the nose pads of the eyeglasses to monitor sweat metabolites and electrolytes. Electronic circuit 20303 located on the arms of the eyeglass frame may include a wireless data transceiver (e.g., a low-energy Bluetooth transceiver) which can be used to transmit lactic acid and / or potassium measurement data to a surgical hub or intermediate device, which can then transfer the measurement data to the surgical hub. The eyeglass-based sensing system 20300 may also use, for example, a signal conditioning unit for filtering and amplifying the electrical signals generated from the electrochemical sensors 20305 or 20304, a microcontroller for digitizing the analog signals, and a wireless (e.g., low-energy Bluetooth) module for transferring the data to a surgical hub or computing device, as shown in Figures 7B-7D.

[0141] Figure 11B shows an example of a wristband-type sensing system 20310 comprising a sensor assembly 20312 (e.g., a photoplethysmography (PPG)-based sensor assembly or an electrocardiogram (ECG)-based sensor assembly). For example, in the sensing system 20310, the sensor assembly 20312 may collect and analyze arterial pulses at the wrist. The sensor assembly 20312 may be used to measure one or more biomarkers (e.g., heart rate, heart rate variability (HRV)). In the sensing system having a PPG-based sensor assembly 20312, light (e.g., green light) can pass through the skin. A certain percentage of the green light may be absorbed by blood vessels, and some of the green light may be reflected and detected by a photodetector. These differences or reflections are associated with variations in tissue blood perfusion, and these variations may be used to detect cardiovascular information (e.g., heart rate). For example, the amount absorbed may vary depending on the blood volume. The sensing system 20310 may determine the heart rate by measuring light reflectance as a function of time. The heart rate recurrence (HRV) may be determined as the interval between the steepest signal gradients before the peak (e.g., the standard deviation), known as the heart rate interval (IBI).

[0142] In the case of a sensing system having an ECG-based sensor assembly 20312, a set of electrodes may be placed in contact with the skin. The sensing system 20310 may measure the voltage across the set of electrodes placed on the skin to determine the heart rate. In this case, the HRV may be measured as the period variation (e.g., standard deviation) between R peaks in a group of QRS complexes, known as the RR interval.

[0143] The sensing system 20310 may, for example, use a signal conditioning unit for filtering and amplifying the analog PPG signal, a microcontroller for digitizing the analog PPG signal, and a wireless (e.g., Bluetooth) module for transferring the data to a surgical hub or computing device, as shown in Figures 7B to 7D.

[0144] Figure 11C shows an exemplary ring sensing system 20320. The ring sensing system 20320 may include a sensor assembly (e.g., a heart rate sensor assembly) 20322. The sensor assembly 20322 may include a light source (e.g., a red or green light-emitting diode (LED)) and a photodiode for detecting reflected and / or absorbed light. The LED in the sensor assembly 20322 may measure heart rate and / or blood oxygen levels by illuminating the finger and the photodiode in the sensor assembly 20322 by detecting changes in blood volume. The ring sensing system 20320 may also include other sensor assemblies for measuring other biomarkers, e.g., a thermistor or infrared thermometer for measuring body surface temperature. The ring sensing system 20320 may, for example, use a signal conditioning unit for filtering and amplifying the analog PPG signal, a microcontroller for digitizing the analog PPG signal, and a wireless (e.g., low-energy Bluetooth) module for transferring the data to a surgical hub or computing device, as shown in Figures 7B to 7D.

[0145] Figure 11D shows an example of an electroencephalogram (EEG) sensing system 20315. As shown in Figure 11D, the sensing system 20315 may include one or more EEG sensor units 20317. The EEG sensor unit 20317 may include a plurality of conductive electrodes positioned in contact with the scalp. The conductive electrodes may be used to measure small potentials that may occur outside the head due to neuronal activity in the brain. The EEG sensing system 20315 may measure biomarkers, such as delirium, by identifying specific brain patterns, such as deceleration or omission of post-dominant rhythms and loss of responsiveness to eye opening and closing. The EEG sensing system 20315 may have, for example, a signal conditioning unit for filtering and amplifying potentials, a microcontroller for digitizing electrical signals, and a wireless (e.g., low-energy Bluetooth) module for transferring data to a smart device, as shown in Figures 7B to 7D.

[0146] Figure 12 shows a block diagram of a computer-implemented patient / surgeon monitoring system 20325 for monitoring one or more patient or surgeon biomarkers before, during, and / or after surgical procedures. As shown in Figure 12, one or more sensing systems 20336 may be used to measure and monitor patient biomarkers, for example, to facilitate patient preparation before surgery and recovery after surgery. Sensing systems 20336 may also be used to measure and monitor surgeon biomarkers in real time to assist surgical tasks by, for example, communicating relevant biomarkers (e.g., surgeon biomarkers) to a surgical hub 20326 and / or surgical device 20337 and adjusting their functions. Adjustable surgical device functions may include power level, forward speed, closing speed, load, waiting time, or other tissue-dependent operating parameters. Sensing systems 20336 may also measure one or more physical attributes associated with the surgeon or patient. Patient biomarkers and / or physical attributes may be measured in real time.

[0147] The computer-implemented wearable patient / surgeon wearable sensing system 20325 may include a surgical hub 20326, one or more sensing systems 20336, and one or more surgical devices 20337. The sensing systems and surgical devices may be communicatively connected to the surgical hub 20326. For example, one or more analysis servers 20338, which are part of an analysis system, may also be communicatively connected to the surgical hub 20326. Although a single surgical hub 20326 is shown, it should be noted that the wearable patient / surgeon wearable sensing system 20325 may include any number of surgical hubs 20326 that can be connected to form a network of surgical hubs 20326 communicatively connected to one or more analysis servers 20338, as described herein.

[0148] In one embodiment, the surgical hub 20326 may be a computing device. The computing device may be a personal computer, laptop, tablet, smart mobile device, etc. In one embodiment, the computing device may be a client computing device of a cloud-based computing system. The client computing device may be a thin client.

[0149] In one embodiment, the surgical hub 20326 may include a processor 20327 coupled to memory 20330 for executing stored instructions, storage 20331 for storing one or more databases such as an EMR database, and a data relay interface 20329 to which data is transmitted to an analysis server 20338. In one embodiment, the surgical hub 20326 may further include an I / O interface 20333 having an input device 20341 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user, and an output device 20335 (e.g., a display screen) for providing output to the user. In one embodiment, the input device and the output device may be a single device. The output may include data from queries entered by the user, suggestions for products or product configurations for use in a given procedure, and / or instructions for actions performed before, during, or after a surgical procedure. The surgical hub 20326 may include an instrument interface 20332 for communicatingly connecting surgical instruments 20337 to the surgical hub 20326. In one embodiment, the instrument interface 20332 may include a transceiver that allows one or more surgical instruments 20337 to connect to the surgical hub 20326 via a wired or wireless interface using one of the wired or wireless wireless protocols described herein. The surgical instruments 20337 may include, for example, a motorized stapler, an energy device or its generator, an imaging system, or other connected systems, such as a fume ventilator, a suction-irrigation device, or a pneumoperitoneum system.

[0150] In one embodiment, the surgical hub 20326 may be communicatively connected to one or more surgeon and / or patient sensing systems 20336. The sensing systems 20336 may be used to measure and / or monitor in real time various biomarkers associated with the surgeon performing the surgery or the patient undergoing the surgery. A list of patient / surgeon biomarkers measured by the sensing systems 20336 is provided herein. In one embodiment, the surgical hub 20326 may be communicatively connected to an environmental sensing system 20334. The environmental sensing system 20334 may be used to measure and / or monitor in real time environmental attributes, such as temperature / humidity in the operating room, surgeon movements, and ambient noise in the operating room caused by the breathing patterns of the surgeon and / or patient.

[0151] When the sensing system 20336 and the surgical device 20337 are connected to the surgical hub 20326, the surgical hub 20326 may receive from the sensing system 20336, for example, measurement data associated with one or more patient biomarkers, physical status associated with the patient, measurement data associated with the surgeon biomarker, and / or physical status associated with the surgeon, as shown in Figures 7B-7D. The surgical hub 20326 may, for example, associate the measurement data associated with the surgeon with other relevant pre-operative data and / or data from the situation awareness system to generate control signals for controlling the surgical device 20337, as shown in Figure 8.

[0152] In one embodiment, the surgical hub 20326 may compare the measurement data from the sensing system 20336 with one or more thresholds defined based on baseline values, pre-operative measurement data, and / or intra-operative measurement data. The surgical hub 20326 can compare the measurement data from the sensing system 20336 with one or more thresholds in real time. The surgical hub 20326 may generate notifications for display. For example, if the measurement data crosses a defined threshold (e.g., is greater than or less than), the surgical hub 20326 may send a notification to the patient human interface system 20339 and / or the surgeon or HCP human interface system 20340 for distribution. The determination of whether or not a notification is sent to one or more of the patient human interface system 20339 and / or the HCP human interface system 2340 may be based on the severity level associated with the notification. The surgical hub 20326 may also generate severity levels associated with the notification for display. The generated severity level may be displayed to the patient and / or the surgeon or HCP. In one embodiment, patient biomarkers that are measured and / or monitored (e.g., measured and / or monitored in real time) may be associated with surgical steps. For example, biomarkers measured and monitored for the vein and artery transection step of thoracic surgery may include blood pressure, tissue perfusion pressure, edema, arteriosclerosis, collagen content, connective tissue thickness, etc., and biomarkers measured and monitored for the lymph node dissection step of surgery may include monitoring of the patient's blood pressure. In one embodiment, data on postoperative complications may be retrieved from an EMR database in storage 20331, and data on staple or incision line leakage may be directly detected or inferred by the situational awareness system. Surgical procedure outcome data may be inferred by the situational awareness system from data received from various data sources, including the surgical device 20337, the sensing system 20336, and a database in storage 20331 to which the surgical hub 20326 is connected.

[0153] The surgical hub 20326 may transmit measurement data and physical condition data received from the sensing system 20336, and / or data associated with the surgical device 20337, to the analysis server 20338 for processing. Each of the analysis servers 20338 may include memory and a memory-coupled processor capable of executing stored instructions for analyzing the received data. The analysis servers 20338 may be connected via a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 20338 may determine optimal and / or preferred operating parameters for various types of modular devices, generate adjustments to the control program for the surgical device 20337, and transmit (or "push") the updated or controlled program to one or more surgical devices 20337. For example, the analysis system 20338 may correlate perioperative data received from the surgical hub 20236 with measurement data associated with the physiological state of the surgeon or HCP and / or the patient's physiological state. The analysis system 20338 may determine when the surgical device 20337 should be controlled and send updates to the surgical hub 20326. The surgical hub 20326 may then transfer the control program to the relevant surgical device 20337.

[0154] Further details regarding the computer-implemented wearable patient / surgeon wearable sensing system 20325, which includes a surgical hub 30326, one or more sensing systems 20336, and various surgical devices 20337 that can be connected thereto, are described in relation to Figures 5 to 7D.

[0155] Figure 13 shows an example of display 29500. Healthcare professionals (HCPs) may be interested in data related to surgical procedures and postoperative recovery. Display 29500 is an example of how data related to surgical procedures and postoperative recovery can be communicated to HCPs. The data may relate to biomarkers, the environment, surgical instruments, and / or high-energy devices. Contextual information related to surgical procedures and postoperative recovery may be determined based on biomarkers, the environment, surgical instruments, high-energy devices, etc. Contextual information may include information about physiological systems and conditions.

[0156] For example, biomarkers may be associated with various physiological systems. HCPs may monitor biomarkers to determine contextual information regarding surgical procedures and postoperative recovery. Biomarker sensing systems may sense biomarkers of patients and / or healthcare professionals. The biomarker sensing systems described herein may sense a variety of biomarkers, including but not limited to sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, hydrogen blood potential, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, mental state, sweat, circulating tumor cells, autonomic tension, circadian rhythm, and / or menstrual cycle.

[0157] For example, environmental data may be related to various physiological systems. Environmental data may include temperature, air quality, airborne chemicals, and light exposure.

[0158] HCPs may be interested in monitoring surgical instrument data during surgical procedures. Surgical instrument data can provide contextual information during surgery. Contextual information may be determined based on other sensing system data. The sensing systems described herein can sense parameters associated with surgical tools. Surgical tools may include surgical staplers.

[0159] Surgical sensing systems may be configured to sense data related to surgical procedures and postoperative recovery. Surgical sensing systems may include systems configured to sense data related to biomarkers, the environment, surgical instruments, and / or high-energy devices. Data related to various biomarkers may include patient or HCP biomarkers.

[0160] A surgical clinical practitioner (HCP) may be interested in analyzing data received from a surgical sensing system in relation to other data received from a separate surgical sensing system. Measurements of biomarkers, the environment, and surgical instruments can provide context to each other. Therefore, it is important for the HCP to know whether measurements from the surgical sensing systems are being taken simultaneously or at different times that are offset from each other. If the data received from each sensing system is not synchronized with each other, the HCP may not be able to determine contextual information from multiple sensing systems.

[0161] An exemplary surgical sensing system may include a clock used to tag events with sensing system time (e.g., time relative to the surgical sensing system clock). The surgical sensing system clock may be a real-time clock (RTC). The RTC may be an integrated circuit. The RTC may measure the passage of time. The RTC may include an internal oscillator with an external crystal and / or external frequency reference. The RTC may measure the passage of time based on the oscillator frequency. The RTC may be set using a reference clock time. In one embodiment, the RTC may be a clock that minimizes time inaccuracies such as drift, variability, and latency differences.

[0162] For example, clock drift may occur over time. Clock drift may include cases where the RTC becomes asynchronous with a reference clock time. Clock drift occurs when the RTC operates at a different rate than the reference clock. Due to clock drift, the RTC may drift and read different times. In one embodiment, one or more surgical sensing systems may include an RTC set to a shared reference clock time. Over time, one or more surgical sensing system RTCs may become asynchronous with the reference clock, and each may read different times from the other. In one embodiment, stream incompleteness may occur using a non-real-time operating system and commodity hardware. Sensors capturing at the same frequency on various hosts may have various data rates. The various data rates may be due to differences in hardware configuration. The various data rates may be due to local clock offset and drift. The various data rates may be due to uninterruptible kernel time. The various data rates may be due to scheduler delay. The various data rates may be due to unpredictable disk operation.

[0163] HCP may consider data from multiple surgical sensing systems, for example, to determine contextual information regarding surgical procedures and postoperative recovery. The relative occurrence of specific measurements from each of the multiple systems, and / or the accuracy of that assessment, may influence such considerations.

[0164] For illustrative purposes, display 29500 includes a visual representation 29502 of a data signal. The visual representation 29502 may include current measurement readings 29504 and a graphical representation 29506. The data signal may be received from various surgical sensing systems, including those related to biomarkers, the environment, surgical instruments, and / or high-energy devices. The received data signal may include measurements from the surgical sensing system. In one embodiment, the received data signal may include heart rate and / or blood pressure measurements.

[0165] The visual representation 29502 may include the current measurement reading 29504 based on the most recently received measurements from the surgical sensing system. The visual representation 29502 may include a graphical representation 29506 of the measurement reading 29504 based on hub clock time. The graphical representation 29506 may include a recent history of the measurement reading 29504. As shown in Figure 13, the measurements may include heart rate and blood pressure.

[0166] In one embodiment, the heart rate sensing system and the blood pressure sensing system may transmit data signals to the display 29500. The data signals may be displayed as a graphic representation 29506. The heart rate sensing system may transmit a heart rate measurement showing a spike at a first time 29508 based on the heart rate sensing system clock. The blood pressure sensing system may transmit a blood pressure measurement showing a spike at a first time 29508 based on the blood pressure sensing system clock. The heart rate sensing system clock and the blood pressure sensing system clock may be set to different reference clocks. Clocks set to different reference clocks may display different clock times. The graphical representation may indicate that a heart rate spike occurred at the same time as a blood pressure spike at a first time 29508, although the spikes may not actually occur simultaneously.

[0167] In one embodiment, both heart rate and blood pressure measurements may spike simultaneously. If the heart rate sensing system and blood pressure sensing system clocks are not set to a common reference clock, the measurements may appear at different times in the visual representation 29502. For example, both measurements may be taken simultaneously, even though the heart rate measurement may show a spike at a second time 29510 and the blood pressure measurement may show a spike at a different third time 29512. The difference between the second time 29510 and the third time 29512 may result from different latency. Latency may include processing delays in the surgical sensing system and / or transit time when transmitting the measurements to the display 29500. The two measurements may be offset even though they are taken simultaneously.

[0168] The healthcare coordinator (HCP) may use visual representations 29502 to guide treatment and decision-making. It is important for the HCP to know whether measurements from the surgical sensing system are being taken simultaneously or asynchronously. Simultaneous measurements may indicate to the HCP that the measurements are related to each other. For example, the HCP may determine specific contextual information based on simultaneously spiking measurements for heart rate and blood pressure. Asynchronous measurements may indicate to the HCP that the measurements are unrelated to each other. For example, the HCP may determine that the measurements are unrelated when one measurement spikes while the other remains constant.

[0169] Figure 14A is a functional block diagram of the surgical data sequencing system 29514. Such a system can be used, for example, to synchronize the relative timing from multiple data feeds. Synchronizing multiple data feeds together enables correlation and monitoring of multiple systems.

[0170] For example, synchronization may include ad-hoc synchronization of data from a multi-link adjustment sensing system. A surgical hub may receive data signals. Data signals may be from at least two separate data feeds. Data signals may be integrated. Data signals may be linked to procedure data. Data signals may be linked to instrument operation. Data signals may be integrated and linked to create an integrated fused display. The HCP may use the integrated fused display.

[0171] As shown in Figure 14A, the surgical data ordering system 29514 may be used to order received data. The surgical data ordering system 29514 may be used to order received data based on the master time clock 29516. The surgical data ordering system 29514 may include one or more surgical sensing systems 29518, a surgical hub 29520, and a downstream data system 29522. The surgical data ordering system 29514 may be configured to receive data signals from at least one surgical sensing system 29518. The surgical data ordering system 29514 may include a surgical hub 29520. The surgical hub 29520 may communicate with one or more surgical sensing systems 29518. The surgical hub 29520 may be configured to order data signals received from at least one surgical sensing system 29518. The surgical data sequencing system 29514 may include at least one downstream data system 29522. The at least one downstream data system 29522 may include a display.

[0172] The surgical hub 29520 may include computing hardware and / or software suitable for processing and / or sequencing sensor data. For example, the surgical hub 29520 may incorporate and / or be incorporated into the surgical hub 20006 disclosed herein. For example, the surgical hub 29520 may be deployed as a standalone unit for sensor processing. The surgical hub 29520 may be configured to collect measurement data from one or more surgical sensing systems 29518.

[0173] The surgical hub 29520 may be configured to send notifications or requests to one or more surgical sensing systems 29520. The surgical hub 29520 may be configured to send information to at least one downstream data system 29522. The surgical hub 29520 may acquire latency values. Latency values ​​may be associated with surgical sensing systems. Latency values ​​may be associated with communication interfaces. Latency values ​​may be associated with combinations of surgical sensing systems and communication interfaces. The surgical hub 29520 may apply time codes to the received measurements. The surgical hub 29520 may apply time codes to the received measurements based on the master time clock 29516. The surgical hub 29520 may output one or more of the received measurements. The surgical hub 29520 may order the outputs.

[0174] The surgical hub 29520 may include a processing unit 29524. The surgical hub 29520 may include a master time clock 29516. The surgical hub 29520 may include a combination of the processing unit 29524 and the master time clock 29516. The surgical hub 29520 may use the processing unit 29524 to order the received data signals. The surgical hub 29520 may order the received data signals based on the master time clock 29516. The surgical hub 29520 may use a combination of the processing unit 29524 and the master time clock 29516 to order the received data signals. The surgical hub may include a master time log 29526. The master time log may be a data structure in memory. The surgical hub 29520 may store the ordered data signals in the master time log 29526.

[0175] Processing unit 29524 may receive one or more measurements. Processing unit 29524 may acquire latency values. Latency values ​​may be associated with a surgical sensing system. Latency values ​​may be associated with a communication interface. Latency values ​​may be associated with a combination of a surgical sensing system and a communication interface. Processing unit 29524 may apply the respective time codes to the received measurements. Processing unit 29524 may apply the respective time codes to the received measurements based on the master time clock 29516. Processing unit 29524 may output one or more of the received measurements. Processing unit 29524 may order the outputs. Processing unit 29524 may store the outputs in the master time log 29526. Processing unit 29524 may send a request to one or more surgical sensing systems 28518. Processing unit 29524 may send a request to one or more surgical sensing systems 28518 to return a data signal.

[0176] The surgical hub 29520 may include a master time clock 29516. The computing master time clock 29516 may include any electrical and / or computing resources suitable for measuring time. For example, the master time clock 29516 may measure the passage of time. The master time clock 29516 may measure time as a real-time clock (RTC). The master time clock 29516 may measure time as a system counter, for example. The master time clock 29516 may measure time as relative time, taking into account defined events. The master time clock 29516 may measure time by clock pulses. The master time clock 29516 may count clock pulses. The master time clock 29516 may measure time in seconds. The master time clock 29516 may measure time in microseconds. The master time clock 29516 may measure time in seconds and / or microseconds based on counted clock pulses. The master time clock 29516 may measure time in processor cycles. The master time clock 29516 may count processor cycles. The master time clock 29516 may be set to time. The master time clock 29516 may be set manually. The master time clock 29516 may be set based on a reference clock. The master time clock 29516 may be set based on an RTC reference.

[0177] The master time clock 29516 may include an RTC. The master time clock 29516 may include an integrated circuit RTC. The RTC may measure the passage of time. The RTC may include an internal oscillator. The internal oscillator may include, for example, a crystal oscillator. The RTC may include a micromechanical resonator. The RTC may include an external frequency reference. The external frequency reference may include a power line frequency. The power line frequency may include the nominal frequency of the oscillation of the alternating current in a wide-area synchronous grid. The RTC may measure the passage of time based on the oscillator frequency. The RTC may be software-based. The master time clock 29516 may be any RTC, such as those known by the trade names of Epson, Intersil, Integrated Device Technology, Maxim, NXP Semiconductors, Texas Instruments, STMicroelectonics, and / or Ricoh.

[0178] The master time log 29526 may contain any information indicating time-based measurements. For example, the master time log 29526 may contain a data structure. The data structure may include, for example, a table data structure, an array, a linked list, a flat file, a record, a delimited data stream, an XML data store, etc. For example, the master time log 29526 may contain one or more records. Each record may represent a respective measurement. A record may indicate, for example, a sensing system ID, the measurement itself, a related measurement such as time, and / or timing information. The master time log 29526 may function as a data repository. The master time log 29526 may replicate data for logging purposes. The master time log 29526 may function as a buffer for output to the downstream system 29522.

[0179] One or more surgical sensing systems 29518 may be surgeon sensing systems and / or patient sensing systems. For example, surgical sensing system 29518 may incorporate and / or be incorporated into sensing system 20069 disclosed herein. Surgical sensing system 29518 may include a sensor unit. Surgical sensing system may include a data processing and communication unit. Surgical sensing system 29518 may include a sensing system clock 29528. Surgical sensing system 29518 may communicate with a surgical hub.

[0180] For example, the surgical sensing system 29518 may include one or more sensor units for measuring one or more biomarkers. The surgical sensing system 29518 may include one or more sensor units for measuring the environment. The surgical sensing system 29518 may include one or more sensor units for measuring surgical instrument parameters and energy data. The surgical sensing system 29518 may include one or more sensor units for measuring capital equipment data. For example, the surgical sensing system 29518 may include one or more sensor units for measuring biomarkers such as sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol consumption, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, hydrogen blood potential, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, airway bacteria, edema, mental state, sweat, circulating tumor cells, autonomic nervous system tone, circadian rhythm, and / or menstrual cycle. These biomarkers can be measured using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The sensors can measure biomarkers as described herein using one or more sensing techniques such as photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedimentary spectroscopy, potentiometric measurement, and current measurement.

[0181] The surgical sensing system 29518 can measure data. The surgical sensing system 29518 can apply sensor time to surgical sensing system events. The surgical sensing system 29518 may apply sensor time to surgical sensing system events based on the sensor time at which the data was measured. The surgical sensing system 29518 may apply sensor time to measured data based on the sensing system clock 29528. For example, the surgical sensing system 29518 may apply sensor time to measured values ​​at the time of measurement. The surgical sensing system 29518 can measure data continuously. The surgical sensing system 29518 may measure data based on a sample rate. The sample rate may be based on the sensing system clock. For example, the surgical sensing system 29518 may perform measurements after a predetermined number of clock cycles. The surgical sensing system 29518 may perform measurements after a specific time interval.

[0182] The surgical sensing system 29518 may transmit the data stream 29530 to different devices such as a surgical hub. The data stream 29530 may include, for example, the sensed measurement value 29532 itself, the sensing system ID 29534, timing information associated with the sensed information 29536, metadata associated with the sensed measurement value, etc. The surgical sensing system 29518 may be configured to transmit the data stream 29520 based on an received request. The surgical sensing system 29518 may be configured to select the sensed measurement values ​​to transmit. The surgical sensing system 29518 may apply sensor time to the data stream based on when the measurement values ​​are transmitted and the sensing system clock 29528.

[0183] In one embodiment, the data feed may be asynchronous. The data feed may reflect one or more patient monitoring parameters. The data feed may include an instrument feed. The instrument feed may include energy device generator data. The instrument feed may include a wireless data stream. The instrument feed may include a wireless data stream from a digitally compatible surgical device. A digitally compatible surgical device may include, for example, a motorized stapler. The instrument feed may include instrument capital equipment. Instrument capital equipment may include, for example, a generator, a fume extractor, and / or a visual system.

[0184] In one embodiment, the displayed fused data feed may show instrument power. The displayed fused data feed may show temperature. The displayed fused data feed may show progress, such as the progress of a procedure. The displayed fused data feed may show one or more of instrument power, temperature, and progress. The displayed fused data feed may show one or more of instrument power, temperature, and progress related to video feeds and physiological effects. In one embodiment, the fused data may aggregate datasets. The fused data may aggregate datasets to show, for example, measures. The fused data may aggregate datasets to show, for example, the results of a surgical task.

[0185] The sensing system clock 29528 may include any electrical and / or computing resources suitable for measuring time. For example, the sensing system clock 29528 may measure the passage of time. The sensing system clock 29528 may measure time as an RTC. The sensing system clock 29528 may measure time as a system counter, for example. The sensing system clock 29528 may measure time as relative time, taking a defined event into consideration. The sensing system clock 29528 may measure time in clock pulses. The sensing system clock 29528 may count clock pulses. The sensing system clock 29528 may measure time in seconds. The sensing system clock 29528 may measure time in microseconds. The sensing system clock 29528 may measure time in seconds and / or microseconds based on counted clock pulses. The sensing system clock 29528 may be set to time. The sensing system clock 29528 may be set manually. The sensing system clock 29528 may be set based on a reference clock. The sensing system clock 29528 may be set based on an RTC reference.

[0186] The sensing system clock 29528 may include an RTC. The sensing system clock 29528 may include an integrated circuit RTC. The RTC may measure the passage of time. The RTC may include an internal oscillator. The internal oscillator may include, for example, a crystal oscillator. The RTC may include a micromechanical resonator. The RTC may include an external frequency reference. The external frequency reference may include a power line frequency. The power line frequency may include the nominal frequency of the oscillation of the alternating current in a wide-area synchronous grid. The RTC may measure the passage of time based on the oscillator frequency. The RTC may be software-based. The sensing system clock 29528 may be an RTC, such as those known by the trade names of Epson, Intersil, Integrated Device Technology, Maxim, NXP Semiconductors, Texas Instruments, STMicroelectonics, and / or Ricoh.

[0187] In one embodiment, one or more surgical sensing systems 29518 do not have to share a common RTC standard. The master time clock 29516 does not have to share a common RTC standard with one or more surgical sensing systems. One or more surgical sensing systems 29518 can be set to various RTC standards. One or more surgical sensing systems 29518 may output various times based on non-common RTC standards. One or more surgical sensing systems 29518 may output a different time from the surgical hub based on non-common RTC standards.

[0188] In one embodiment, the surgical hub 29520 may be configured to order received measurements 29532 from at least one surgical sensing system 29518. The surgical hub 29520 may receive a data stream 29530 from at least one surgical sensing system 29518. The data stream 29530 may include measurements 29532, sensing system ID 29534, one or more sensor times associated with measurement 29536, metadata associated with measurement 29532, and so on. The surgical hub 29520 may apply the received time associated with the data stream 29530. The surgical hub 29520 may apply the received time associated with the data stream 29530 based on a master time clock 29516.

[0189] In one embodiment, the surgical hub 29520 may acquire a latency value associated with the data stream 29530. The surgical hub 29520 may acquire a latency value associated with the data stream 29530 based on one or more sensor times associated with the measurement 29536. The surgical hub 29520 may acquire a latency value associated with the data stream 29530 based on the reception time associated with the data stream 29530. The surgical hub 29520 may acquire a latency value associated with the data stream 29530 based on any combination of one or more sensor times and / or reception times.

[0190] In one embodiment, the surgical hub 29520 may determine a time code. The surgical hub 29520 may determine a time code based on acquired latency values. The surgical hub 29520 may determine a time code based on the master time clock 29516. The surgical hub 29520 may determine a time code based on any combination of acquired latency values ​​and / or the master time clock 29516. The time code may be a number. The time code may be a time relative to the master clock time 29516. The time code may be a time relative to real time. The surgical hub 29520 may apply the determined time code to the data stream 29530.

[0191] In one embodiment, the surgical hub 29520 may provide a synchronization timestamp to each sensing system 29518 on a common network. The master time clock 29516 may record the relational matrix of each of the other clocks at any given time. In one embodiment, synchronization may include a signal processing method. The signal processing method may handle non-uniformly sampled data. Non-uniformly sampled data may include, for example, data with large time gaps.

[0192] In one embodiment, the surgical hub 29520 may use data distribution rate and frequency specifications. The surgical hub 29520 may control the sampling rate. The surgical hub 29520 may control the time indexing of data points. For example, the surgical hub 29520 may instruct when data is sent to the surgical hub 29520 by the sensing system 29518. The surgical hub 29520 may use a clock and trigger system. A trigger signal may be sent to a given sensing system 29518. Data may be sent back and recorded. Data may be sent back and recorded based on the trigger signal. Knowing the transmission latency allows the surgical hub 29520 to set a timestamp for system recording events. The sensing system 29518 may measure data continuously. The sensing system 29518 may transmit data when prompted by a hub trigger signal.

[0193] In one embodiment, a shift register-based system may be used. The shift register-based system may monitor inputs from multiple connections, such as the sensing system 29518. The shift register-based system may sequentially output the state of each input. The shift register-based system may sequentially output the state of each input based on a trigger signal. A clock signal may be used. The clock signal may travel through various inputs. The clock signal may travel through various inputs in a specific sequence. The clock signal may travel through various inputs in a specified sequence based on the frequency of the clock signal.

[0194] In one embodiment, the combination of trigger and clock may determine the capture frequency. In one embodiment, a single cycle through a shift register-based system may provide a snapshot of all inputs over a specified time. A fast clock frequency may provide high temporal resolution over read samples. In one embodiment, the latency and clock duration of the sensing system allow for calculations that trace back in time to the actual timestamp. For example, if it takes 1 microsecond to transmit the trigger signal, 1 microsecond for the clock signal read cycle, and 1 microsecond for the data signal return, the occurrence time may be estimated to be 3 microseconds before the data is read.

[0195] In one embodiment, ad-hoc software synchronization may be used for data stream fusion and / or pinning. Data stream fusion and pinning enable synchronization without predefined transformations. In one embodiment, multisensor data fusion and distributed signal processing may be used. Multisensor data fusion and distributed signal processing may be used to fuse the data feed into a global system timecode. A fused signal array may be used. The fused signal array may be used to assign extended block timestamps. Extended block timestamps may be used to adjust the offset of the data feed.

[0196] Shifts with a certain delay may occur in fused signals. Synchronizers may be used to fuse asynchronous data. Synchronizers may be used to add an integrated timing code. The integrated timing code may be an extended timestamp. Dropped frames and / or data points may be detected based on the integrated timing code. Feeds may be linked together. Feeds may be linked to each other based on the integrated timing code. In one embodiment, software-generated blocks and timestamps may be used with a non-real-time operating system. Extended timestamps may be used.

[0197] Extended timestamps may include, for example, tolerant timestamp matches, full blockstamp matches, and / or overlay timestamp matches. Tolerant timestamp matches may be used for synchronization. Tolerant timestamp matches may be used for synchronization based on tolerance. Ascending timestamps matching within tolerance may indicate synchronization. Tolerant timestamp matches may be used for sources with equal rates. Full blockstamp matches may be used in single-source dataflow partitioning scenarios across multiple processing pipelines affected by stream and processing incompleteness. Full blockstamp matches may be used, for example, for processors with different speeds. Full blockstamp matches may be used to detect dropped frames. Overlay timestamp matches may use timestamp intervals. Timestamp intervals of several dataflows with different sampling rates may be grouped. Timestamp intervals of several dataflows with different sampling rates may be grouped in the case of interval overlays. The output rate of synchronized data may be higher than the rate of the fastest stream. Overlay timestamp matches may be used in combination with blockstamp information regarding the sequence integrity of the streams. Overlay timestamps can be used to synchronize streams with large frequency differences.

[0198] In one embodiment, the combined blockstamp and timestamp may be extended to the signal. The combined blockstamp and timestamp may be extended to the signal at the pipeline capture node. Extended timestamps may be generated. Extended timestamps may be generated, for example, by an incremental counter. Extended timestamps may be generated, for example, by a blockstamp. Extended timestamps may be generated, for example, by a local clock. Extended timestamps may be generated, for example, by a timestamp. Timestamps allow for the mixing of sensor signals at different data rates. Blockstamps allow data originating from one source and divided across multiple computing nodes to be synchronously fused, for example, on multiple hosts or multicore processors. Statistical ad-hoc regression algorithms may be used. Statistical ad-hoc regression algorithms may filter the estimation of inter-frame timing. Statistical ad-hoc regression algorithms may include Widrow and / or Kalman.

[0199] In one embodiment, the surgical hub 29520 may generate an output. The surgical hub 29520 may output the received data stream 29530. The surgical hub 29520 may order the received data stream 29530. The surgical hub 29520 may order the received data stream 29530 based on the master time clock 20516. The surgical hub 29520 may order the received data stream 29530 based on the applied time code. The surgical hub 29520 may order the received data stream 29530 based on when the measurement associated with the data stream 29530 was sensed. The surgical hub 29520 may send the ordered output to the master time log 29526. The surgical hub 29520 may send the ordered output to the downstream data system 29522.

[0200] In one embodiment, the downstream data system 29522 may receive ordered outputs from the surgical hub 29520. The downstream data system 29522 may include, for example, a display. The display may show the ordered data stream 29530. For example, the HCP may use the downstream data system 29522 to monitor information associated with the surgical sensing system. The HCP may use the downstream data system 29522 to monitor information associated with the surgical sensing system. The HCP may use the downstream data system 29522 to analyze multiple data streams from the surgical sensing system occurring simultaneously. For example, the HCP may analyze multiple data streams from the surgical sensing system to determine contextual information about the surgical procedure, patient recovery, the patient, and / or the surgeon. The HCP may analyze, for example, heart rate measurement spiking simultaneously with blood pressure measurement spiking.

[0201] Figure 14B is a block diagram of an exemplary processing unit 29538. The processing unit 29538 may receive surgical sensing system data 29540. The processing unit 29538 may order the received surgical sensing system data 29540. As shown in Figure 14B, the timeline 29542 shows multiple measurements associated with various sensing systems 29544, 29546, and 29548. Multiple measurements may be sensed at time 29550. Multiple measurements may be received at a later time 29552. Multiple measurements may be received in an order different from the order in which they were sensed. The processing unit 29538 may order the received surgical sensing system data 29540. The ordered surgical sensing system data may emulate how the measurements were sensed. The processing unit may order the received surgical sensing system data 29540 based on the master time clock 29554. The processing unit 29538 may determine the time code associated with each surgical sensing system data. The processing unit 29538 may determine the time code associated with each surgical sensing system data based on the master time clock 29554. In 29556, each time code may be applied to the respective surgical sensing system data. The surgical sensing system data may be ordered based on the applied time code.

[0202] The processing unit 29538 may include computing hardware and / or software suitable for processing and / or ordering sensor data. For example, the processing unit 29538 may incorporate and / or be integrated into the processor module 20057. For example, the processing unit 29538 may be deployed as a standalone unit for sensor processing. The processing unit 29538 may be any single-core or multi-core processing unit. The processing unit 29538 may be any processor, such as the one known by the trade name ARM Cortex by Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0203] The processing unit 29538 may receive input. The processing unit 29538 may receive data as input. The processing unit 29538 may generate output. The processing unit 29538 may process data. The processing unit 29538 may generate output based on the processed data. The processing unit 29538 may receive surgical sensing system data 29540. The processing unit 29538 may be configured to collect measurement data from one or more surgical sensing systems. The processing unit 29538 may determine a time code. The processing unit 29538 may determine a time code based on a master time clock 29554. The processing unit 29538 may apply a time code to the received surgical sensing system data. The processing unit 29538 may order the received surgical sensing system data. The processing unit 29538 may order the received surgical sensing system data based on the applied time code. The processing unit 29538 may output ordered surgical sensing system data. The processing unit 29538 may transmit the ordered surgical sensing system data to a downstream data system. The processing unit 29538 may transmit the ordered surgical sensing system data to a storage medium, such as a master time log.

[0204] The surgical sensing system data 29540 may be associated with one or more surgical sensing systems. The surgical sensing system data 29540 may be transmitted as a data stream 29558. The surgical sensing system data 29540 may include a measurement value 29560 associated with a surgical sensing system. The surgical sensing system data 29540 may include a sensing system ID 29562 associated with a surgical sensing system measurement value. The surgical sensing system data 29540 may include at least one time 29564 associated with a surgical sensing system measurement value. At least one time 29564 associated with a surgical sensing system measurement value may include the time the measurement value was sensed. At least one time 29564 associated with a surgical sensing system measurement value may include the time the measurement value was transmitted.

[0205] As shown in Figure 14B, in 29550, a plurality of measurements 29544, 29546, and 29548 associated with at least one surgical sensing system can be sensed. The plurality of measurements can be sensed at various times. For example, the first measurement 29544 may be sensed before the second measurement 29546 and the third measurement 29548. The second measurement 29546 may be sensed before the third measurement 29548. In 19552, the plurality of measurements can be received by, for example, a processing unit 29538. The plurality of measurements can be received in an order different from the order in which they were sensed. The plurality of measurements can be received in an order different from the order in which they were sensed, based on various latency values ​​associated with the surgical sensing system. For example, the second measurement 29546 may be received before the third measurement 29548 and the first measurement 29544. The third measurement 29548 may be received before the first measurement 29544. The processing unit 29538 may order multiple measurements. The processing unit 29538 may order multiple measurements to emulate the order in which the measurements were perceived. For example, the processing unit 29538 may order multiple measurements to show that the first measurement 29544 occurred before the second measurement 29546 occurred before the third measurement 29548.

[0206] In one embodiment, the processing unit 29538 may include a master time clock 29554. The master time clock 29554 may incorporate and / or be incorporated into the master time clock 29516, as shown in Figure 14A. The processing unit 29538 may use the master time clock 29554 to order the received measurements. The processing unit 29538 may apply the received time associated with each measurement. The processing unit 29538 may apply the received time associated with each measurement based on the master time clock 29554. The processing unit 29538 may order the received measurements based on the received time associated with each measurement.

[0207] In one embodiment, the processing unit 29538 may acquire a delay value. The processing unit 29538 may acquire a delay value associated with the received measurement. The delay value associated with the received measurement may include the elapsed time between the sensed measurement and the received measurement.

[0208] The processing unit 29538 may be configured to select a method for obtaining the latency value. For example, the processing unit 29538 may obtain the latency value based on prior knowledge. In one embodiment, the processing unit 29538 may obtain the latency value and return a sensed measurement using a method of sending a request to the surgical sensing system. In one embodiment, the processing unit 29538 may obtain the latency value and return the most recently sensed measurement using a method of sending a request to the surgical sensing system. In one embodiment, the processing unit 29538 may obtain the latency value and return the next sensed measurement using a method of sending a request to the surgical sensing system. In one embodiment, the processing unit 29538 may obtain the latency value and return the sensed measurement that is temporally closest to the time the surgical sensing system receives the request. The temporally closest sensed measurement may include the most recently sensed measurement and / or the next measurement that the surgical sensing system senses. The surgical sensing system may be configured to determine the remaining time until a subsequent measurement is sensed based on the sampling rate. The surgical sensing system may be configured to select the most recently sensed measurement value that is temporally closest to the received request, based on the time of the most recently sensed measurement value and the sampling rate.

[0209] The processing unit 29538 may determine a time code. The processing unit 29538 may determine a time code of a received measurement. The processing unit 29538 may determine a time code based on the master time clock 29554. The processing unit 29538 may determine a time code based on the acquired latency value. The processing unit 29538 may determine a time code based on the time the data stream was received. The processing unit 29538 may determine a time code based on the master time clock 29554 and / or any combination of the acquired latency. The time code may be any number. The time code may be a time relative to the master time clock 29554. The time code may be a time relative to real time.

[0210] In 29556, processing unit 29538 may apply time codes. Processing unit 29538 may apply time codes to received measurements. Processing unit 29538 may apply time codes to received measurements associated with a surgical sensing system. Processing unit 29538 may order received measurements based on the applied time codes. Processing unit 29538 may order multiple received measurements based on the applied time codes. The ordered measurements may emulate the order in which the measurements were sensed. The ordered measurements may be integrated into a common time system based on the applied time codes.

[0211] Figure 15 shows a timeline 29568 illustrating an exemplary method for obtaining latency values ​​29570 associated with surgical sensing system measurements. The surgical sensing system may continuously sense measurements. The surgical sensing system may transmit the sensed measurements. The sensed measurements may be transmitted to a computing device such as a surgical hub.

[0212] As shown in Figure 15, the surgical sensing system may sense a measurement at a first time 29570 on timeline 29568. The surgical sensing system may transmit the sensed measurement at a second time 29572 on timeline 29568. The second time 29572 may occur after the first time 29570. The measurement may be received at a third time 29574 on timeline 29568. The third time 29574 may occur after the second time 29572.

[0213] The latency value 29570 may include the delay between the sensed measurement and the measurement received by a processing device such as a surgical hub. The latency value 29570 may include the delay before transmitting the measurement 29578. The delay before transmitting the measurement 29578 may be the time difference between the sensed measurement and the measurement being transmitted to a processing device such as a surgical hub. The latency value 29570 may include the transit time 29580. The transit time 29580 may be the time difference between the transmitted measurement and the measurement received by a processing device such as a surgical hub. The latency value 29570 may include a combination of the delay before transmitting the measurement 29578 and the transit time 29580.

[0214] In one embodiment, a delay before transmitting measurement value 29578 may be determined. The delay before transmitting measurement value 29578 may be determined based on the surgical sensing system clock. The surgical sensing system clock may tag measurement values ​​with sensor event times. The surgical sensing system clock may tag measurement values ​​with sensor event times based on the surgical sensing system clock. Sensor event times may include any time during which the surgical sensing system senses a measurement value. Sensor even times may include any time during which the surgical sensing system transmits a measurement value. The delay before transmitting a measurement value may be determined, for example, by calculating the time difference between the time the measurement value is sensed and the time the measurement value is transmitted.

[0215] In one embodiment, the transit time 29580 may be determined. The transit time 29580 may be determined based on prior knowledge.

[0216] Figure 16 shows a timeline 29582 illustrating an exemplary method for obtaining latency values ​​associated with surgical sensing system measurements. The surgical sensing system may continuously sense measurements. The surgical sensing system may receive requests to return sensed measurements. A surgical hub may be configured to send requests. The surgical hub may be configured to send requests to return sensed measurements. The surgical sensing system may receive requests and return sensed measurements from the surgical hub.

[0217] In one embodiment, as shown in Figure 16, the surgical hub may send a request to the surgical sensing system. In 29584, the surgical hub may send a request to the surgical sensing system. The request may be a request to return a sensed measurement. In 29586, the surgical sensing system may receive a request to return a sensed measurement. In 29588, it may sense the measurement. In 29590, it may return the measurement. In 29592, it may receive the measurement. The measurement may be received by the surgical hub.

[0218] A latency value associated with a measurement can be obtained. This latency value may be obtained based on a round-trip latency value of 29594. The round-trip latency value of 29594 may be the time difference between the time a request to return a measurement is sent and the time the measurement is received. The latency value associated with a measurement can be approximated, for example, by calculating half of the round-trip latency value of 29594.

[0219] In one embodiment, the round-trip latency value 29594 may be obtained based on a clock. The clock may include a master time clock. The surgical hub may include a master time clock. The master time clock may be an RTC. The master time clock may tag surgical hub events with hub event time. Hub event time may include the time it takes for the surgical hub to send a request to return a measurement. Hub event time may include the time it takes for the surgical hub to receive a sensed measurement. The round-trip latency value 29594 may be calculated using the difference between the tagged time when the surgical hub sends a request to return a measurement and the tagged time when the surgical hub receives a sensed measurement.

[0220] In one embodiment, the latency value may be obtained based on a master time clock and a surgical sensing system clock. The master time clock may tag surgical hub events with hub event times. The surgical sensing system clock may tag surgical sensing system events with sensing system event times. The latency value may be calculated based on the hub event time and the sensing system event time. The latency value may be calculated based on the hub event time, the surgical sensing system event time, a first transit time while the surgical hub transmits a request, and a second transit time until the surgical sensing system returns a measurement and the surgical hub receives the measurement.

[0221] In one embodiment, the surgical sensing system may be configured to continuously sense measurements. A surgical hub may be configured to send requests to the surgical sensing system. The surgical sensing system may be configured to return measurements to the surgical hub based on the received requests. The surgical sensing system may be configured to select which sensed measurements to return based on the received requests. The surgical sensing system may select measurements based on instructions from the surgical hub requests. In one embodiment, the surgical sensing system may select measurements based on the sensing time associated with the measurement. The surgical sensing system may select the measurement that is the most recently sensed measurement relative to the time the request was received. The surgical sensing system may select the measurement that is the next measurement to be sensed relative to the time the request was received. The surgical sensing system may select the measurement that is temporally closest to the time the request was received. The surgical sensing system may determine the measurement that is temporally closest to the time the request was received based on the sensing time associated with the measurement and the sampling rate associated with the surgical sensing system. The surgical sensing system may determine the length of time until the next measurement is sensed based on the sampling rate.

[0222] Figure 17 shows a timeline 29596 illustrating an exemplary method for selecting a sensed measurement. At 29598, the surgical hub may send a request to the surgical sensing system. At 29600, the surgical sensing system may receive the request. At 29602, the surgical sensing system may sense a first measurement. At 29604, the surgical sensing system may sense a second measurement. As shown in Figure 17, the time 29602 when the first measurement is sensed may be earlier than the time 29604 when the second measurement is sensed. The time 29600 when the surgical sensing system receives the request may be later than the time 29602 when the first measurement is sensed, but earlier than the time 29604 when the second measurement is sensed. At 29606, the surgical sensing system may return the selected measurement to the surgical hub. At 29608, the surgical hub may receive the selected measurement. The round-trip latency 29610 can be obtained based on the time the surgical hub sent the request and the time the surgical hub received the selected measurement.

[0223] In one embodiment, the surgical sensing system may select the most recently sensed measurement for the time the request was received from the surgical hub. As shown in Figure 17, the most recently sensed measurement for the time the request was received may be a first measurement sensed at 29602. The surgical sensing system may select the first measurement sensed at 29602 and return it to the surgical hub. The surgical sensing system may select the first measurement sensed at 29602 and return it to the surgical hub based on the received request.

[0224] In one embodiment, the surgical sensing system may select a measurement value that is sensed immediately after receiving a request from the surgical hub. As shown in Figure 17, the measurement value sensed immediately after receiving the request may be a second measurement value sensed at 29604. The surgical sensing system may select the second measurement value sensed at 29604 and return it to the surgical hub. The surgical sensing system may select the second measurement value sensed at 29604 and return it to the surgical hub based on the received request.

[0225] In one embodiment, the surgical sensing system may select the measurement closest to the time the request was received from the surgical hub. The sensing measurement closest to the time the request was received may be the most recently sensed measurement. The sensing measurement closest to the time the request was received may be the sensing measurement that was sensed immediately after the request was received.

[0226] In one embodiment, the surgical sensing system may determine the time between receiving a request and sensing a measurement. The surgical sensing system may determine the time between receiving a request and sensing a measurement based on a sampling rate 29612. The sampling rate 29612 may be the frequency with which the surgical sensing system senses a measurement. The sampling rate may be the time difference between a first measurement at 29602 and a second measurement at 29604. The surgical sensing system may calculate a first measurement delay 29614 and a second measurement delay 29616.

[0227] In one embodiment, a measurement that is more temporally close to the time the surgical sensing system receives request 29600 may be determined. The measurement that is more temporally close to the time the surgical sensing system receives request 29600 may be determined based on a first delay 29614 and a second delay 29616. The measurement that is more temporally close to the time the surgical sensing system receives the request may be, for example, the smaller of the first delay and the second delay. A first measurement delay 29614 may be determined. The first measurement delay 29614 may be the time difference between when the surgical sensing system senses the first measurement 29602 and when the surgical sensing system receives request 29600. A second measurement delay 29616 may be determined. The second measurement delay 29616 may be the time difference between when the surgical sensing system senses the second measurement 29604 and when the surgical sensing system receives request 29600.

[0228] Figure 18 shows an exemplary master time log 29618 having surgical sensing system data. The master time log 29618 may store surgical sensing system data using a data structure in memory. The surgical sensing system data may include measured values ​​and associated measured value metadata. The measured value metadata may include a sensing system identifier 29620, measured value 28622, measured value reception time 29624, latency value 29626, and time code 29628. The sensing system identifier 29620 may include identifier metadata associated with the surgical sensing system. The measured value 29624 may include a measured value received from the surgical sensing system. The measured value reception time 29624 may include the time when the time metadata associated with the surgical sensing system data was received by the surgical hub. The latency value 29626 may include the acquired latency associated with the surgical sensing system measured value. Timecode 29628 may be a value calculated by the surgical hub, used to order measurements against a common time system.

[0229] In one embodiment, multiple surgical sensing systems may sense measurements. These multiple surgical sensing systems may transmit the sensed measurements to a surgical hub. Each measurement may include metadata associated with the sensed measurement. The surgical hub may tag each measurement with its measurement reception time, based on the master clock time when the measurement is received by the surgical hub. The surgical hub may determine a timecode for each received measurement based on the measurement metadata. The measurement metadata used to determine the timecode may include a measurement latency value and the measurement reception time associated with the measurement. The surgical hub may apply the determined timecode to the measurement. The surgical hub may order multiple measurements based on the applied timecode. As shown in Figure 18, the order in which the surgical hub receives multiple measurements may differ from the order based on the applied timecode. The order based on the applied timecode may be emulated when the measurements are sensed.

[0230] Figure 19 shows an exemplary method for ordering surgical sensing system data. In 29630, measurements may be received. Measurements may be associated with a surgical sensing system. For example, a surgical data ordering system may receive measurements associated with a surgical sensing system. For example, a surgical sensing system may sense a measurement and transmit that measurement to the surgical data ordering system. For example, the surgical data ordering system may send a request to the surgical sensing system to return the sensed measurement.

[0231] For example, a surgical data ordering system may send a request to a surgical sensing system to return selected sensed measurements. The selected sensed measurements may include the most recently sensed measurement when the request is received, the next measurement to be sensed after the request is received, and / or the sensed measurement that is closest in time when the request is received. A surgical hub and / or processing unit may receive measurements associated with a surgical sensing system as described herein.

[0232] Synchronization may be performed in 29631. For example, the synchronization may include any of the techniques disclosed herein, such as latency compensation, network synchronous timestamps, master time clocks, signal processing methods, trigger signals, shift register-based systems, ad-hoc software synchronization, data stream fusion and / or fixation, and extended timestamps (including, for example, tolerable timestamp matching, full blockstamp matching, and / or overlay timestamp matching).

[0233] In one embodiment, in 29632, a latency value may be acquired. The latency value may be associated with a received measurement. The latency value may be associated with a surgical sensing system. The latency value may be acquired in advance. The latency value may include a delay time. The delay time may be the time elapsed between sensing the measurement and transmitting the measurement. The latency value may include a transit time. The transit time may be the time from transmitting the measurement until the measurement is received. The latency value may be acquired, for example, based on one or more of the delay time and the transit time. The latency value may be acquired based on a round-trip latency value. The latency value may be approximated as half of the round-trip latency value.

[0234] In 29634, a time code may be applied. The time code may be applied to a received measurement. The time code may be applied based on the latency value associated with the received measurement. The time code may be applied based on the latency value associated with the surgical sensing system. In one embodiment, the applied time code may be based on the acquired latency and / or measurement reception time. For example, a surgical data ordering system may apply a time code to a received measurement. For example, a surgical hub and / or processing unit may apply a time code to a received measurement.

[0235] In 29636, received measurements may be ordered. Received measurements may be ordered based on the applied timecode. Multiple received measurements may be ordered. Multiple received measurements may be ordered based on the applied timecode. The ordered output may include multiple received measurements in an order different from the order in which they were received. The ordered output may include multiple received measurements in an order different from the order in which they were perceived. The ordered measurements may emulate the order in which multiple measurements were perceived. For example, a surgical data ordering system may order one or more measurements. A surgical data ordering system may order one or more measurements based on the applied timecode.

[0236] In 29638, ordered measurements may be entered into a master time log. The master time log may contain data structures in memory. These data structures may include, for example, table data structures, arrays, linked lists, flat files, records, delimited data streams, XML data stores, etc. The master time log may contain any information indicating time-based measurements. The master time log may include, for example, a sensing system ID, the measurement itself, time, and related measurement and / or timing information. For example, a surgical data ordering system may enter measurements into the master time log. For example, a surgical hub and / or processing unit may enter ordered measurements into the master time log.

[0237] In one embodiment, master time log data may be transmitted to a downstream data system. For example, the downstream data system may include a display. The display may be used to monitor at least one surgical sensing system. The display may include a visual representation of surgical sensing system data. The visual representation may include current measurement readings and / or a graphical representation. The graphical representation may include surgical sensing system measurements as a function of time. For example, the display may be used to monitor two or more surgical sensing systems on an integrated time system.

[0238] In one embodiment, the received measurements may be ordered. The measurements may be associated with a surgical sensing system. The measurements may be associated with a communication interface. The measurements may be associated with both the surgical sensing system and the communication interface. The measurements may be received first. A latency value may be received. The latency value may be associated with a surgical sensing system. The latency value may be associated with a communication interface. The latency value may be associated with both the surgical sensing system and the communication interface. A time code may be applied. A time code may be applied to the received measurements. A time code may be applied based on a first time. A time code may be applied based on an acquired latency value. A time code may be applied based on a first time and an acquired latency value. The outputs may be ordered. The outputs may be ordered based on the received measurements. The outputs may be ordered based on the applied time code. The outputs may be ordered based on the received measurements and the applied time code.

[0239] In one embodiment, the surgical sensing system may include one or more sensors. The surgical sensing system may include one or more sensors configured to sense at least one biomarker. The surgical sensing system may include one or more sensors configured to sense at least one environment. The surgical sensing system may include one or more sensors configured to sense one or more of the biomarker and / or environment. The surgical sensing system may be configured to sense at least one surgical instrument parameter.

[0240] In one embodiment, the master time log may receive a data stream. The data stream may include, for example, one or more ordered measurements. The master time log may include storage. The master time log may store the received data stream in storage. The master time log may store, for example, one or more received ordered measurements in storage.

[0241] In one embodiment, the surgical sensing system may transmit a measurement value. The surgical sensing system may transmit the sensed measurement value. The surgical sensing system may receive a request. The surgical sensing system may receive a request to transmit a measurement value. The surgical sensing system may receive a request at a second time. Based on the received request, the surgical sensing system may transmit the sensed measurement value. The surgical sensing system may transmit the sensed measurement value at a third time. The third time may be later than the second time.

[0242] In one embodiment, the surgical sensing system may select a measurement value. The surgical sensing system may select a measurement value based on the time the measurement value was detected. The surgical sensing system may, for example, select the most recently detected measurement value. The surgical sensing system may select a measurement value based on a received request. The surgical sensing system may, for example, select a future measurement value that has not yet been detected. The surgical sensing system may, based on a received request, select a future measurement value that has not yet been detected. The surgical sensing system may, for example, select the measurement value that is temporally closest to the received request.

[0243] In one embodiment, the output may be displayed. The output may be displayed based on a master time log. The output may be displayed on a display. A visual representation may be generated based on the output. The visual representation may include current measurement readings. The visual representation may include a graphical representation. The visual representation may be displayed.

[0244] In one embodiment, two or more received measurements may be ordered. A first measurement may be received. The first measurement may be received at a first time. The first measurement may be associated with a first sensing system. The first measurement may be associated with a communication interface. A second measurement may be received. The second measurement may be received at a second time. The second measurement may be associated with a second sensing system. The second measurement may be associated with a communication interface. One or more latency values ​​may be obtained. One or more latency values ​​may be associated with a first surgical sensing system. One or more latency values ​​may be associated with a second surgical sensing system. One or more latency values ​​may be associated with a communication interface. One or more time codes may be applied. One or more time codes may be applied based on one or more latency values. Outputs may be ordered. Outputs may be ordered based on the first measurement and the second measurement. Outputs may be ordered based on one or more time codes.

[0245] The following list of embodiments forms part of the description. 1. A device for ordering received measurement values, Equipped with a processor, the processor, In the first time period, receiving a measurement value, the measurement value being associated with the sensing system and the communication interface, To obtain latency values ​​associated with surgical sensing systems and communication interfaces, Based on the first time and the acquired waiting time value, a time code is applied to the received measurement, Based on the received measurements and time codes, the outputs are ordered. A device configured to perform the following. 2. The apparatus according to embodiment 1, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of biomarkers, the environment, and surgical instrument parameters. 3. The apparatus according to embodiment 1, further comprising a master time log on which the output is stored. 4. The surgical sensing system In the second time, a request is received to return the measurement value, Selecting a measurement based on at least one sensor reading and a second time, In the third time, return the measured values ​​to the device. The apparatus according to embodiment 1, configured to carry out the following. 5. The processor, In the second time, a request is sent to the surgical sensing system, Based on the second time and the first time, calculate the waiting time value. The apparatus according to embodiment 1, further configured to carry out the following. 6. The processor, The apparatus according to embodiment 1, further configured to assign extended timestamps to measured values. 7. The apparatus according to embodiment 1, further comprising a display configured to display a visual representation based on the output. 8. A method for an apparatus for ordering received measurements, In the first time period, the measurement is received, and the measurement is associated with the sensing system and the communication interface. To obtain latency values ​​associated with surgical sensing systems and communication interfaces, Based on the first time and the acquired waiting time value, a time code is applied to the received measurement, Based on the received measurements and time codes, the outputs are ordered. Methods that include... 9. The method according to aspect 8, wherein ordering the outputs includes inputting the outputs into a master time log. 10. The method according to embodiment 8, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, the environment, or a surgical instrument parameter. 11. The method of aspect 8, wherein obtaining a waiting time value includes, at a second time, sending a request to a surgical sensing system, and calculating a waiting time value based on the second time and the first time. 12. To generate a visual representation based on the output, Displaying visual representations and The method according to embodiment 8, further comprising: 13. The method according to aspect 8, wherein applying a timecode includes assigning an extended timestamp to a measurement. 14. The method according to embodiment 14, wherein the extended timestamp comprises at least one of an overlay timestamp match, a full blockstamp match, and an allowable timestamp match. 15. A device for ordering two or more received measurements, Equipped with a processor, the processor, Receiving a first measurement value at a first time, wherein the first measurement value is associated with a first sensing system and a communication interface. Receiving a second measurement at a second time, wherein the second measurement is associated with a second sensing system and a communication interface. Acquiring a first surgical sensing system and a second surgical sensing system, as well as a first and second latency value associated with the communication interface, Applying the first time code and the second time code to the first measured value and the second measured value based on the first time and the second time, and the first and second waiting time values, To order the outputs based on the first measurement and the second measurement, and the first time code and the second time code. A device configured to perform the following. 16. The apparatus according to embodiment 15, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, the environment, or a surgical instrument parameter. 17. The apparatus according to embodiment 15, wherein the processor is further configured to determine a delay constant based on a first latency value and a second latency value. 18. The processor, In the third time, a request is transmitted to the first surgical sensing system and the second surgical sensing system. Based on the third time and the first time, the first waiting time value is determined, Based on the third and second time periods, the second waiting time value is determined, The delay constant is calculated based on the first and second waiting time values. The apparatus according to embodiment 15, further configured to carry out the following. 19. The apparatus according to embodiment 15, further comprising a master time log on which the output is stored. 20. The processor, To generate a visual representation based on the output, Transmitting visual representations to a display, The apparatus according to embodiment 15, further configured to carry out the following. [Examples]

[0246] Example 1. An apparatus for ordering received measurements, comprising a processor, wherein the processor is configured to receive, in a first time, a measurement, the measurement being associated with a surgical sensing system and a communication interface; to obtain a latency value associated with the surgical sensing system and the communication interface; to apply a time code to the received measurement based on the first time and the obtained latency value; and to order outputs based on the received measurement and the time code.

[0247] For example, in Example 1, the measured value may be associated with the surgical sensing system by being measured by the surgical sensing system, and / or the measured value may be associated with the communication interface by being transmitted to the processor by or through the communication interface.

[0248] For example, in Example 1, the latency value may include the delay between the sensed measurement and the measurement received by the processor. The latency value may include the delay before transmitting the measurement. The delay before transmitting the measurement may be the time difference between the sensed measurement and the measurement sent to the processor. The latency value may include the transit time. The transit time may be the time difference between the measurement sent by the processor and the received measurement. The latency value may include a combination of the delay before transmitting the measurement and the transit time. The latency value may be obtained by subtracting one time from another. The latency value may be obtained based on prior knowledge, such as prior knowledge of the transit time between the surgical sensing system and the processor, or prior knowledge of the processing time of the surgical sensing system.

[0249] For example, in Embodiment 1, the processor may determine the timecode by subtracting the acquired latency from a first time. For example, in Embodiment 1, the processor may determine the timecode based on a master time clock. The processor may determine the timecode based on the acquired latency value. The processor may determine the timecode based on the time the data stream was received. The processor may determine the timecode based on a combination of the master time clock and / or the acquired latency. The timecode may be time relative to the master time clock. The timecode may be time relative to real time.

[0250] For example, in Example 1, the output may include a measured value or consist of measured values. The output may include multiple measured values, including a measured value.

[0251] For example, in Embodiment 1, ordering the output based on received measurements and time codes may include ordering the measurements within the output based on the applied time code. The output may include multiple measurements, including the measurement. Ordering the output may include ordering multiple received measurements based on the applied time code. Ordering the output may include ordering multiple received measurements in an order different from the order in which they were received. Ordering the output may include ordering multiple received measurements in an order different from the order in which they were perceived. Ordering the output may include ordering multiple measurements to emulate the order in which the measurements were perceived. The processor may be configured to generate the output. Ordering multiple received measurements based on the applied time code may include ordering the measurements from the minimum time code to the maximum time code. The minimum time code may represent the earliest measurement. The maximum time code may represent the most recent measurement.

[0252] Medical professionals may be interested in analyzing the received data in relation to other data. Biomarker, environmental, and surgical instrument measurements can provide context for each other. Therefore, it is important for medical professionals to know whether measurements from the surgical sensing system are being taken simultaneously or at different times offset from each other. For example, clock drift can occur when the real-time clock of the sensing system becomes asynchronous with the device from which it is transmitting data, or changing data rates or processing delays can cause measurements, or a stream of measurements or data, to become asynchronous. Medical professionals can determine context information from multiple sensing systems when the data received from each sensing system is synchronized with each other. The apparatus of Example 1 enables the use of time codes to compensate for measurement latency, whereby measurements can be accurately ordered and synchronized with other data and accurate context information derived by medical professionals.

[0253] Example 2. The apparatus according to Example 1, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of biomarker, environmental, and surgical instrument parameters.

[0254] For example, in either Example 1 or 2, the measurement may be a measurement of at least one of biomarker, environmental, or surgical instrument parameters. In Example 2, the one or more sensors may be configured to measure the measurement value.

[0255] The apparatus of Example 2 orders biomarker, environmental, or surgical instrument data, thereby enabling it to be safely synchronized with other data and allowing medical professionals to derive context data therefrom. For example, synchronizing heart rate data with blood pressure data may help medical professionals better understand the patient's cardiovascular system status and accurately diagnose any problems or abnormalities.

[0256] Apparatus according to any one of Examples 1 to 2, further comprising a master time log in which the output is stored.

[0257] For example, in any one of Examples 1 to 3, the processor may be further configured to input the output into the master time log. The master time log may be a data structure in the memory. The master time log may store the output and / or measurements and / or associated measurement metadata. The measurement metadata may include a sensing system identification, a first time, a latency value, and a time code.

[0258] The apparatus of Example 3 allows the output to be stored in the master time log while maintaining its order, so that the synchronized data can be viewed again for further analysis.

[0259] Apparatus according to any one of Examples 1 to 3, wherein the surgical sensing system is configured to receive a request to return a measurement value at a second time, select a measurement value based on at least one sensor reading and the second time, and return the measurement value to the apparatus at a third time.

[0260] For example, in Example 4, the second and third times are before the first time. Receiving a request to return a measurement value at the second time may include receiving, from the processor, a request to return a measurement value at the second time. Selecting a measurement value based on at least one sensor reading and the second time may include, for example, selecting the measurement value most recently sensed at the second time, the next measurement value sensed at the second time, and / or the measurement value sensed closest in time to the second time.

[0261] In Example 4, the surgical sensing system may select which measurement to send to the device depending on when the request for measurement data arrives. For example, the system may choose to send the most recent measurement, the next measurement, or the measurement that is temporally closest to the time of the request. This allows the device to better adapt to the specific context in which it is used and to provide the user with accurate data.

[0262] Example 5. The apparatus according to any one of Examples 1 to 4, wherein the processor is further configured to perform the following: transmit a request to a surgical sensing system at a fourth time; and calculate a waiting time value based on the fourth time and the first time.

[0263] For example, in Example 5, the fourth time is before the first time. The request may be a request to send the measured value to the processor. The latency value can be calculated by subtracting the fourth time from the first time, and further by halving the resulting round-trip latency value.

[0264] In Example 5, the apparatus may calculate the measurement delay value, for example, by measuring the round-trip waiting time value. This makes it possible to determine the measurement waiting time even when prior data that would allow the measurement waiting time to be calculated is not available.

[0265] Example 6. The apparatus according to any one of Examples 1 to 5, wherein the processor is further configured to assign extended timestamps to the measured values.

[0266] For example, in Example 6, the extended timestamp may include at least one of the following: an overlay timestamp match, a full blockstamp match, and an allowable timestamp match.

[0267] By using extended timestamps, the apparatus of Example 6 enables the mixing of sensor signals at different data rates. Data originating from a single source, split across multiple computing nodes, can be synchronously fused on multiple hosts or multi-core processors, for example.

[0268] Example 7. The apparatus according to any one of Examples 1 to 6, further comprising a display configured to show a visual representation based on the output.

[0269] For example, in Embodiment 7, the display may be configured to display either the output or the measured value.

[0270] For example, in any one of Examples 1 to 7, the processor may be further configured to generate a visual representation based on the output and transmit the visual representation to a display and / or to a display on which it is to be shown. The visual representation may be based on the output, either by being an output or by displaying a measurement.

[0271] As in Example 7, using a display makes it possible to display the ordered outputs in real time, thereby allowing medical professionals to analyze the ordered outputs.

[0272] Example 8. A method for an apparatus for ordering received measurements, comprising: receiving a measurement at a first time, the measurement being associated with a surgical sensing system and a communication interface; obtaining a latency value associated with the surgical sensing system and the communication interface; applying a time code to the received measurement based on the first time and the obtained latency value; and ordering outputs based on the received measurement and the time code.

[0273] For example, in Example 8, the measured value may be associated with the surgical sensing system by being measured by the surgical sensing system, and / or the measured value may be associated with the communication interface by being transmitted to the device by or through the communication interface.

[0274] For example, in Example 8, the latency value may include the delay between the sensed measurement and the measurement received by the device. The latency value may include the delay before transmitting the measurement. The delay before transmitting the measurement may be the time difference between the sensed measurement and the measurement transmitted to the device. The latency value may include the transit time. The transit time may be the time difference between the transmitted measurement and the measurement received by the device. The latency value may include a combination of the delay before transmitting the measurement and the transit time. Obtaining the latency value may include subtracting one time from another. Obtaining the latency value may include using prior knowledge, such as prior knowledge of transit time between sensing devices or prior knowledge of processing time for a surgical sensing system.

[0275] For example, in Embodiment 8, the method may further include determining the time code by subtracting the acquired latency from a first time. For example, in Embodiment 8, the time code may be determined based on a master time clock. The time code may be determined based on the acquired latency value. The time code may be determined based on the time the data stream was received. The time code may be determined based on a combination of the master time clock and / or the acquired latency. The time code may be time relative to the master time clock. The time code may be time relative to real time.

[0276] For example, in Example 8, the output may include a measured value or consist of measured values. The output may include multiple measured values, including a measured value.

[0277] For example, in Embodiment 8, ordering the output based on received measurements and time codes may include ordering the measurements within the output based on the applied time code. The output may include multiple measurements, including the measurement. Ordering the output may include ordering multiple received measurements based on the applied time code. Ordering the output may include ordering multiple received measurements in an order different from the order in which they were received. Ordering the output may include ordering multiple received measurements in an order different from the order in which they were perceived. Ordering the output may include ordering multiple measurements to emulate the order in which the measurements were perceived. The device may be configured to generate an output. Ordering multiple received measurements based on the applied time code may include ordering the measurements from the minimum time code to the maximum time code. The minimum time code may represent the earliest measurement. The maximum time code may represent the most recent measurement.

[0278] Medical professionals may be interested in analyzing received data in relation to other data. Measurements of biomarkers, the environment, and surgical instruments can provide context to each other. Therefore, it is important for medical professionals to know whether measurements from surgical sensing systems are being taken simultaneously or at different times offset from each other. For example, clock drift can occur when the real-time clock of a sensing system becomes asynchronous with the device from which it transmits data, or a changing data rate or processing delay can asynchronously affect measurements, or a stream of measurements or data. Medical professionals can determine contextual information from multiple sensing systems if the data received from each sensing system is synchronized with each other. The method of Example 8 allows for compensation of measurement latency using timecodes, thereby enabling measurements to be precisely ordered and synchronized with other data and precise contextual information derived by the medical professional.

[0279] Example 9. The method according to Example 8, wherein ordering the output includes entering the output into a master time log.

[0280] For example, in any one of Examples 8 or 9, the master time log may be a data structure in memory. The master time log may store the output and / or measurements and / or associated measurement metadata. The measurement metadata may include a sensing system identifier, a measurement value, a first time, a latency value, and a time code.

[0281] The method of Example 9 allows the output to be stored in the master time log while maintaining its order so that the synchronized data can be reviewed again for further analysis.

[0282] Example 10. The method according to any one of Examples 8 or 9, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, an environment, or a surgical instrument parameter.

[0283] For example, in any one of Examples 8 to 10, the measurement value may be a measurement value of at least one of a biomarker, an environment, or a surgical instrument parameter. In Example 10, the one or more sensors may be configured to measure the measurement value.

[0284] The method of Example 10 orders biomarker, environment, or surgical instrument data, thereby enabling it to be safely synchronized with other data and allowing medical professionals to derive contextual data therefrom. For example, synchronizing heart rate data with blood pressure data may help medical professionals better understand the patient's cardiovascular system status and accurately diagnose any problems or abnormalities.

[0285] Example 11. The method according to any one of Examples 8 to 10, wherein obtaining a waiting time value includes sending a request to a surgical sensing system at a second time and calculating a waiting time value based on the second time and the first time.

[0286] For example, in Example 11, the second time is before the first time. The request may be a request to send the measured value to the processor. The latency value can be calculated by subtracting the second time from the first time, and further by halving the resulting round-trip latency value.

[0287] In Example 11, the method may obtain a measurement latency value by, for example, measuring the round-trip latency. This makes it possible to determine the measurement latency even when prior data that would allow the measurement latency to be calculated is not available.

[0288] Example 12. The method according to any one of Examples 8 to 11, further comprising generating a visual representation based on the output and displaying the visual representation.

[0289] For example, in Example 12, the visual representation may be based on the output, either by being an output itself or by displaying a measured value.

[0290] As shown in Example 12, using a display makes it possible to display the ordered outputs in real time, thereby allowing medical professionals to analyze the ordered outputs.

[0291] Example 13. The method according to any one of Examples 8-12, wherein applying a timecode includes assigning an extended timestamp to a measurement.

[0292] By using extended timestamps, the method of Example 13 enables the mixing of sensor signals at different data rates. Data originating from a single source, split across multiple computing nodes, can be synchronously fused on multiple hosts or multi-core processors, for example.

[0293] Example 14. The method according to Example 13, wherein the extended timestamp comprises at least one of an overlay timestamp match, a full block stamp match, and an allowable timestamp match.

[0294] Example 15. An apparatus for ordering received measurements, comprising a processor configured to perform the method described in any one of Examples 8 to 14.

[0295] Medical professionals may be interested in analyzing received data in relation to other data. Measurements of biomarkers, the environment, and surgical instruments can provide context to each other. Therefore, it is important for medical professionals to know whether measurements from surgical sensing systems are being taken simultaneously or at different times that are offset from each other. For example, clock drift can occur if the real-time clock of a sensing system becomes asynchronous with the device from which it transmits data, or if a changing data rate or processing delay can asynchronously affect measurements, or a stream of measurements or data. Medical professionals can determine contextual information from multiple sensing systems if the data received from each sensing system is synchronized with each other. The apparatus of Example 15 allows for compensation of measurement latency using timecodes, thereby enabling measurements to be precisely ordered and synchronized with other data and precise contextual information derived by the medical professional.

[0296] Example 16. Apparatus for ordering two or more received measurements, comprising a processor, wherein the processor is configured to: receive a first measurement at a first time, the first measurement being associated with a first surgical sensing system and a communication interface; receive a second measurement at a second time, the second measurement being associated with a second surgical sensing system and a communication interface; acquire a first delay value and a second delay value associated with the first and second surgical sensing systems and the communication interface; apply a first time code and a second time code to the first and second measurements based on the first and second times and the first and second delay values; and order outputs based on the first and second measurements and the first and second time codes.

[0297] For example, in Example 16, the first measurement and / or the second measurement may be associated with the first surgical sensing system and / or the second surgical sensing system, respectively, by being measured by the first surgical sensing system and / or the second surgical sensing system. For example, in Example 16, the first measurement and / or the second measurement may be associated with the communication interface by being transmitted to the processor via the communication interface.

[0298] For example, in Embodiment 16, the first and / or second latency values ​​may include the delay between the sensed first and / or second measurement values ​​and the measurement values ​​received by the processor. The first and / or second latency values ​​may include the delay before transmitting the first and / or second measurement values. The delay before transmitting the first and / or second measurement values ​​may be the time difference between the sensed first and / or second measurement values ​​and the first and / or second measurement values ​​transmitted to the processor. The first and / or second latency values ​​may include transit time. Transit time may be the time difference between the transmitted first and / or second measurement values ​​and the first and / or second measurement values ​​received by the processor. The first and / or second latency values ​​may include a combination of the delay before transmitting the first and / or second measurement values ​​and transit time. The first and / or second latency values ​​may be obtained by subtracting one time from another. The first and / or second latency values ​​may be obtained based on prior knowledge, such as prior knowledge of the transit time between the surgical sensing system and the processor, or prior knowledge of the processing time of the surgical sensing system.

[0299] For example, in Embodiment 16, the processor may determine the first time code and / or the second time code by subtracting the acquired latency from the first time. For example, in Embodiment 16, the processor may determine the first time code and / or the second time code based on the master time clock. The processor may determine the first time code and / or the second time code based on the acquired first latency value and / or the second latency value. The processor may determine the first time code and / or the second time code based on the time the data stream was received. The processor may determine the first time code and / or the second time code based on the master time clock and / or a combination of the acquired latency. The first time code and / or the second time code may be time relative to the master time clock. The first time code and / or the second time code may be time relative to real time.

[0300] For example, in Example 16, the output may include a first measurement and / or a second measurement, or may consist of them. The output may include multiple measurements, including a first measurement and / or a second measurement.

[0301] For example, in Embodiment 16, ordering the output based on the received first and second measurements, as well as the first and second time codes, may include ordering the first and second measurements in the output based on the applied first and second time codes. The output may include a plurality of measurements, including the first and / or second measurements. Ordering the output may include ordering a plurality of first received measurements and second received measurements based on the applied time codes. Ordering the output may include ordering the first and second measurements in an order different from the order in which they were received. Ordering the output may include ordering the first and second measurements in an order different from the order in which they were perceived. Ordering the output may include ordering the first and second measurements to emulate the order in which the plurality of measurements were perceived. The processor may be configured to generate the output. Ordering the first and second measurements based on the applied timecode may include ordering the first and second measurements from the minimum timecode to the maximum timecode. The minimum timecode may represent the earliest measurement. The maximum timecode may represent the most recent measurement.

[0302] Medical professionals may be interested in analyzing received data in relation to other data. Measurements of biomarkers, the environment, and surgical instruments can provide context to each other. Therefore, it is important for medical professionals to know whether measurements from surgical sensing systems are being taken simultaneously or at different times offset from each other. For example, clock drift can occur when the real-time clock of a sensing system becomes asynchronous with the device from which it transmits data, or a changing data rate or processing delay can asynchronously affect measurements, or a stream of measurements or data. Medical professionals can determine contextual information from multiple sensing systems if the data received from each sensing system is synchronized with each other. The apparatus of Example 16 allows for compensation of measurement latency using timecodes, thereby enabling measurements from various surgical sensing systems to be precisely ordered and synchronized with other data and precise contextual information derived by the medical professional.

[0303] Example 17. The apparatus according to any one of Examples 1 to 7 or 15, further configured to perform receiving, obtaining a second latency value associated with the second surgical sensing system and communication interface, and applying a second time code to the second measurement based on the second time and the second latency value, wherein the apparatus is configured to perform ordering outputs based on the received measurement and time code, and ordering outputs based on the first measurement and the second measurement, and the first time code and the second time code.

[0304] For example, in Example 17, the first measurement and / or the second measurement may be associated with the first surgical sensing system and / or the second surgical sensing system by being measured by the first surgical sensing system and / or the second surgical sensing system, respectively. For example, in Example 17, the first measurement and / or the second measurement may be associated with the communication interface by being transmitted to the processor via the communication interface.

[0305] For example, in Embodiment 17, the first and / or second latency values ​​may include the delay between the sensed first and / or second measurement values ​​and the measurement values ​​received by the processor. The first and / or second latency values ​​may include the delay before transmitting the first and / or second measurement values. The delay before transmitting the first and / or second measurement values ​​may be the time difference between the sensed first and / or second measurement values ​​and the first and / or second measurement values ​​transmitted to the processor. The first and / or second latency values ​​may include transit time. Transit time may be the time difference between the transmitted first and / or second measurement values ​​and the first and / or second measurement values ​​received by the processor. The first and / or second latency values ​​may include a combination of the delay before transmitting the first and / or second measurement values ​​and transit time. The first and / or second latency values ​​may be obtained by subtracting one time from another. The first and / or second latency values ​​may be obtained based on prior knowledge, such as prior knowledge of the transit time between the surgical sensing system and the processor, or prior knowledge of the processing time of the surgical sensing system.

[0306] For example, in Embodiment 17, the processor may determine the first time code and / or the second time code by subtracting the acquired latency from the first time. For example, in Embodiment 17, the processor may determine the first time code and / or the second time code based on the master time clock. The processor may determine the first time code and / or the second time code based on the acquired first latency value and / or the second latency value. The processor may determine the first time code and / or the second time code based on the time the data stream was received. The processor may determine the first time code and / or the second time code based on the master time clock and / or a combination of the acquired latency. The first time code and / or the second time code may be time relative to the master time clock. The first time code and / or the second time code may be time relative to real time.

[0307] For example, in Example 17, the output may include or consist of a first measurement and / or a second measurement. The output may include multiple measurements, including the first measurement and / or the second measurement.

[0308] For example, in Embodiment 17, ordering the output based on the received first measurement and / or second measurement, and the first timecode and / or second timecode may include ordering the first measurement and / or second measurement in the output based on the applied first timecode and / or second timecode. The output may include multiple measurements, including the first measurement and / or second measurement. Ordering the output may include ordering multiple received measurements based on the applied timecode. Ordering the output may include ordering multiple received measurements in an order different from the order in which they were received. Ordering the output may include ordering multiple received measurements in an order different from the order in which they were perceived. Ordering the output may include ordering multiple measurements to emulate the order in which they were perceived. The processor may be configured to generate the output. Ordering multiple received measurements based on the applied timecode may include ordering the measurements from the minimum timecode to the maximum timecode. The minimum time code may represent the earliest measurement. The maximum time code may represent the most recent measurement.

[0309] Medical professionals may be interested in analyzing received data in relation to other data. Measurements of biomarkers, the environment, and surgical instruments can provide context to each other. Therefore, it is important for medical professionals to know whether measurements from surgical sensing systems are being taken simultaneously or at different times that are offset from each other. For example, clock drift can occur when the real-time clock of a sensing system becomes asynchronous with the device from which it transmits data, or a changing data rate or processing delay can asynchronously affect measurements, or a stream of measurements or data. Medical professionals can determine contextual information from multiple sensing systems if the data received from each sensing system is synchronized with each other. The apparatus of Example 17 allows for compensation of measurement latency using time codes, thereby enabling measurements from various surgical sensing systems to be precisely ordered and synchronized with other data and precise contextual information derived by the medical professional.

[0310] Example 18. The apparatus according to any one of Examples 16 and 17, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, the environment, or a surgical instrument parameter.

[0311] For example, in any one of Examples 16 to 18, the first measurement and / or the second measurement may be a measurement of at least one of a biomarker, environment, or surgical instrument parameter. In Example 18, one or more sensors may be configured to measure the first measurement and / or the second measurement.

[0312] The apparatus of Example 18 sequences biomarker, environmental, or surgical instrument data, thereby enabling it to be securely synchronized with other data and allowing medical professionals to derive contextual data from it. For example, heart rate data may be synchronized with blood pressure data to help medical professionals better understand the state of a patient's cardiovascular system and accurately diagnose any problems or abnormalities.

[0313] Example 19. The apparatus according to any one of Examples 16 to 18, wherein the processor is further configured to determine a delay constant based on a first latency value and a second latency value.

[0314] For example, in Example 19, the delay constant may be the average of the first and second waiting time values.

[0315] In Example 19, the calculation of the delay constant allows for the detection and correction of longer-term desync.

[0316] Example 20. The apparatus according to any one of Examples 16 to 19, wherein the processor is further configured to perform the following: transmit requests to a first surgical sensing system and a second surgical sensing system in a third time; determine a first latency value based on the third time and the first time; determine a second latency value based on the third time and the second time; and optionally calculate a delay constant based on the first latency value and the second latency value.

[0317] For example, in Example 20, the third time is before the first and second times. The request may be a request to send the first and second measurements to the processor. The first latency value may be calculated by subtracting the third time from the first time, and then halving the resulting round-trip latency value. The second latency value may be calculated by subtracting the third time from the second time, and then halving the resulting round-trip latency value.

[0318] In Example 20, the apparatus may calculate the first and second measurement latency values, for example, by measuring the round-trip latency. This allows the measurement latency to be determined even when prior data that would enable the calculation of the measurement latency is not available.

[0319] Example 21. The apparatus according to any one of Examples 16 to 20, further comprising a master time log on which the output is stored.

[0320] For example, in any one of Examples 16 to 21, the processor may be further configured to input the output to a master time log. The master time log may be a data structure in memory. The master time log may store the output, and / or a first measurement and a second measurement, and / or associated first measurement metadata and second measurement metadata. The first measurement metadata and / or second measurement metadata may include a sensing system identifier, a measurement, a first time, a latency value, and a time code.

[0321] The apparatus of Example 21 allows the output to be stored in a master time log while maintaining its order, so that the synchronized data can be viewed again for further analysis.

[0322] Example 22. The apparatus according to any one of Examples 16 to 21, wherein the processor is further configured to perform the tasks of generating a visual representation based on the output and transmitting the visual representation to a display.

[0323] For example, in Example 22, the visual representation may be based on the output, either by being an output or by displaying a measured value. The apparatus may further include a display.

[0324] As in Example 22, using a display makes it possible to display the ordered outputs in real time, thereby allowing medical professionals to analyze the ordered outputs.

[0325] For example, in any one of Examples 1 to 22, the apparatus may further include a surgical sensing system and / or a communication interface.

[0326] [Implementation Method] (1) A device for ordering received measurement values, A processor is provided, and the processor is In the first time, receiving the measurement value, wherein the measurement value is associated with the surgical sensing system and the communication interface. To acquire the waiting time value associated with the surgical sensing system and the communication interface, Based on the first time and the acquired waiting time value, a time code is applied to the received measurement value, Based on the received measurement values ​​and the time codes, the outputs are ordered. A device configured to perform the following. (2) The apparatus according to Embodiment 1, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of biomarkers, the environment, and surgical instrument parameters. (3) The apparatus according to any one of embodiments 1 to 2, further comprising a master time log in which the output is stored. (4) The surgical sensing system In the second time, a request is received to return the aforementioned measurement, Selecting the measurement based on at least one sensor reading and the second time, In the third time, the measured values ​​are returned to the device. The apparatus according to any one of embodiments 1 to 3, configured to carry out the following. (5) The processor In the fourth time, a request is transmitted to the surgical sensing system, The waiting time value is calculated based on the fourth time and the first time. The apparatus according to any one of embodiments 1 to 4, further configured to carry out the following.

[0327] (6) The apparatus according to any one of embodiments 1 to 5, wherein the processor is further configured to assign an extended timestamp to the measurement value. (7) The apparatus according to any one of embodiments 1 to 6, further comprising a display configured to display a visual representation based on the output. (8) A method for an apparatus for ordering received measurements, In the first time period, the measurement is received, and the measurement is associated with the surgical sensing system and the communication interface. To acquire the waiting time value associated with the surgical sensing system and the communication interface, Based on the first time and the acquired waiting time value, a time code is applied to the received measurement value, Based on the received measurement values ​​and the time codes, the outputs are ordered. Methods that include... (9) The method according to embodiment 8, wherein ordering the outputs includes inputting the outputs into a master time log. (10) The method according to any one of embodiments 8 or 9, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, the environment, or a surgical instrument parameter.

[0328] (11) The method according to any one of embodiments 8 to 10, wherein obtaining a waiting time value includes transmitting a request to the surgical sensing system at a second time and calculating the waiting time value based on the second time and the first time. (12) To generate a visual representation based on the output, Displaying the aforementioned visual representation The method according to any one of embodiments 8 to 11, further including the following. (13) The method according to any one of embodiments 8 to 12, wherein applying a timecode includes assigning an extended timestamp to the measurement. (14) The method according to embodiment 13, wherein the extended timestamp comprises at least one of an overlay timestamp match, a full blockstamp match, and an allowable timestamp match. (15) An apparatus for ordering received measurements, comprising a processor configured to perform the method described in any of embodiments 8 to 14.

[0329] (16) A device for ordering two or more received measurements, A processor is provided, and the processor is Receiving a first measurement value at a first time, wherein the first measurement value is associated with a first surgical sensing system and a communication interface. Receiving a second measurement at a second time, wherein the second measurement is associated with the second surgical sensing system and the communication interface. To acquire the first surgical sensing system and the second surgical sensing system, and the first and second waiting time values ​​associated with the communication interface, Applying a first time code and a second time code to the first measured value and the second measured value based on the first time and the second time, and the first waiting time value and the second waiting time value, The outputs are ordered based on the first measurement and the second measurement, and the first time code and the second time code. A device configured to perform the following. (17) The measurement value is a first measurement value, the surgical sensing system is a first surgical sensing system, the waiting time value is a first waiting time value, the time code is a first time code, and the processor is Receiving a second measurement at a second time, wherein the second measurement is associated with the second surgical sensing system and the communication interface. To obtain a second latency value associated with the second surgical sensing system and the communication interface, Applying a second time code to the second measurement based on the second time and the second waiting time value. Further configured to implement, The apparatus according to any one of embodiments 1 to 7 or 15, wherein ordering the outputs based on the received measurement values ​​and the time codes includes ordering the outputs based on the first measurement values ​​and the second measurement values, and the first time codes and the second time codes. (18) The apparatus according to any one of embodiments 16 and 17, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, the environment, or a surgical instrument parameter. (19) The apparatus according to any one of embodiments 16 to 18, wherein the processor is further configured to determine a delay constant based on the first latency value and the second latency value. (20) The processor In the third time, a request is transmitted to the first surgical sensing system and the second surgical sensing system. Based on the third time and the first time, the first waiting time value is determined, Based on the third time and the second time, the second waiting time value is determined, Optionally, the delay constant may be calculated based on the first and second waiting time values. The apparatus according to any one of embodiments 16 to 19, further configured to carry out the following.

[0330] (21) The apparatus according to any one of embodiments 16 to 20, further comprising a master time log in which the output is stored. (22) The apparatus according to any one of embodiments 16 to 21, wherein the processor is further configured to perform the tasks of generating a visual representation based on the output and transmitting the visual representation to a display.

Claims

1. A device for ordering received measurement values, A processor is provided, and the processor is In the first time, a request is sent to the surgical sensing system to return the measured value, In the fourth time, receiving the measurement value, wherein the measurement value is associated with the surgical sensing system and the communication interface. Based on the difference between the first time and the fourth time, a latency value associated with the surgical sensing system and the communication interface is obtained, Based on the fourth time and the acquired waiting time value, a time code is applied to the received measurement value. Based on the received measurement values ​​and the time codes, the outputs are ordered. A device configured to perform the following.

2. The apparatus according to claim 1, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of biomarkers, the environment, and surgical instrument parameters.

3. The apparatus according to any one of claims 1 to 2, further comprising a master time log in which the output is stored.

4. The surgical sensing system, In the second time, a request is received to return the measured value, Selecting the measurement value based on at least one sensor reading and the second time, In the third time, the measured value is returned to the device. The apparatus according to any one of claims 1 to 3, configured to carry out the following.

5. The apparatus according to any one of claims 1 to 4, wherein the processor is further configured to assign an extended timestamp to the measurement value.

6. The apparatus according to any one of claims 1 to 5, further comprising a display configured to display a visual representation based on the output.

7. A method for an apparatus for ordering received measurements, In the first time, a request is sent to the surgical sensing system to return the measured value, In the second time, the measurement is received, and the measurement is associated with the surgical sensing system and the communication interface. Based on the difference between the first time and the second time, a waiting time value associated with the surgical sensing system and the communication interface is obtained, Based on the second time and the acquired waiting time value, a time code is applied to the received measurement value, Based on the received measurement values ​​and the time codes, the outputs are ordered. Methods that include...

8. The method according to claim 7, wherein ordering the outputs includes inputting the outputs into a master time log.

9. The method according to any one of claims 7 or 8, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, the environment, or a surgical instrument parameter.

10. Based on the output, a visual representation is generated, Displaying the aforementioned visual representation The method according to any one of claims 7 to 9, further comprising:

11. The method according to any one of claims 7 to 10, wherein applying a timecode includes assigning an extended timestamp to the measurement.

12. The method according to claim 11, wherein the extended timestamp comprises at least one of an overlay timestamp match, a full blockstamp match, and an allowable timestamp match.

13. An apparatus for ordering received measurements, comprising a processor configured to carry out the method described in any one of claims 7 to 12.

14. A device for ordering two or more received measurement values, A processor is provided, and the processor is In the first time, a request to return the measured value is transmitted to the first surgical sensing system and the second surgical sensing system, In the second time, receiving a first measurement value, wherein the first measurement value is associated with the first surgical sensing system and the first communication interface. In the third time, receiving a second measurement, wherein the second measurement is associated with the second surgical sensing system and the second communication interface. Based on the difference between the first time and the second time, a first waiting time value associated with the first surgical sensing system and the first communication interface is obtained, Based on the difference between the first time and the third time, a second waiting time value associated with the second surgical sensing system and the second communication interface is obtained, Applying a first time code to the first measurement value based on the second time and the first waiting time value, Applying a second time code to the second measurement value based on the third time and the second waiting time value, The outputs are ordered based on the first measurement and the second measurement, and the first time code and the second time code. A device configured to perform the following.

15. The apparatus according to claim 14, wherein the surgical sensing system comprises one or more sensors configured to sense at least one of a biomarker, the environment, or surgical instrument parameters.

16. The apparatus according to claim 14 or 15, wherein the processor is further configured to determine a delay constant based on the first and second waiting values.

17. The apparatus according to any one of claims 14 to 16, further comprising a master time log in which the output is stored.

18. The apparatus according to any one of claims 14 to 17, wherein the processor is further configured to perform the tasks of generating a visual representation based on the output and transmitting the visual representation to a display.

Images