Modification of globally or locally provided surgical information related to surgical procedures

Abstract

Systems, methods, and means are provided whereby a surgical computing device may adjust global or regional surgical information provided by an enterprise cloud server based on at least one criterion. The criteria may include one or more of privacy laws, procedures, techniques, or device availability within the medical facility where the surgical procedure is being performed. The surgical computing device / edge computing device may receive global or regional surgical information associated with the surgical procedure from the enterprise cloud server. The surgical computing device / edge computing device may obtain local surgical information associated with the patient and / or the patient's location. The surgical computing device / edge computing device may adjust or modify at least a portion of the global or regional surgical information associated with the local surgical procedure and / or the patient. The surgical computing device / edge computing device may transmit the adjusted global or regional surgical information to the surgical instrument.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is related to the following concurrently filed applications, the contents of each of which are incorporated herein by reference: Attorney Docket No. END9438USNP1, Invention Title: "A METHOD FOR ADVANCED ALGORITHM SUPPORT". [Background technology]

[0002] Patient care, in general, improves when it is tailored to the individual. Because every person has different needs, surgical and interventional solutions that center every patient's unique journey can represent an efficient and innovative path to healing. At the same time, the high stakes of patient care, particularly the surgical process, often focus on conservative, repeatable activities.

[0003] For example, innovative medical technologies such as advanced surgical assistance computing systems and intelligent surgical instruments may improve approaches to patient care and address the specific needs of healthcare providers.

[0004] The ever-increasing availability of data and computing resources makes non-traditional algorithms, such as machine learning algorithms, a particular technological opportunity in the healthcare system. However, incorporating such non-traditional algorithms into any medical technology presents many challenges. Summary of the Invention [Means for solving the problem]

[0005] According to one embodiment of the present invention, there is provided a method implemented by a processor of a first surgical computing device configured to couple to a surgical device and a second surgical computing device for performing surgical tasks of a surgical procedure, the method including receiving, from the second surgical computing device, parameter values ​​and / or control algorithms for the surgical device based on the surgical tasks of the surgical procedure; acquiring patient information, the patient information including parameters associated with the patient and details of the surgical procedure to be performed / being performed on the patient; adjusting the parameter values ​​and / or control algorithms based on the patient information; and transmitting the adjusted parameter values ​​and / or control algorithms to the surgical device.

[0006] Situations are envisioned in which the second surgical computing device does not have access to the patient information, or portions of the patient information, that the first surgical computing device has access to. By adjusting the received parameter values ​​and / or control algorithms based on the patient information, the first surgical computing device can tailor the received parameter values ​​and / or algorithms to the patient undergoing the surgical procedure. This may provide improved surgical outcomes.

[0007] In one example, due to privacy laws, the second surgical computing device does not have access to at least the patient-identifying portion of the patient information. For example, the first surgical computing device is located within a privacy boundary (e.g., in an operating room, hospital, or protected network) and the second surgical computing device is located outside the privacy boundary. In such a case, when the first surgical computing device transmits patient information to the second surgical computing device for generating parameter values ​​and / or algorithms, the first surgical computing device anonymizes the patient information (e.g., by redacting). Thus, the received parameter values ​​and / or control algorithms are generated based on the anonymized patient information, which may result in more general recommendations than if the patient-identifying data were known to the second surgical computing device. Thus, by adjusting the parameter values ​​and / or control algorithms based on the patient information, the first surgical computing system mitigates the influence of the second surgical computing device, which does not have access to this information to generate the parameter values ​​and / or control algorithms.

[0008] In various jurisdictions, the storage and processing of personal data is regulated, for example, through the General Data Protection Regulation (GDPR) in the European Union and the Data Protection Act 2018 in the UK. Personal data is any information relating to an identified or identifiable natural person. A patient is identifiable if they can be identified directly or indirectly.

[0009] Health data is a special category of personal data that, due to its perceived content, receives a higher level of protection (see Article 9 GDPR or HIPPA Privacy Rule), requiring heightened security considerations. A breach of sensitive personal data may result in the accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or access to, sensitive data, which may have serious human consequences.

[0010] Given the potential risks to fundamental rights and freedoms, in many circumstances the processing of health data is prohibited by data protection regulations unless the patient has given their explicit consent to the processing of their health data for one or more specified purposes.

[0011] Anonymized personal data is not subject to data protection rules. Anonymous information is information that does not relate to an identified or identifiable natural person, or to personal data that has been made anonymous so that the data subject is not or can no longer be identified.

[0012] Thus, privacy laws may mean that the second computing device does not have access to the patient information (or portions thereof) received at the first computing device, and if it does access it (e.g., by receiving it from the first computing device), it may be in anonymized form and therefore lacking specific patient-specific details.

[0013] In one example, the first surgical computing device may send a request for parameter values ​​and / or algorithms, or the second surgical computing device may push this to the first surgical computing device. The second surgical computing device may generate parameter values ​​and / or algorithms using patient information other than patient identification, such as the surgical procedure being / about to be performed and the surgical instruments intended for use for the particular tasks in the surgical procedure. The second surgical computing device may use a database or lookup table to provide recommended parameter values ​​and / or control algorithms for the surgical devices performing the surgical tasks in the surgical procedure. In particular, the second surgical computing device may include a record of the surgical procedure, the steps / tasks performed in the surgical procedure, the surgical devices used to perform each of the steps / tasks, and the optimal parameter values ​​and / or control algorithms for each device when used to perform the steps / tasks.

[0014] In one example, lookup table / database recommendations are generated based on historical data from many past patient procedures, and correlation analysis is performed to determine parameter values ​​and / or control algorithms that maximize the surgical outcome for each device when used to perform those steps / tasks (e.g., result in the best seal line, lower risk of staple failure, lower risk of staple line bleeding, etc.).

[0015] For example, historical data may show that for a stapling procedure, clamp wait time (after clamping and before firing) correlates with seal line outcome (e.g., how well the seal line seals, which may be assessed by the volume of blood loss across the seal line). Thus, the recommended parameters or control algorithms may include an optimal clamp wait time. As another example, the stapler's FTC and FTF may also be correlated with seal line outcome, thus the recommended parameter values ​​and / or control algorithms may also include optimal FTC and FTF values.

[0016] Other factors, such as the type of tissue (e.g., which tissue is being stapled), tissue thickness, the staple cartridge or staples being used, the particular device being used, etc., may also affect the staple line and may be taken into account to generate the parameter values ​​and / or control algorithms that are transmitted to the computing device.

[0017] Similar recommendations can be generated for other surgical instruments, for example, energy devices, and the recommended parameter values ​​and / or algorithms can optimize the power level, frequency of delivered energy, time of energy application, etc. for a given surgical task of a given surgical procedure for a given energy device (and optionally a given blade length, etc.).

[0018] The database / lookup table may contain varying degrees of detail, for example, if the tissue type is known, the device's recommendations may be based on that particular tissue type or tissue characteristics. Thus, the second surgical computing device may generate recommended parameters / algorithms based on the tissue type and tissue characteristics as determined during the surgical procedure (e.g., by processing sensed information or image feeds) or by other patient information such as medical records or treatment plans.

[0019] The parameter values ​​and / or control algorithms that the first surgical computing device receives from the second surgical computing device cannot be based on patient information (or portions thereof) that it does not have access to. Accordingly, the first surgical computing device is configured to modify the parameter values ​​or algorithms to take into account the patient information. This may optimize recommendations for the patient on whom the procedure is being / will be performed. For example, the patient information may include details regarding the patient's BMI, blood pressure, co-morbidities, etc., which may affect the desired control settings of the surgical device. Accordingly, the first surgical computing device may include a database or look-up table that associates patient-identifying parameters with desired adjustments to a given parameter value or algorithm.

[0020] The patient information may also include details about the planned surgical procedure for that patient, such as the surgeon who will perform the procedure. The computing device may be configured to adjust parameter values ​​and / or algorithms in response to this data. For example, based on the surgeon's experience level, certain autonomous operations of the surgical instrument may be activated / deactivated through adjustments to the control algorithms, etc.

[0021] The patient information may also include health data generated during the surgical procedure being performed, such as data from various sensors in the surgical instruments, wearable sensors worn by the patient and / or surgeon, and visualization systems. The first surgical computing device may transmit some of this data to the second surgical computing device for generation of parameter values ​​and / or control algorithms, and may prevent certain portions of the data from being transmitted to the second surgical computing device, where they identify the patient.

[0022] In one example, a video feed or images from a visualization system (e.g., an endoscope) may be transmitted to a second surgical computing device. The images / video may be processed to determine the type of tissue being operated on, the surgical device being used, and the current step in the surgical procedure, from which parameter values ​​and / or control algorithms may be generated.

[0023] Another reason the second computing device may be unable to access the patient information (or a portion thereof) is that there is insufficient bandwidth to transmit the patient information (or the complete patient information) from the first surgical computing device to the second surgical computing device.

[0024] Another reason may be that the surgical task is time-sensitive and faster results can be reached by receiving general parameters and / or algorithms and adjusting them based on patient information at the local computing device (i.e., the first surgical computing device).

[0025] The first surgical computing device may be a hub or edge device located within the privacy boundary with the surgical device, and the second surgical computing device may be an enterprise cloud computing device located outside the privacy boundary.

[0026] Patient information may be exchanged within the privacy-protected boundary without modification, but data transmitted outside the protected boundary may be anonymized so that the data cannot be traced back to the patient, thereby reducing the risk of sensitive health data being leaked into the digital domain. Thus, the systems and methods of the present invention optimize control settings for surgical devices while maintaining data privacy.

[0027] The method may further include updating the surgical device by setting the adjusted parameter values ​​and / or control algorithms.

[0028] In one example, the updates may be automatic, thereby reducing human intervention and potentially reducing human error in incorrectly entering adjusted parameter values ​​and / or algorithms.

[0029] The method may further include receiving an indication of pre-identified parameters or variables of the control algorithm that may be adjusted by a processor, the processor being configured to adjust the pre-identified parameters or variables of the control algorithm based on the patient information.

[0030] The method may further include transmitting a portion of the patient information to a second surgical computing device or enabling the second surgical computing device to access the portion of the patient information, the second surgical computing device configured to generate parameter values ​​and / or control algorithms based on the portion of the patient information.

[0031] The patient information may include a first portion of the patient information that is patient identification and a second portion of the patient information that is not patient identification, and the second computing device is configured to generate the parameter values ​​and / or the control algorithm based only on the second portion of the patient information.

[0032] The first surgical computing device may be located within a privacy-protected boundary or protected network, and the second surgical computing device is located outside the privacy-protected boundary or protected network.

[0033] The received parameter values ​​and / or control algorithms may be based on a regional or global analysis of past procedures of patients who have undergone that surgical procedure.

[0034] The regional or global analysis may correlate variables in the past treatment data with outcomes to generate recommended parameter values ​​or control algorithms that optimize the outcomes for that region or globally. Generally, by using historical data from a vast amount of past patient treatments, it may be possible to derive more reliable correlations and therefore better recommendations for device settings. It should be noted, however, that the first surgical computing device may have specific information about the patient that will result in necessary fine-tuning of the recommendations based on the regional or global surgical information.

[0035] The patient information may include the patient's location, and the second surgical computing device may use the patient's location in a lookup table or database to obtain regional parameter values ​​and / or algorithms for that location. Regional recommendations may be based on common patient demographics for the region, which may result in different control parameters and control algorithms across different regions.

[0036] The method may further include generating a request message requesting the parameter value and / or the control algorithm, sending the request message to a second surgical computing device, and receiving the parameter value and / or the control algorithm in response to the request message.

[0037] The request message may include the local information in a redacted or anonymized form.

[0038] The method may further include receiving image / video data from a visualization device used during the surgical procedure and transmitting the image / video data to a second surgical computing device, wherein the parameter values ​​and / or control algorithms are based on interpretation of the image / video data.

[0039] The request message may be generated based on the occurrence of a trigger event, the trigger event being a transition stage from a first surgical task of the surgical procedure to a second surgical task of the surgical procedure.

[0040] The protected network may be protected based on local privacy laws associated with the patient's location.

[0041] The patient information may include at least one of demographics, a patient's medical treatment, or supplies or inventory for the patient treatment.

[0042] The parameter values ​​and / or control algorithms may be further adjusted based on at least one of privacy laws, procedures, techniques, or device availability within the medical facility where the surgical procedure is being performed.

[0043] The method may further include transmitting the adjusted parameter values ​​and / or control algorithm to a second surgical computing device.

[0044] The second surgical computing device may be configured to update the associated stored parameter values ​​and / or control algorithms with the adjusted parameter values ​​and / or control algorithms.

[0045] According to a further embodiment of the present invention, there is provided a first computing system configured to couple with a surgical device and a second surgical computing device for performing surgical tasks of a surgical procedure, the first surgical computing device comprising a processor configured to perform any of the methods described above.

[0046] According to a further embodiment of the present invention there is provided a computing program which, when executed by a processor, causes the processor to perform any of the methods described above.

[0047] Systems, methods, and means related to modifying global or regional information associated with a surgical procedure may be described herein. A surgical computing device / edge computing device may receive global or regional surgical information associated with a surgical procedure (e.g., one or more surgical tasks of the surgical procedure) from an enterprise cloud server. In one example, the surgical computing device / edge computing device may receive the global or regional surgical information in response to a request message sent by the surgical computing device / edge computing device to the enterprise cloud server. The request message may be generated based on the occurrence of a trigger event.

[0048] The surgical computing device / edge computing device may obtain local surgical information (e.g., from the surgical instrument). The local surgical information may be associated with the patient and / or the patient's location. The local surgical information may include at least one of demographics, local medical procedures, supplies or inventory status, or control algorithms associated with the surgical instrument. The local surgical data may be based on characteristics of the local surgical procedure.

[0049] The surgical computing device / edge computing device may adjust or modify at least a portion of global or regional surgical information associated with the local surgical procedure and / or patient. In one example, adjusting or modifying the portion of the global or regional surgical information may include adjusting or modifying a global control algorithm using at least one local update. In one example, the portion of the global or regional surgical information may be adjusted or modified based on at least one of privacy laws, procedures, techniques, or device availability within the medical facility where the surgical procedure is being performed.

[0050] Adjusting at least a portion of the global or regional surgical information may be based on a neural network analysis of the global or regional surgical information, the local surgical data, and / or the patient-related data. The neural network may be trained using the global or regional surgical information, the local surgical information, and the patient-related surgical information to determine what portions and / or to what extent of the global or regional surgical information should be adjusted.

[0051] The surgical computing device / edge computing device may transmit the adjusted global or regional surgical information to the surgical instrument. In one example, adjusted global or regional control algorithms received from an enterprise server may be transmitted to the surgical instrument. [Brief explanation of the drawings]

[0052] [Figure 1] FIG. 1 is a block diagram of a computer-implemented surgical system. [Figure 2] 1 illustrates an exemplary surgical system in an operating room. [Figure 3] 1 illustrates exemplary surgical hubs paired with various systems. [Figure 4] Illustrates a surgical data network having a set of communicating surgical hubs configured to connect with a set of sensing systems, an environmental sensing system, a set of devices, and the like. [Figure 5] 1 illustrates a logic diagram of a control system for a surgical tool. [Figure 6] 1 illustrates an exemplary surgical system including a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter. [Figure 7A] 1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 7B]1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 7C] 1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 7D] 1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 8A] 1A and 1B show an exemplary supervised learning framework and an exemplary unsupervised learning framework, respectively. [Figure 8B] 1A and 1B show an exemplary supervised learning framework and an exemplary unsupervised learning framework, respectively. [Figure 9] 1 illustrates an example overview of receiving global or regional information and modifying the global or regional information based on local information. [Figure 10] 1 illustrates an example of a message sequence diagram illustrating communication and modification of global or regional information on a local device. [Figure 11] 1 illustrates an example of a relationship between a surgical computing device / edge computing device and a remote server. [Figure 12] 1 shows an example of a flowchart for modifying globally or regionally provided information. DETAILED DESCRIPTION OF THE INVENTION

[0053] 1 is a block diagram of a computer-implemented surgical system 100. An exemplary surgical system, such as surgical system 100, may include one or more surgical systems (e.g., surgical subsystems) 102, 103, 104. For example, surgical system 102 may include a computer-implemented bidirectional surgical system. For example, surgical systems 102, 103, 104 may include a surgical computing system, such as a surgical hub 106 and / or a computing device 116, that communicates with a cloud computing system 108. Cloud computing system 108 may include a cloud server 109 and a cloud storage unit 110.

[0054] The surgical systems 102, 103, 104 may each be computer-enabled surgical instruments and devices. For example, the surgical systems 102, 103, 104 may include a wearable sensing system 111, a human interface system 112, a robotic system 113, one or more intelligent instruments 114, an environmental sensing system 115, and / or others. The wearable sensing system 111 may include one or more devices used to sense aspects of an individual's condition and activity within the surgical environment. For example, the wearable sensing system 111 may include a healthcare provider sensing system and / or a patient sensing system.

[0055] The human interface system 112 may include devices that allow individuals to interact with the surgical systems 102, 103, 104 and / or the cloud computing system 108. The human interface system 112 may include human interface devices.

[0056] The robotic system 113 may include a surgical robotic device, such as a surgical robot. The robotic system 113 may enable robotic surgical procedures. The robotic system 113 may receive information, settings, programming, control, etc. from the surgical hub 106; for example, the robotic system 113 may transmit data, such as sensor data, feedback information, video information, and operation logs, to the surgical hub 106.

[0057] Environmental sensing system 115 may include, for example, one or more devices used to measure one or more environmental attributes, for example, as further described in Figure 2. Robotic system 113 may include multiple devices used to perform a surgical procedure, for example, as further described in Figure 2.

[0058] The surgical system 102 may communicate with a remote server 109, which may be part of a cloud computing system 108. In one example, the surgical system 102 may communicate with the remote server 109 via a networked connection, such as an Internet connection (e.g., business internet service, T3, cable / FIOS networking node, etc.). The surgical system 102 and / or its components may communicate with the remote server 109 via a cellular transmission / reception point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), long term evolution (LTE) or 4G, LTE-Advanced (LTE-A), new radio (NR), or 5G.

[0059] In one example, the surgical hub 106 may facilitate displaying images from a surgical imaging device, such as a laparoscope. The surgical hub 106 may have collaborative interaction with other local systems to facilitate displaying information related to those local systems. The surgical hub 106 may interact with one or more sensing systems 111, 115, one or more intelligent instruments 114, and / or multiple displays. For example, the surgical hub 106 may be configured to collect measurement data from one or more sensing systems 111, 115 and send notification or control messages to one or more sensing systems 111, 115. The surgical hub 106 may send and / or receive information, including notification information, to and / or from a human interface system 112. The human interface system 112 may include one or more human interface devices (HIDs). The surgical hub 106 may send and / or receive notification or control information for audio, display, and / or control information to various devices in communication with the surgical hub.

[0060] For example, the sensing systems 111, 115 may include a wearable sensing system 111 (which may include one or more HCP sensing systems and one or more patient sensing systems) and an environmental sensing system 115. The one or more sensing systems 111, 115 may measure data related to various biomarkers. The one or more sensing systems 111, 115 may measure the biomarkers using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The one or more sensors may measure the biomarkers as described herein using one of many of the following sensing technologies: photoplethysmography, electrocardiography, electroencephalography, colorimetric, impedimentary, potentiometric, amperometric, etc.

[0061] Biomarkers measured by one or more sensing systems 111, 115 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood potential of hydrogen, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, coughing and sneezing, gastrointestinal motility, gastrointestinal imaging, airway bacteria, edema, mental status, sweat, circulating tumor cells, autonomic tone, circadian rhythm, and / or menstrual cycle.

[0062] The biomarkers may relate to physiological systems, which may include, but are not limited to, behavioral and psychological, cardiovascular, renal, dermatological, nervous, gastrointestinal, respiratory, endocrine, immune, oncological, musculoskeletal, and / or reproductive systems. Information from the biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgical system 100. Information from the biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgical system 100 to improve the system and / or improve patient outcomes. One or more sensing systems 111, 115, biomarkers, and physiological systems are described in more detail in U.S. patent application Ser. No. 17 / 156,287 (Attorney Docket No. END9290USNP1), entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS," filed on January 22, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0063] FIG. 2 shows an example of a surgical system 202 in an operating room. As illustrated in FIG. 2, a patient is being operated on by one or more health care professionals (HCPs). The HCPs are monitored by one or more HCP sensing systems 220 worn by the HCPs. The HCPs and the environment surrounding the HCPs may also be monitored by one or more environmental sensing systems including, for example, a set of cameras 221, a set of microphones 222, and other sensors that may be deployed in the operating room. The HCP sensing systems 220 and the environmental sensing systems may communicate with a surgical hub 206, which may then communicate with one or more cloud servers 209 of a cloud computing system 208, as shown in FIG. 1. The environmental sensing systems may be used to measure one or more environmental attributes, such as HCP position within the operating room, HCP movement, ambient noise within the operating room, temperature / humidity within the operating room, etc.

[0064] As illustrated in FIG. 2 , a primary display 223 and one or more audio output devices (e.g., speaker 219) are positioned in the sterile field for visibility to the operator at the operating table 224. Additionally, a visualization / notification tower 226 is positioned outside the sterile field. The visualization / notification tower 226 may include a first non-sterile human interactive device (HID) 227 and a second non-sterile HID 229 facing opposite each other. The HIDs may be displays or displays with touchscreens that allow a human to directly interface with the HIDs. A human interface system guided by the surgical hub 206 may be configured to utilize the HIDs 227, 229, and 223 to coordinate information flow to operators inside and outside the sterile field. In one example, the surgical hub 206 may cause an HID (e.g., primary HID 223) to display notifications and / or information regarding the patient and / or surgical procedure steps. In one example, the surgical hub 206 may prompt and / or receive input from a person in the sterile field or non-sterile area. In one example, the surgical hub 206 may cause the HID to display a snapshot of the surgical site as recorded by the imaging device 230 on the non-sterile HID 227 or 229 while maintaining a live view of the surgical site on the main HID 223. The snapshot on the non-sterile display 227 or 229 may, for example, allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0065] In one aspect, the surgical hub 206 may be configured to send diagnostic input or feedback entered by a non-sterile operator at the visualization tower 226 to a primary display 223 in the sterile field for viewing by a sterile operator at the operating table. In one example, the input may be in the form of a correction to a snapshot displayed on the non-sterile display 227 or 229, which may be sent by the surgical hub 206 to the primary display 223.

[0066] 2, a surgical instrument 231 is used as part of a surgical system 202 in a surgical procedure. A hub 206 may be configured to coordinate information flow to the display of the surgical instrument 231, as described, for example, in U.S. Patent Application Publication No. 2019-0200844(A1), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 226 can be sent by the hub 206 to a surgical instrument display in the sterile field, where it can be viewed by the operator of the surgical instrument 231. Exemplary surgical instruments suitable for use with surgical system 202 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety.

[0067] FIG. 2 illustrates an example of a surgical system 202 being used to perform a surgical procedure on a patient lying on an operating table 224 in an operating room 235. A robotic system 234 may be used as part of the surgical system 202 in the surgical procedure. The robotic system 234 may include a surgeon's console 236, a patient side cart 232 (surgical robot), and a surgical robot hub 233. The patient side cart 232 can manipulate at least one detachably coupled surgical tool 237 through a minimally invasive incision within the patient's body while the surgeon views the surgical site through the surgeon's console 236. Images of the surgical site can be acquired by a medical imaging device 230, which can be manipulated by the patient side cart 232 to orient the imaging device 230. The robotic hub 233 can be used to process the images of the surgical site and then display them to the surgeon through the surgeon's console 236.

[0068] Other types of robotic systems can be readily adapted for use with surgical system 202. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0069] Various examples of cloud-based analytics methods that may be performed by cloud computing system 208 and that are suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,403), the disclosure of which is incorporated herein by reference in its entirety.

[0070] In various embodiments, the imaging device 230 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors.

[0071] The optics of the imaging device 230 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0072] The one or more illumination sources can be configured to emit electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or luminous spectrum, is the portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and is sometimes referred to as visible light or simply light. The human eye typically responds to wavelengths in air between about 380 nm and about 750 nm.

[0073] The invisible spectrum (e.g., non-radiative spectrum) is the portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which constitutes invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which constitutes invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0074] In various aspects, imaging device 230 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0075] The imaging device may use multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and UV light. Spectral imaging allows for the extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is complete to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical equipment is necessary during any surgery. The strict hygiene and sterilization conditions required in an "operating room," i.e., an operating room or procedure room, require the highest possible sterility of all medical devices and equipment. Part of the sterilization process is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 230 and its accessories and components. It will be understood that the sterile field may be considered a specific area, such as in a tray or on a sterile towel, that is deemed free of microorganisms, or the sterile field may be considered the area immediately surrounding the patient being prepared for the surgical procedure. The sterile field may include properly clothed and hand-washed team members, as well as all equipment and fixtures in the area.

[0076] The wearable sensing system 211 illustrated in FIG. 1 may include one or more sensing systems, such as an HCP sensing system 220 as shown in FIG. 2. The HCP sensing system 220 may include a sensing system that monitors and detects a set of physical conditions and / or a set of physiological conditions of a healthcare professional (HCP). An HCP may generally be one or more healthcare professionals assisting a surgeon or other healthcare provider. In one example, the sensing system 220 may measure a set of biomarkers to monitor the HCP's heart rate. In one example, the sensing system 220 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movement and / or shaking and determine tremor magnitude and frequency. The sensing system 220 may transmit measurement data associated with the set of biomarkers and data associated with the surgeon's physical condition to the surgical hub 206 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 206. For example, the environmental sensing device may include a camera 221 for detecting the HCP's hand / body position. The environmental sensing devices may include a microphone 222 for measuring ambient noise within the operating room. Other environmental sensing devices may include devices such as a thermometer to measure temperature and a hygrometer to measure ambient humidity within the operating room. The surgical hub 206, alone or in communication with a cloud computing system, may use the surgeon biomarker measurement data and / or environmental sensing information to modify the control algorithms of handheld instruments or the average latency of a robotic interface, for example, to minimize tremor. In one example, the HCP sensing system 220 may measure one or more surgeon biomarkers associated with the HCP and transmit measurement data associated with the surgeon biomarkers to the surgical hub 206. The HCP sensing system 220 may use one or more RF protocols, such as Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), or Wi-Fi, to communicate with the surgical hub 206.Surgeon biomarkers may include one or more of stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels associated with the surgeon or patient, surgeon and / or staff movement, surgeon and / or staff attention level, etc.

[0077] The surgical hub 206 may use the surgeon biomarker measurement data associated with the HCP to adaptively control one or more surgical instruments 231. For example, the surgical hub 206 may send a control program to the surgical instrument 231 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 206 may send the control program based on situational awareness and / or context regarding the importance or criticality of the task. The control program may instruct the instrument to change operation to provide more control when control is needed.

[0078] FIG. 3 illustrates an exemplary surgical system 302 having a surgical hub 306. The surgical hub 306 may be paired with a wearable sensing system 311, an environmental sensing system 315, a human interface system 312, a robotic system 313, and an intelligent instrument 314 via a modular control. The hub 306 includes a display 348, an imaging module 349, a generator module 350, a communications module 356, a processor module 357, a storage array 358, and an operating room mapping module 359. In certain embodiments, as illustrated in FIG. 3, the hub 306 further includes a smoke evacuation module 354 and / or a suction / irrigation module 355. The various modules and systems may be connected to the modular control either directly via a router or via the communications module 356. Operating room devices may be coupled to cloud computing resources and data storage via the modular control. The human interface system 312 may include a display subsystem and a notification subsystem.

[0079] The modular controller may be coupled to a non-contact sensor module. The non-contact sensor module may use ultrasonic, laser-type, and / or similar non-contact measurement devices to measure the dimensions of the operating room and generate a map of the operating room. Other distance sensors may be used to determine the boundaries of the operating room. As described in U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," which is incorporated herein by reference in its entirety, under the heading "Surgical Hub Spatial Awareness Within an Operating Room," an ultrasound-based non-contact sensor module may scan the operating room by transmitting bursts of ultrasound and receiving echoes as the bursts reflect off the exterior walls of the operating room. The sensor module may be configured to determine the size of the operating room and adjust Bluetooth pairing distance limits. For example, a laser-based non-contact sensor module can scan an operating room by transmitting laser light pulses, receiving laser light pulses that reflect off the exterior walls of the operating room, and comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust Bluetooth pairing distance limits.

[0080] During surgical procedures, the application of energy to tissue for sealing and / or cutting is generally associated with smoke evacuation, aspiration of excess fluid, and / or irrigation of tissue. Fluid, power, and / or data lines from different sources often become tangled during surgical procedures. Addressing this issue can result in valuable time being lost during a surgical procedure. Untangling the lines may require unplugging them from their corresponding modules, which may require resetting the modules. The hub modular enclosure 360 ​​reduces the frequency of such line tangles by providing a unified environment for managing power, data, and fluid lines. Aspects of the present disclosure present a surgical hub 306 for use in surgical procedures involving the application of energy to tissue at a surgical site.

[0081] The surgical hub 306 includes a hub enclosure 360 ​​and a combination generator module slidably received within a docking station of the hub enclosure 360. The docking station includes data and power contacts. The combination generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed within a single unit. In one aspect, the combination generator module also includes a smoke evacuation component, at least one energy delivery cable for connecting the combination generator module to a surgical instrument, at least one smoke evacuation component configured to evacuate smoke, fluid, and / or particulates generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the smoke evacuation component. In one aspect, the fluid line may be a first fluid line and a second fluid line may extend from the remote surgical site to the suction and irrigation module 355, which is slidably received within the hub enclosure 360. In one aspect, the hub enclosure 360 ​​may include a fluid interface.

[0082] Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another, different energy type may be more beneficial for sealing tissue. For example, a bipolar generator may be used to seal tissue, while an ultrasonic generator may be used to cut the sealed tissue. Aspects of the present disclosure present a solution in which a hub modular enclosure 360 ​​is configured to house different generators and facilitate bidirectional communication between them. The hub modular enclosure 360 ​​may allow for quick removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue and a first docking station including a first docking port including first data contacts and first power contacts, wherein the first energy generator module is slidably movable into electrical engagement with the power contacts and the data contacts, and the first energy generator module is slidably movable out of electrical engagement with the first power contacts and the first data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy different from the first energy for application to tissue and a second docking station including a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable into electrical engagement with the power contacts and the data contacts, and the second energy generator module is slidably movable out of electrical engagement with the second power contacts and the second data contacts. In addition, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port configured to facilitate communication between the first energy generator module and the second energy generator module.Referring to FIG. 3 , an aspect of the disclosure is presented regarding a hub modular enclosure 360 ​​that allows for modular integration of a generator module 350, a smoke evacuation module 354, and a suction / irrigation module 355. The hub modular enclosure 360 ​​further facilitates bidirectional communication between the modules 359, 354, and 355. The generator module 350 can comprise integrated monopolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably insertable into the hub modular enclosure 360. The generator module 350 can be configured to connect to a monopolar device 351, a bipolar device 352, and an ultrasonic device 353. Alternatively, the generator module 350 may comprise a series of monopolar, bipolar, and / or ultrasonic generator modules that interact through the hub modular enclosure 360. The hub modular enclosure 360 ​​may be configured to facilitate the insertion of multiple generators and bidirectional communication between generators docked to the hub modular enclosure 360 ​​so that the multiple generators act as a single generator.

[0083] FIG. 4 illustrates a surgical data network having a set of communication hubs configured to connect a set of sensing systems, environmental sensing systems, and other modular devices located in one or more operating rooms, patient recovery rooms, or rooms within a medical facility specially equipped for surgery to a cloud in accordance with at least one embodiment of the present disclosure.

[0084] 4, surgical hub system 460 may include a modular communications hub 465 configured to connect modular devices located within a medical facility to a cloud-based system (e.g., a cloud computing system 464, which may include a remote server 467 coupled to remote storage 468). The modular communications hub 465 and devices may be connected in a room within the medical facility specially equipped for surgery. In one aspect, modular communications hub 465 may include a network hub 461 and / or a network switch 462 in communication with a network router 466. Modular communications hub 465 may be coupled to a local computer system 463 to provide local computer processing and data manipulation.

[0085] The computer system 463 may include a processor and a network interface. The processor may be coupled to a communication module, storage, memory, non-volatile memory, and input / output (I / O) interfaces via a system bus. The system bus can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures, including, but not limited to, a 9-bit bus, an Industrial Standard Architecture (ISA), a Micro-Channel Architecture (MSA), an Extended ISA (EISA), an Intelligent Drive Electronics (IDE), a VESA Local Bus (VLB), a Peripheral Component Interconnect (PCI), a USB, an Advanced Graphics Port (AGP), a Personal Computer Memory Card International Association (PCMCIA), a Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0086] The processor may be any single-core or multi-core processor, such as that known by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, including on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle serial random access memory (SRAM), internal read-only memory (ROM) loaded with StellarisWare® software, 2 KB of electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more Quadrature Encoder Input (QEI) analog, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which are available in the product data sheet.

[0087] In one example, the processor may include a safety controller, which includes two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller may be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0088] It should be understood that the computer system 463, in a suitable operating environment, may include software that acts as an intermediary between the described users and the basic computer resources. Such software may include an operating system. The operating system, which may be stored on disk storage, may function to control and allocate resources of the computer system. System applications may take advantage of resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0089] A user may input commands or information into the computer system 463 through input devices coupled to the I / O interface. Input devices may include, but are not limited to, pointing devices such as a mouse, trackball, stylus, or touchpad; keyboards; microphones; joysticks; gamepads; satellite dishes; scanners; TV tuner cards; digital cameras; digital video cameras; webcams; and the like. These and other input devices connect to the processor through the system bus via interface ports. Interface ports include, for example, serial ports, parallel ports, game ports, and USB. Output devices use some of the same types of ports as input devices. Thus, for example, a USB port may be used to provide input to the computer system 463 and to output information from the computer system 463 to an output device. Output adapters may be provided to illustrate that there may be several output devices, such as monitors, displays, speakers, and printers, among other output devices that may require special adapters. Output adapters may include, by way of example and not limitation, video and sound cards that provide a means of connection between the output device and the system bus. It should be noted that other devices and / or systems of devices, such as remote computers, may provide both input and output capabilities.

[0090] The computer system 463 may operate in a networked environment using logical connections to one or more remote computers, such as a cloud computer, or a local computer. The remote cloud computer may be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device, or other common network node, but typically includes many or all of the elements described with respect to a computer system. For simplicity, only a memory storage device is illustrated with the remote computer. The remote computer may be logically connected to the computer system through a network interface and then physically connected through a communications connection. The network interface may encompass communications networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies may include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, etc. WAN technologies may include, but are not limited to, point-to-point links, circuit-switched networks such as Integrated Services Digital Networks (ISDN) and its variants, packet-switched networks, and Digital Subscriber Lines (DSL).

[0091] In various examples, computer system 463 may include an image processor, image processing engine, media processor, or any specialized digital signal processor (DSP) used to process digital images. The image processor may use parallel computing using Single Instruction, Multiple Data (SIMD) or Multiple Instruction, Multiple Data (MIMD) techniques to increase speed and efficiency. The digital image processing engine may perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0092] The communications connection may refer to the hardware / software used to connect the network interface to the bus. The communications connection is shown internal to computer system 463 for clarity of illustration, but can also be external to computer system 463. For purposes of illustration only, the hardware / software required to connect to the network interface may include internal and external technologies such as modems, including regular telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0093] The surgical data network associated with the surgical hub system 460 may be configured as passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, allowing it to go from one device (or segment) to another and to cloud computing resources. An intelligent surgical data network includes additional features that allow traffic to pass through the monitored surgical data network and configure each port within the network hub 461 or network switch 462. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0094] Modular devices 1a-1n located in an operating room may be coupled to a modular communication hub 465. Network hub 461 and / or network switch 462 may be coupled to a network router 466 to connect devices 1a-1n to a cloud computing system 464 or a local computer system 463. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to local computer system 463 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be coupled to network switch 462. Network switch 462 may be coupled to network hub 461 and / or to network router 466 to connect devices 2a-2m to a cloud 464. Data associated with devices 2a-2m may be transferred to cloud computing system 464 via network router 466 for data processing and manipulation. Data associated with the devices 2a-2m may also be transferred to a local computer system 463 for local data processing and manipulation.

[0095] 4, a computing system such as surgical hub system 460 may include a modular communications hub 465 configured to connect modular devices (e.g., surgical devices) located within a medical facility to a cloud-based system (e.g., a cloud computing system 464 that may include a remote server 467 coupled to remote storage 468). The modular communications hub 465 and devices may be connected in a room within the medical facility specially equipped for surgery. In one aspect, the modular communications hub 465 may include a network hub 461 and / or a network switch 462 in communication with a network router 466. The modular communications hub 465 may be coupled to a local computer system (e.g., a computing device) to provide local computer processing and data manipulation.

[0096] FIG. 5 illustrates a logic diagram of a surgical instrument or surgical tool control system 520 according to one or more embodiments of the present disclosure. The surgical instrument or surgical tool may be configurable. The surgical instrument may include surgical fasteners specific to the procedure at hand, such as imaging devices, surgical staplers, energy devices, endocutter devices, etc. For example, the surgical instrument may include any of a power stapler, a power stapler generator, an energy device, an advanced energy device, an advanced energy jaw device, an endocutter clamp, an energy device generator, an operating room imaging system, a smoke evacuation device, a suction irrigation device, an insufflation system, etc. The system 520 may include control circuitry. The control circuitry may include a microcontroller 521 with a processor 522 and a memory 523. For example, one or more of sensors 525, 526, 527 provide real-time feedback to the processor 522. A motor 530 driven by a motor driver 529 operably couples a longitudinally movable displacement member to drive the I-beam knife element. The tracking system 528 may be configured to determine the position of the longitudinally movable displacement member. The position information may be provided to the processor 522, which may be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the firing member, firing bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control the firing of the I-beam, the movement of the obturator tube, the rotation of the shaft, and the articulation. The display 524 may display various operating states of the instrument and may also include touch screen functionality for data entry. Information displayed on the display 524 may be overlaid with images acquired via the endoscopic imaging module.

[0097] Microcontroller 521 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex manufactured by Texas Instruments. In one aspect, main microcontroller 521 may be, for example, an LM4F230H5QR ARM Cortex-M4F Processor Core available from Texas Instruments, with on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2 KB of EEPROM, one or more PWM modules, one or more QEI analog, and / or one or more 12-bit ADCs with 12 analog input channels, details of which are available in the product datasheet.

[0098] The microcontroller 521 may include a safety controller, which includes two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller can be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0099] The microcontroller 521 may be programmed to perform various functions, such as precise control over the speed and position of the knife and articulation system. In one embodiment, the microcontroller 521 may include a processor 522 and memory 523. The electric motor 530 may be a brushed direct current (DC) motor with a gearbox and mechanical linkage to the articulation or knife system. In one embodiment, the motor driver 529 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers may be easily substituted for use in the tracking system 528 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication No. 2017 / 0296213, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," published October 19, 2017, which is incorporated herein by reference in its entirety.

[0100] The microcontroller 521 may be programmed to provide precise control over the velocity and position of the displacement members and articulation system. The microcontroller 521 may be configured to calculate a response in the microcontroller 521 software. The calculated response may be compared to the measured response of the actual system to obtain an "observed" response, which is used for actual feedback decision-making. The observed response may be a suitably adjusted value that balances the smooth, continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0101] The motor 530 may be controlled by a motor driver 529 and may be used by the surgical instrument or tool firing system. In various forms, the motor 530 may be a brushed DC drive motor having a maximum rotational speed of approximately 25,000 RPM. In some examples, the motor 530 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 529 may include, for example, an H-bridge driver including field-effect transistors (FETs). The motor 530 may be powered by a power supply assembly releasably attached to the handle assembly or tool housing to provide control power to the surgical instrument or tool. The power supply assembly may include a battery, which may include multiple battery cells connected in series, that may be used as a power source to power the surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be lithium-ion batteries, which may be connectable to and detachable from the power supply assembly.

[0102] The motor driver 529 may be the A3941, available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with external N-channel power metal-oxide semiconductor field-effect transistors (MOSFETs) specifically designed for inductive loads, such as brushed DC motors. The driver 529 may include an intrinsic charge pump regulator, which provides full (>10V) gate drive for battery voltages up to 7V, allowing the A3941 to operate with reduced gate drive down to 5.5V. A bootstrap capacitor may be used to provide the required battery supply voltage above the N-channel MOSFET. An internal charge pump for the high-side drive allows DC (100% duty cycle) operation. The full-bridge may be driven in fast or slow decay mode using diode or synchronous rectification. In slow decay mode, current recirculation is possible through either the high-side or low-side FET. The power FETs may be protected from shoot-through by a resistor-adjustable dead time. Integrated diagnostics indicate undervoltage, overtemperature, and power bridge faults and can be configured to protect the power MOSFETs under most short circuit conditions. Other motor drivers may be easily substituted for use in tracking system 528 with an absolute positioning system.

[0103] The tracking system 528 may include controlled motor drive circuitry including a position sensor 525 according to one aspect of the present disclosure. The position sensor 525 for the absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member including a rack of drive teeth for meshing engagement with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a firing member that may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a firing bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may be used generally to refer to any movable member of a surgical instrument or tool, such as a drive member, firing member, firing bar, I-beam, or any element that can be displaced. In one aspect, a longitudinally movable drive member may be coupled to a firing member, firing bar, and I-beam. Thus, the absolute positioning system may actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member may be coupled to any suitable position sensor 525 for measuring linear displacement. Thus, the longitudinally movable drive member, firing member, firing bar, or I-beam, or combinations thereof, may be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor.The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system with a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system with a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system with a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system with a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0104] The electric motor 530 may include a rotatable shaft operably interfaced with a gear assembly mounted for meshing engagement with a set of drive teeth, or rack, on the displacement member. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 525 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor arrangement may be connected to a linear actuator by a rack-and-pinion arrangement or to a rotary actuator by a spur gear or other connection. A power source may provide power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member with a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable firing member, a firing bar, an I-beam, or a combination thereof.

[0105] One revolution of the sensor element associated with the position sensor 525 may correspond to a linear longitudinal displacement d1 of the displacement member, where d1 is the linear longitudinal distance the displacement member travels from point "a" to point "b" after one revolution of the sensor element coupled to the displacement member. The sensor arrangement may be connected via a gear reduction such that the position sensor 525 completes one or more revolutions for a full stroke of the displacement member. The position sensor 525 may complete multiple revolutions for a full stroke of the displacement member.

[0106] A series of switches (where n is an integer greater than 1) may be used alone or in combination with a gear reduction to provide a unique position signal for two or more revolutions of the position sensor 525. The state of the switches may be fed back to the microcontroller 521, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 525 is provided to the microcontroller 521. The position sensor 525 of the sensor arrangement may comprise a magnetic sensor, an analog rotary sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0107] The position sensor 525 may comprise any number of magnetic sensing elements, such as magnetic sensors classified according to whether they measure the total magnetic field or vector components of the magnetic field. The technologies used to produce both types of magnetic sensors may encompass many aspects of physics and electronics. Technologies used to sense magnetic fields may include search coils, fluxgates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetoimpedance, magnetostrictive / piezoelectric composites, magnetodiodes, magnetotransistors, optical fiber, magneto-optical, and microelectromechanical systems-based magnetic sensors, among others.

[0108] The position sensor 525 of the tracking system 528 with an absolute positioning system may comprise a magnetic rotary absolute positioning system. The position sensor 525 may be implemented as an AS5055EQFT single-chip magnetic rotary position sensor available from Austria Microsystems, AG. The position sensor 525 interfaces with the microcontroller 521 to provide the absolute positioning system. The position sensor 525 may be a low-voltage, low-power component and may include four Hall-effect elements in an area of ​​the position sensor 525 that may be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on-chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement simple, efficient algorithms for calculating hyperbolic and trigonometric functions, requiring only addition, subtraction, bit shifting, and table lookup operations. The angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 521 through a standard serial communications interface, such as a serial peripheral interface (SPI) interface. The position sensor 525 may provide 12-bit or 14-bit resolution and may be an AS5055 chip provided in a small QFN 16-pin 4x4x0.85mm package.

[0109] The tracking system 528, which comprises an absolute positioning system, may include and / or be programmed to implement a feedback controller, such as a PID, state feedback, and adaptive controller. The power supply converts the signal from the feedback controller into a physical input to the system, in this case a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 525, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors may include sensor arrangements such as those described in U.S. Pat. No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combination circuitry to combine the calculated response with the measured response using algorithms such as weighted averages and theoretical control loops that drive the calculated response towards the measured response. The calculated response of the physical system may take into account properties such as mass, inertia, viscous friction, induced drag, etc., in order to predict what the state and output of the physical system will be given knowledge of the input.

[0110] The absolute positioning system may provide the absolute position of the displacement member upon powering up of the instrument without forcing the displacement member to retract or advance to a reset (zero or home) position, as may be required with conventional rotary encoders that simply count the number of forward or backward steps taken by the motor 530 to infer the position of the device actuator, drive bar, knife, etc.

[0111] Sensor 526, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as the amplitude of strain exerted on the anvil during clamping, which can be indicative of the closure force applied to the anvil. The measured strain may be converted to a digital signal and provided to processor 522. Alternatively, or in addition to sensor 526, sensor 527, such as a load sensor, can measure the closure force applied to the anvil by the closure drive system. For example, sensor 527, such as a load sensor, can measure the firing force applied to the I-beam during the firing stroke of the surgical instrument or tool. The I-beam is configured to engage a wedge-shaped sled, which is configured to cam the staple driver upward and drive the staples into deforming contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 531 can be used to measure the current drawn by motor 530. The force required to advance the firing member may correspond, for example, to the current drawn by motor 530. The measured force may be converted to a digital signal and provided to processor 522.

[0112] For example, a strain gauge sensor 526 can be used to measure the force applied to tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to grasped tissue by the end effector may include a strain gauge sensor 526, such as, for example, a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one aspect, the strain gauge sensor 526 can measure the amplitude or magnitude of strain exerted on the jaw members of the end effector during clamping, which can indicate tissue compression. The measured strain can be converted to a digital signal and provided to the processor 522 of the microcontroller 521. A load sensor 527 can measure the force used to operate the knife element, for example, to cut tissue captured between the anvil and the staple cartridge. A magnetic field sensor can be used to measure the thickness of the captured tissue. The magnetic field sensor's measurements can also be converted to a digital signal and provided to the processor 522.

[0113] Measurements of tissue compression, tissue thickness, and / or force required to close the end effector on the tissue, measured by sensors 526, 527, respectively, may be used by microcontroller 521 to characterize a selected position of the firing member and / or a corresponding value of firing member velocity. In one example, memory 523 may store techniques, equations, and / or look-up tables that may be used by microcontroller 521 during the evaluation.

[0114] The surgical instrument or tool control system 520 may also include wired or wireless communication circuitry for communicating with a surgical hub, such as surgical hub 460, as shown in FIG.

[0115] FIG. 6 illustrates an exemplary surgical system 680 according to the present disclosure and may include a surgical instrument 682 that can communicate with a console 694 or a portable device 696 through a local area network 692 and / or a cloud network 693 via wired and / or wireless connections. The console 694 and the portable device 696 may be any suitable computing devices. The surgical instrument 682 may include a handle 697, an adapter 685, and a loading unit 687. The adapter 685 releasably couples to the handle 697, and the loading unit 687 releasably couples to the adapter 685 such that the adapter 685 transmits force from the drive shaft to the loading unit 687. The adapter 685 or the loading unit 687 may include a force gauge (not explicitly shown) disposed therein to measure force applied to the loading unit 687. The loading unit 687 may include an end effector 689 having a first jaw 691 and a second jaw 690. The loading unit 687 may be an in vivo loading or multi-firing loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without having to remove the loading unit 687 from the surgical site to reload it.

[0116] The first jaw 691 and the second jaw 690 may be configured to clamp tissue therebetween, fire fasteners through the clamped tissue, and sever the clamped tissue. The first jaw 691 may be configured to fire at least one fastener multiple times, or may be configured to contain a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) that may be fired two or more times before being replaced. The second jaw 690 may include an anvil that deforms or otherwise secures fasteners as they are ejected from the multi-fire fastener cartridge.

[0117] The handle 697 may include a motor coupled to the drive shaft to affect rotation of the drive shaft. The handle 697 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, a touch screen, and any other suitable input mechanism or user interface that may be engaged by a clinician to activate the motor.

[0118] The control interface of the handle 697 may be in communication with a controller 698 of the handle 697 to selectively activate the motors to affect rotation of the drive shaft. The controller 698 may be disposed within the handle 697 and may be configured to receive input from the control interface and adapter data from the adapter 685 or loading unit data from the loading unit 687. The controller 698 may analyze the input from the control interface and data received from the adapter 685 and / or loading unit 687 to selectively activate the motors. The handle 697 may also include a display viewable by a clinician while using the handle 697. The display may be configured to display portions of the adapter or loading unit data before, during, or after firing of the instrument 682.

[0119] The adapter 685 may include an adapter identification device 684 disposed therein, and the loading unit 687 may include a loading unit identification device 688 disposed therein. The adapter identification device 684 may be in communication with a controller 698, and the loading unit identification device 688 may be in communication with the controller 698. It will be appreciated that the loading unit identification device 688 may be in communication with the adapter identification device 684, which relays or passes communications from the loading unit identification device 688 to the controller 698.

[0120] The adapter 685 may also include multiple sensors 686 (one shown) disposed about its periphery to detect various conditions of the adapter 685 or the environment (e.g., whether the adapter 685 is connected to the loading unit, whether the adapter 685 is connected to the handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain on the drive shaft, the temperature within the adapter 685, the number of times the adapter 685 has been fired, the peak force of the adapter 685 during firing, the total amount of force applied to the adapter 685, the peak retract force of the adapter 685, the number of times the adapter 685 has been paused during firing, etc.). The multiple sensors 686 may provide input to the adapter identification device 684 in the form of data signals. The data signals of the multiple sensors 686 may be stored in the adapter identification device 684 or may be used to update adapter data stored in the adapter identification device 684. The data signals of the multiple sensors 686 may be analog or digital. The multiple sensors 686 may include a force gauge that measures the force exerted on the loading unit 687 during firing.

[0121] The handle 697 and adapter 685 may be configured to interconnect the adapter identification device 684 and the loading unit identification device 688 with the controller 698 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface may be a contactless electrical interface for wirelessly transmitting (e.g., inductively transmitting) energy and signals therebetween. It is also contemplated that the adapter identification device 684 and the controller 698 may communicate wirelessly with each other via a wireless connection that is separate from the electrical interface.

[0122] The handle 697 may include a transceiver 683 configured to transmit instrument data from the controller 698 to other components of the system 680 (e.g., the LAN 20292, the cloud 693, the console 694, or the portable device 696). The controller 698 may also transmit instrument data and / or measurement data associated with one or more sensors 686 to the surgical hub. The transceiver 683 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 670. The transceiver 683 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 680. For example, the controller 698 may transmit instrument data to the console 694, including the serial number of an attached adapter (e.g., adapter 685) attached to the handle 697, the serial number of a loading unit (e.g., loading unit 687) attached to the adapter 685, and the serial number of a multi-fire fastener cartridge loaded in the loading unit. The console 694 may then return data associated with the attached cartridge, loading unit, and adapter (e.g., cartridge data, loading unit data, or adapter data), respectively, to the controller 698. The controller 698 may display a message on a local instrument display or may send a message via the transceiver 683 to the console 694 or portable device 696, respectively, to display the message on the display 695 or portable device screen.

[0123] 7A illustrates a surgical system 700 that may include a matrix of surgical information. This surgical information may include any discrete atoms of information related to the procedure. Generally described, such surgical information may include information related to the context and scope of the procedure itself (e.g., healthcare information 728). Such information may include data such as, for example, procedure data and patient record data. The procedure data and / or patient record data may be associated with an associated healthcare data system 716 that is in communication with the surgical hub 704.

[0124] The surgical information may include information related to the configuration and / or control of devices being used in the procedure (e.g., device operation information 729). Such device operation information 729 may include information regarding the initial configuration of a surgical device. Device operation information 729 may include information regarding changes to the configuration of a surgical device. Device operation information 729 may include information regarding controls sent from the surgical hub 704 to the device and the information flow associated with such controls.

[0125] Surgical information may include information generated during the surgery itself (e.g., surgical information 727). Such surgical information 727 may include any information generated by a surgical data source 726. The data source 726 may include any device within the surgical context that may generate useful surgical information 727. This surgical information 727 may present itself as an observable quality of the data source 726. The observable quality may include static qualities such as a device's model number, serial number, etc. The observable quality may include dynamic qualities such as the state of a device's configurable settings. Surgical information 727 may present itself, for example, as the result of sensor observations. Sensor observations may include observations from specific sensors in the operating room, sensors for monitoring conditions such as the patient's condition, sensors embedded in surgical devices, etc. Sensor observations may include information used during the surgery, such as video, audio, etc. Surgical information 727 may present itself as device event data. The surgical device may generate notifications and / or log events, and such events may be included in surgical information 727 for communication to the surgical hub 704. Surgical information 727 may present itself, for example, as a result of manual recording. A medical professional may record during a procedure by asking the patient to take notes, capturing still images from the display, etc.

[0126] The surgical data sources 726 may include, for example, modular devices (e.g., which may include sensors configured to detect parameters associated with the patient, the HCP, and the environment, and / or the modular devices themselves), local databases (e.g., a local EMR database containing patient records), patient monitoring devices (e.g., blood pressure (BP) monitors and electrocardiography (EKG) monitors), HCP monitoring devices, environmental monitoring devices, surgical instruments, surgical support equipment, etc.

[0127] The surgical hub 704 can be configured to derive contextual information about the surgical procedure from the data based on, for example, a particular combination of received data or a particular order in which data is received from the data sources 726. The contextual information inferred from the received data can include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure the surgeon is performing, the type of tissue being operated on, or the body cavity that is the target of the procedure. This ability, by some aspects of the surgical hub 704, to derive or infer information about the surgical procedure from the received data can be referred to as “situational awareness.” For example, the surgical hub 704 can incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 704 that derives contextual information about the surgical procedure from the received data and / or surgical planning information received from the edge computing system 714 or the healthcare data system 716 (e.g., an enterprise cloud server).

[0128] In operation, this matrix of surgical information may exist as one or more information flows. For example, surgical information may flow from a surgical data source 726 to the surgical hub 704. Surgical information may flow from the surgical hub 704 to a surgical data source 726 (e.g., a surgical device). Surgical information may flow between the surgical hub 704 and one or more healthcare data systems 716. Surgical information may flow between the surgical hub 704 and one or more edge computing devices 714.

[0129] The surgical information as presented in one or more information flows may be used in conjunction with one or more artificial intelligence (AI) systems to further enhance the operation of the surgical system 700. For example, a machine learning system, such as those described herein, may operate on one or more of the information flows to further enhance the operation of the surgical system 700.

[0130] 7B shows an exemplary computer-implemented surgical system 730 having multiple information flows 732. The surgical computing device 704 may communicate with and / or incorporate one or more surgical data sources. For example, the imaging module 733 (and endoscope) may exchange surgical information with the surgical computing device 704. Such information may include information from the imaging module 733 (and endoscope), such as video information, current settings, system status information, etc. The imaging module 733 may receive information from the surgical computing device 704, such as control information, configuration information, operational updates (software / firmware, etc.).

[0131] For example, the generator module 734 (and corresponding energy devices) may exchange surgical information with the surgical computing device 704. Such information may include information from the generator module 734 (and corresponding energy devices), such as electrical information (e.g., current, voltage, impedance, frequency, wattage), activity state information, sensor information such as temperature, current settings, system events, active duration, and startup timestamps. The generator module 734 may receive information from the surgical computing device 704, such as control information, configuration information, changes in the nature of visible and audible notifications to the medical professional (e.g., changes in the pitch, duration, and melody of an audible tone), electrical application profiles and / or application logic that may instruct the generator module to provide energy having a defined characteristic curve over the application time, operational updates (e.g., software / firmware), and the like.

[0132] For example, the smoke evacuator 735 may exchange surgical information with the surgical computing device 704. Such information may include information from the smoke evacuator 735 such as operational information (e.g., revolutions per minute), activity status information, sensor information such as temperature, current settings, system events, active duration, and boot timestamps. The smoke evacuator 735 may receive information from the surgical computing device 704 such as control information, configuration information, operational updates (software / firmware, etc.).

[0133] For example, the aspirate / irrigate module 736 may exchange surgical information with the surgical computing device 704. Such information may include information from the aspirate / irrigate module 736, such as operational information (e.g., liters per minute), activity status information, internal sensor information, current settings, system events, active duration, and startup timestamps. The aspirate / irrigate module 736 may receive information from the surgical computing device 704, such as control information, configuration information, operational updates (software / firmware, etc.).

[0134] For example, the communications module 739, the processor module 737, and / or the storage array 738 may exchange surgical information with the surgical computing device 704. In one example, the communications module 739, the processor module 737, and / or the storage array 738 may comprise all or part of the computing platform on which the surgical computing device 704 operates. In one example, the communications module 739, the processor module 737, and / or the storage array 738 may provide local computing resources to other devices in the surgical system 730. Information from the communications module 739, the processor module 737, and / or the storage array 738 to the surgical computing device 704 may include logical computing related reports such as processing load, processing power, process identification, CPU %, CPU time, threads, GPU %, GPU time, memory utilization, memory threads, memory ports, energy usage, bandwidth related information, packets in, packets out, data rates, channel utilization, buffer status, packet loss information, system events, and other status information. The communications module 739, processor module 737, and / or storage array 738 may receive information, such as control information, configuration information, operational updates (software / firmware, etc.), etc., from the surgical computing device 704. The communications module 739, processor module 737, and / or storage array 738 may also receive information from the surgical computing device 704 generated by another element or device of the surgical system 730. For example, data source information may be transmitted to and stored in the storage array. For example, the data source information may be processed by the processor module 737.

[0135] For example, the intelligent instrument 740 (with or without a corresponding display) may exchange surgical information with the surgical computing device 704. Such information may include information from the intelligent instrument 740 regarding the operation of the instrument, such as device electrical and / or mechanical information (e.g., current, voltage, impedance, frequency, wattage, torque, force, pressure, etc.), load status information (e.g., information regarding the identity, type, and / or status of reusables such as staple cartridges), clamping force, tissue compression pressure, and / or internal sensor information such as time, system events, active duration, and activation timestamp. The intelligent instrument 740 may receive information from the surgical computing device 704, such as control information, configuration information, changes in the nature of visible and audible notifications to the medical professional (e.g., changes in the pitch, duration, and melody of an audible tone), mechanical application profiles and / or application logic that may instruct the instrument's mechanical components to operate with defined characteristics (e.g., blade / anvil advancement speed, mechanical advantage, firing time, etc.), operational updates (software / firmware, etc.), and the like.

[0136] For example, the sensor module 741 may exchange surgical information with the surgical computing device 704. Such information may include information from the sensor module 741 about its sensor capabilities, such as the sensor results themselves, observation frequency and / or resolution, observation type, device alerts such as alerts for sensor failure, observations exceeding a defined range, observations exceeding an observable range, etc. The sensor module 741 may receive information from the surgical computing device 704, such as control information, configuration information, changes in the nature of the observations (e.g., frequency, resolution, observation type, etc.), triggers defining specific events for observations, on controls, off controls, data buffering, data pre-processing algorithms, operational updates (software / firmware, etc.), etc.

[0137] For example, the visualization system 742 may exchange surgical information with the surgical computing device 704. Such information may include information from the visualization system 742, such visualization data itself (e.g., still images, video, advanced spectral visualization, etc.), visualization metadata (e.g., visualization type, resolution, frame rate, encoding, bandwidth, etc.), etc. The visualization system 742 may receive information from the surgical computing device 704, such as control information, configuration information, changes in video settings (e.g., visualization type, resolution, frame rate, encoding, etc.), visual display overlay data, data buffering sizes, data pre-processing algorithms, operational updates (software / firmware, etc.), etc.

[0138] For example, the surgical robot 743 may exchange surgical information with the surgical computing device 704. Information from the surgical robot 743 may include any of the aforementioned information as it applies to robotic instruments, sensors, and devices. Information from the surgical robot 743 may also include information related to the robotic operation or control of such instruments, such as electrical / mechanical feedback of the robotic articulator, system events, system settings, mechanical resolution, control operation logs, articulator path information, etc. The surgical robot 743 may receive information from the surgical computing device 704, such as control information, configuration information, operation updates (software / firmware, etc.).

[0139] 7C illustrates an example information flow associated with multiple surgical computing systems 704a, 704b within a common environment. As the overall configuration of a computer-implemented surgical system (e.g., computer-implemented surgical system 750) changes (e.g., as data sources are added and / or removed from the surgical computing system), additional surgical information may be generated to reflect the changes. In this example, a second surgical computing system 704b (e.g., a surgical hub) may be added (along with a corresponding surgical robot) to surgical system 750 with existing surgical computing system 704a. The messaging flows described herein represent additional surgical information flow 755 (e.g., further integrated, analyzed, and / or processed according to algorithms such as machine learning algorithms) used as disclosed herein.

[0140] Here, two surgical computing systems 704a, 704b request permission from the surgical procedure task surgeon for the second surgical computing system 704b (having a corresponding surgical robot 756) to take control of the operating room from the existing surgical computing system 704a. The second surgical computing system 704b presents control of the corresponding surgical robot 756, robotic visualization tower 758, Monohat tool 759, and robotic stapler 749 in the operating room. Permission may be requested through the surgeon interface or console 751. Once permission is granted, the second surgical computing system 704b sends a message to the existing surgical computing system 704a requesting transfer of control of the operating room.

[0141] In one example, the surgical computing systems 704a, 704b can negotiate the nature of their interaction without external input based on previously collected data. For example, the surgical computing systems 704a, 704b may collectively determine that an upcoming surgical task requires the use of a robotic system. Such a determination may cause the existing surgical computing system 704a to autonomously hand over control of the operating room to a second surgical computing system 704b. Upon completion of the surgical task, the second surgical computing system 704b may then autonomously return control of the operating room to the existing surgical computing system 704a.

[0142] As illustrated in Figure 7C, the existing surgical computing system 704a has transferred control to a second surgical computing system 704b, which also assumes control of the surgeon interface 751 and secondary display 752. The second surgical computing system 704b assigns new identification numbers to the newly transferred devices. The existing surgical computing system 704a retains control of the handheld stapler 753, handheld powered dissector 754, and visualization tower 757. In addition, the existing surgical computing system 704a may perform a support role, with the processing and storage capabilities of the existing surgical computing system 704a now available to the second surgical computing system 704b.

[0143] 7D illustrates an exemplary surgical information flow in the context of a surgical procedure and corresponding exemplary uses of the surgical information for predictive modeling. The surgical information disclosed herein may provide data regarding one or more surgical procedures, including surgical tasks, instruments, instrument settings, motion information, procedural variations, and corresponding desirable metrics such as improved patient outcomes, lower costs (e.g., fewer resources utilized, shorter surgical time, etc.). The surgical information disclosed herein (e.g., that disclosed with respect to FIGS. 7A-7C ), in the context of one or more surgical systems and devices disclosed herein, provides a platform upon which certain machine learning algorithms and techniques disclosed herein may be used.

[0144] Surgical information 762 from multiple surgical procedures 764 (e.g., a subset of surgical information from each procedure) may be collected. Surgical information 762 may be collected from multiple surgical procedures 764, for example, by collecting data represented by one or more information flows disclosed herein.

[0145] To illustrate, an exemplary instance of surgical information 766 may be generated from an exemplary procedure 768 (e.g., a lung segmentectomy procedure as shown on timeline 769). Surgical information 766 may be generated during preoperative planning and may include patient record information. Surgical information 766 may be generated from data sources (e.g., data sources 726) during the course of a surgical procedure, including data generated each time medical personnel utilize a modular device paired with the surgical computing system (e.g., surgical computing system 704). The surgical computing system may receive this data from the paired modular device and other data sources. The surgical computing system itself may generate surgical information as part of its operation during a procedure. For example, the surgical computing system may record information related to configuration and control operations. The surgical computing system may record information related to situational awareness activities. For example, the surgical computing system may record recommendations, prompts, and / or other information provided to the medical team (e.g., provided via a display screen) that may be related to the next procedure step. For example, the surgical computing system may record configuration and control changes (e.g., adjustments to modular devices based on context) that may include activating a monitor, adjusting the field of view (FOV) of a medical imaging device, changing the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument, etc.

[0146] Hospital personnel retrieve the patient's EMR from the hospital's EMR database at 770. Based on the selected patient data in the EMR, the surgical computing system determines that the procedure to be performed is a thoracic procedure.

[0147] At 771, personnel scan incoming medical supplies for a procedure. The surgical computing system may cross-reference the scanned supplies with a list of supplies utilized in various types of procedures. The surgical computing system may verify that the mix of supplies corresponds to a thoracic procedure. Additionally, the surgical computing system may determine that the procedure is not a wedge resection (because the incoming supplies either do not include certain supplies needed for a thoracic wedge resection or are otherwise not compatible with a thoracic wedge resection). The medical personnel may scan a patient band via a scanner communicatively connected to the surgical computing system. The surgical computing system may verify the patient's identity based on the scanned data.

[0148] At 774, medical personnel turn on auxiliary equipment. The auxiliary equipment utilized may vary according to the type of surgical procedure and the technology used by the surgeon. In this example, the auxiliary equipment may include a smoke evacuator, an aspirator, and a medical imaging device. Once activated, the auxiliary equipment may pair with the surgical computing system. The surgical computing system may derive contextual information regarding the surgical procedure based on the paired type. In this example, the surgical computing system determines that the surgical procedure is a VATS procedure based on this particular combination of paired devices. The contextual information regarding the surgical procedure may be ascertained by the surgical computing system via information from the patient's EMR.

[0149] The surgical computing system may retrieve the steps of the procedure to be performed. For example, the steps may be associated with a treatment plan (e.g., a treatment plan specific to this patient's surgery, a treatment plan associated with a particular surgeon, a treatment plan template for the procedure in general, etc.).

[0150] At 775, staff attach EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices pair with the surgical computing system. The surgical computing system may receive data from the patient monitoring devices.

[0151] At 776, medical personnel induce anesthesia in the patient. The surgical computing system may record information related to this procedure step, such as data from the modular devices and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof.

[0152] At 777, the lung of the patient undergoing surgery is collapsed (and ventilation may be switched to the contralateral lung). The surgical computing system may determine that this procedure step has begun and may collect surgical information accordingly, including, for example, ventilator data, one or more timestamps, etc.

[0153] At 778, a medical imaging device (e.g., a scope) is inserted and video from the medical imaging device is initiated. The surgical computing system may receive medical imaging device data (i.e., video or image data) through a connection to the medical imaging device. The data from the medical imaging device may include imaging data and / or imaging metadata, such as the angle at which the medical imaging device is oriented relative to visualization of the patient's anatomy, the number of medical imaging devices currently active, etc. The surgical computing system may record positioning information for the medical imaging device. For example, one technique for performing a VATS lobectomy places the camera in the anterior-inferior corner of the patient's thoracic cavity above the diaphragm. Another technique for performing a VATS segmentectomy places the camera in an anterior intercostal position relative to the segmental fissure.

[0154] For example, using pattern recognition or machine learning techniques, the surgical computing system may be trained to recognize the positioning of a medical imaging device according to visualization of the patient's anatomy. For example, one technique for performing a VATS lobectomy utilizes a single medical imaging device. Another technique for performing a VATS segmentectomy uses multiple cameras. Yet another technique for performing a VATS segmentectomy uses an infrared light source (which may be communicatively coupled to the surgical computing system as part of the visualization system).

[0155] At 779, the surgical team begins the incision step of the procedure. The surgical computing system may collect data from the RF or ultrasonic generator indicating that the energy instrument is being fired. The surgical computing system may cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy instrument being fired at this point in the process (i.e., after previously discussed steps of the procedure have been completed) corresponds to the incision step. In one example, the energy instrument may be an energy tool mounted on a robotic arm of a robotic surgical system.

[0156] At 780, the surgical team proceeds to the ligation step of the procedure. The surgical computing system may collect surgical information 766 related to the surgeon ligating the arteries and veins based on receiving data from the surgical stapling and severing instrument indicating that such instrument is being fired. Next, the segmentectomy portion of the procedure is performed. The surgical computing system may collect information related to the surgeon transecting the parenchyma. For example, the surgical computing system may receive surgical information 766 from the surgical stapling and severing instrument, including data related to its cartridge, settings, firing details, etc.

[0157] At 782, the node dissection step is then performed. The surgical computing system may collect surgical information 766 related to the surgical team dissecting the node and performing the leak test. For example, the surgical computing system may collect data received from the generator indicating that an RF or ultrasonic instrument is being fired, including electrical and status information associated with the firing. The surgeon periodically alternates between the surgical stapling / cutting instrument and the surgical energy (i.e., RF or ultrasonic) instrument depending on the particular step in the procedure. The surgical computing system may collect surgical information 766 taking into account the particular sequence in which the stapling / cutting instrument and the surgical energy instrument are used. In one example, a robotic tool may be used for one or more steps in the surgical procedure. The surgeon may, for example, alternate between using the robotic tool and a handheld surgical instrument and / or use the devices simultaneously.

[0158] The incisions are then closed and the post-operative portion of the procedure begins. The patient is de-anesthetized at 784. The surgical computing system may collect surgical information regarding the patient emerging from anesthesia, for example, based on ventilator data (i.e., the patient's breathing rate begins to increase).

[0159] At 785, medical personnel remove various patient monitoring devices from the patient. The surgical computing system may collect information regarding the outcome of the procedure. For example, the surgical computing system may collect information related to the loss of EKG, BP, and other data from the patient monitoring devices.

[0160] The surgical information 762 (including the surgical information 766) may be structured and / or labeled. The surgical computing system may inherently provide such structure and / or labeling in the data collection. For example, the surgical information 762 may be labeled according to particular characteristics, desired results (e.g., efficiency, patient outcome, cost, and / or combinations thereof, etc.), particular surgical techniques, aspects of instrumentation (e.g., surgical instrument selection, timing, and activation, instrument settings, nature of instrument use, etc.), identities of medical professionals involved, particular patient characteristics, etc., each of which may be present in the data collection.

[0161] The surgical information (e.g., surgical information 762 collected over the procedure 764) may be used in connection with one or more artificial intelligence (AI) systems. AI may be used to perform computer cognitive tasks. For example, AI may be used to perform complex tasks based on observation of data. AI may be used to enable computing systems to perform cognitive tasks and solve complex tasks. AI may include using machine learning and machine learning techniques. ML techniques may include, for example, performing complex tasks without being programmed (e.g., explicitly programmed). For example, ML techniques may improve over time based on completing tasks with different inputs. An ML process may train itself, for example, using input data and / or a training dataset.

[0162] Machine learning (ML) techniques may be used, for example, in the medical field. For example, ML may be used on a set of data (e.g., a set of surgical data) to generate output (e.g., reduced surgical data, processed surgical data). In an example, the output of the ML process may include identified trends or relationships in the data input for processing. The output may include verifying results and / or outcomes associated with the input data. In an example, the input to the ML process may include medical data such as surgical images and patient scans. The ML process may output a determined medical condition based on the input surgical images and patient scans. The ML process may be used to diagnose a medical condition, for example, based on the surgical scans.

[0163] An ML process may improve itself, for example, using historical data and / or input data that trained the ML process. Thus, an ML process may continually improve with added inputs and processing. The ML process may update based on the input data. For example, over time, an ML process that generates medical outcomes based on medical data may improve and become more accurate and consistent in medical diagnoses.

[0164] ML processes may be used to solve different complex tasks (e.g., medical tasks). For example, ML processes may be used for data reduction, data preparation, data processing, trend identification, outcome determination, medical diagnosis, and / or the like. For example, an ML process may take surgical data as input and process the data for use in medical analysis. The processed data may be used to determine a medical diagnosis. Finally, an ML process may take raw surgical data and generate useful medical information (e.g., medical trends and / or diagnoses) associated with the raw surgical data.

[0165] ML processes may be combined to perform different discrete tasks on an input data set. For example, an ML process may include testing different combinations of ML subprocesses performing discrete tasks to determine which combination performs best (e.g., competitive use of different process / algorithm types and training to determine the best combination for a data set). For example, an ML process may include subprocess (e.g., algorithm) control and monitoring to refine and / or validate results and / or outcomes (e.g., error bounds).

[0166] An ML process may be initialized and / or set up to perform a task. For example, the ML process may be initialized based on initialization configuration information. The initialized ML process may be an untrained ML process and / or a base ML process for performing the task. An untrained ML process may be inaccurate in performing a specified task. As the ML process is trained, the task may be performed more accurately.

[0167] The initialization configuration information for the ML process may include initial settings and / or parameters. For example, the initial settings and / or parameters may include defined ranges for use by the ML process. The ranges may include manually entered and / or received data ranges. The ranges may include default ranges and / or randomized ranges for unreceived variables that may be used, for example, to complete the dataset for processing. For example, if a dataset lacks a data range, the default data range may be used as a substitute to run the ML process.

[0168] The initialization configuration information for the ML process may include data storage locations. For example, locations or data storage and / or databases associated with data interactions may be included. The databases associated with data interactions may be used to identify trends in the dataset. The databases associated with data interactions may include mappings of data to medical conditions. For example, the databases associated with data interactions may include mappings of heart rate data to arrhythmias, etc.

[0169] The initialization configuration information may include parameters associated with defining the system. The initialization configuration information may include instructions (e.g., methods) associated with displaying, verifying, and / or providing information to a user. For example, the initialization configuration may include instructions for an ML process to output data in a particular format for visualization to a user.

[0170] ML techniques may be used, for example, to perform data reduction. ML techniques for data reduction may include using multiple different data reduction techniques. For example, ML techniques for data reduction may include using one or more of the following: CUR matrix decomposition; decision trees; expectation-maximization (EM) processes (e.g., algorithms); explicit semantic analysis (ESA); exponential smoothing forecasting; generalized linear models; k-means clustering (e.g., nearest neighbor); naive Bayes; neural network processes; multivariate analysis; o-cluster; singular value decomposition; Q-learning; temporal difference (TD); deep adversarial networks; support vector machines (SVM); linear regression; dimensionality reduction; linear discriminant analysis (LDA); adaptive boosting (e.g., AdaBoost); gradient descent (e.g., stochastic gradient descent (SGD)); outlier detection; and / or others.

[0171] ML techniques may be used to perform data reduction, for example, using CUR matrix decomposition. CUR matrix decomposition may include using a matrix decomposition model (e.g., process, algorithm), such as a low-rank matrix decomposition model. For example, CUR matrix decomposition may include a low-rank matrix decomposition process that is expressed (e.g., explicitly expressed) in several (e.g., a small number) columns and / or rows of a data matrix (e.g., the CUR matrix decomposition may be interpretable). CUR matrix decomposition may include selecting columns and / or rows associated with statistical leverage and / or large influence in the data matrix. Using CUR matrix decomposition may enable identifying attributes and / or rows within the data matrix. Simplification of larger datasets (e.g., using CUR matrix decomposition) may enable users to review and interact (e.g., with the data). CUR matrix decomposition may facilitate regression, classification, clustering, and / or other processes.

[0172] ML techniques may be used to perform data reduction using, for example, decision trees (e.g., decision tree models). Decision trees may be used, for example, as a framework for quantifying outcome values ​​and / or the probability of an outcome occurring. Decision trees may be used, for example, to calculate values ​​for uncertain outcome nodes (e.g., in a decision tree). Decision trees may be used, for example, to calculate values ​​for decision nodes (e.g., in a decision tree). Decision trees may be models that enable classification and / or regression (e.g., applicable to classification and / or regression problems). Decision trees may be used to analyze numerical (e.g., continuous) and / or categorical data. Decision trees may be more successful and / or more efficient with large datasets (e.g., compared to other data reduction techniques).

[0173] Decision trees may be used in combination with other decision trees. For example, a random forest may refer to a collection of decision trees (e.g., an ensemble of decision trees). A random forest may include a collection of decision trees whose results may be aggregated into a result. A random forest may be a supervised learning algorithm. A random forest may be trained, for example, using a bagging training process.

[0174] A random decision forest (e.g., random forest) may add randomness (e.g., additional randomness) to a model, for example, while growing a tree. A random forest may be used, for example, to search for the best feature among a random subset of features, rather than searching for the most important feature (e.g., while splitting a node). Searching for the best feature among a random subset of features may result in a wide variety, which may result in a better (e.g., more efficient and / or accurate) model.

[0175] Random forests may include using parallel ensembles. Parallel ensembles may include, for example, fitting (e.g., several) decision tree classifiers in parallel on different dataset subsamples. Parallel ensembles may include using majority voting or averaging on the results or final outcome. Parallel ensembles may be used to minimize overfitting and / or increase prediction accuracy and control. Random forests with multiple decision trees may (e.g., generally) be more accurate than single decision tree-based models. A set of decision trees with controlled variation may be constructed, for example, by combining bootstrap aggregation (e.g., bagging) with random feature selection.

[0176] ML techniques may be used to perform data reduction, for example, using an expectation-maximization (EM) model (e.g., process, algorithm). For example, an EM model may be used to find likelihood (e.g., local maximum likelihood) parameters of a statistical model. An EM model may be used when equations cannot be solved directly. An EM model may consider latent variables and / or unknown parameters and known data observations. For example, an EM model may determine that missing values ​​are present in a dataset. An EM model receives configuration information indicating to assume the presence of missing (e.g., unobserved) data points in the dataset.

[0177] The EM model may use component clustering. For example, component clustering may allow EM components to be grouped into high-level clusters. For example, if component clustering is disabled (e.g., in the EM model), the components may be treated as clustered.

[0178] ML techniques may be used to perform data reduction, for example, using Explicit Semantic Analysis (ESA). ESA may be used at the level of semantics (e.g., meaning) rather than the vocabulary (e.g., surface form vocabulary) of words or documents. ESA may focus on the meaning of a set of text, for example, as a combination of concepts found within the text. ESA may be used for document classification. ESA may be used for semantic relevance computation (e.g., how similar words or fragments of text are to each other). ESA may be used for information retrieval.

[0179] ESAs may be used, for example, for document classification. Document classification may include tagging documents for management and sorting. Tagging documents (e.g., with keywords) may make them easier to search. Keyword tagging (e.g., using only keyword tagging) may limit the accuracy and / or efficiency of document classification. For example, using keyword tagging may reveal (e.g., only) documents that have the keyword, but not documents that have words with similar meanings to the keyword. Semantically classifying text (e.g., using ESAs) may improve a model's understanding of the text. Semantically classifying text may include representing documents as concepts and reducing reliance on specific keywords.

[0180] ML techniques may be used to perform data reduction, for example, using an exponential smoothing forecasting model. Exponential smoothing may be used to smooth time series data, for example, using an exponential window function. For example, in a moving average, past observations may be weighted equally, but using an exponential function, weights may be assigned that decrease exponentially over time.

[0181] ML techniques may be used to perform data reduction, for example, using linear regression. Linear regression may be used to predict continuous outcomes. For example, linear regression may be used to predict the value of a variable (e.g., a dependent variable) based on the values ​​of different variables (e.g., independent variables). Linear regression may apply a linear approach to model the relationship between a scalar response and one or more explanatory variables (e.g., a dependent variable and / or independent variables). Simple linear regression may refer to a linear regression use case associated with one explanatory variable. Multiple linear regression may refer to a linear regression use case associated with two or more explanatory variables. Linear regression may model the relationship, for example, using a linear prediction function. The linear prediction function may estimate unknown model parameters from a dataset.

[0182] For example, linear regression may be used to identify patterns within a training dataset. The identified patterns may relate to groupings of values ​​and / or labels. The model may learn the relationship between (e.g., each) label and expected outcomes. After training, the model may be used on raw data outside the training dataset (e.g., data that does not have a mapped and / or known output). A trained model using linear regression may determine a calculated prediction associated with the raw data, such as identifying seasonal changes in sales data.

[0183] ML techniques may be used to perform data reduction, e.g., generalized linear models (GLMs). GLMs may be used as flexible generalizations of linear regression. GLMs may generalize linear regression, for example, by allowing linear models to relate response variables.

[0184] ML techniques may be used to perform data reduction, for example, using k-means clustering (e.g., nearest neighbor models). K-means clustering may be used in vector quantization. K-means clustering may be used in signal processing. K-means clustering may, for example, aim to divide n observations into k clusters, with each observation falling into the cluster with the closest mean.

[0185] K-means clustering may include K-Nearest Neighbor (KNN) learning. KNN may be instance-based learning (e.g., non-generalized learning, lazy learning). KNN may refrain from building a general internal model. KNN may include storing instances corresponding to training data in an n-dimensional space. KNN may use the data to classify data points, for example, based on a similarity measure (e.g., a Euclidean distance function). Classification may be calculated, for example, based on a majority vote of the k neighbors of a point (e.g., each point). KNN may be robust to noisy training data. Accuracy may depend on data quality (e.g., for KNN). KNN may include selecting the number of neighbors to consider (e.g., an optimal number of neighbors to consider). KNN may be used for classification and / or regression.

[0186] ML techniques may be used to perform data reduction, for example, using a naive Bayes model (e.g., process). For example, a naive Bayes model may be used to build a classifier. Using the naive Bayes model, a class label may be assigned to a problem instance (e.g., represented as a vector of feature values). The class label may be drawn from a set (e.g., a finite set). Different processes (e.g., algorithms) may be used to train the classifier. A family of processes (e.g., a family of algorithms) may be used. A family of processes may be based on the principle that a naive Bayes classifier (e.g., all naive Bayes) classifier assumes that feature values ​​are independent of different feature values ​​(e.g., given a class variable).

[0187] ML techniques may be used to perform data reduction, for example, using neural networks. The neural network may learn (e.g., be trained) by processing examples, for example, to perform other tasks (e.g., similar tasks). The processed examples may include inputs and results (e.g., inputs mapped to results). The neural network may learn by forming probability-weighted associations between inputs and results. The probability-weighted associations may be stored within the neural network's data structure. Training the neural network from a given example may be performed by determining the difference between the network's processed output (e.g., prediction) and a target output. This difference may be an error. The neural network may adjust the weighted associations (e.g., stored weighted associations), for example, according to a learning rule and an error value.

[0188] ML techniques may be used to perform data reduction, for example, using multivariate analysis, which may include performing multivariate state estimation and / or non-negative matrix factorization.

[0189] ML techniques may be used to perform data reduction using, for example, a support vector machine (SVM). SVMs may be used in multidimensional spaces (e.g., high-dimensional spaces, infinite-dimensional spaces). SVMs may be used to construct hyperplanes (e.g., a set of hyperplanes). A hyperplane with the greatest distance (e.g., compared to other constructed hyperplanes) from the nearest training data point within a class (e.g., any class) may achieve strong separation (e.g., generally, the larger the margin, the lower the generalization error of the classifier). SVMs may be effective in high-dimensional spaces. SVMs may behave differently, for example, based on different mathematical functions (e.g., kernels, kernel functions). For example, kernel functions may include one or more of linear, polynomial, radial basis function (RBF), sigmoid, etc. Kernel functions may be used as SVM classifiers. SVMs may be limited, for example, in use cases where the dataset contains a large amount of noise (e.g., overlapping target classes).

[0190] ML techniques may be used to perform data reduction, such as, for example, dimensionality reduction. Reducing the dimensionality of a sample of data (e.g., unlabeled data) may help refine groups and / or clusters. Reducing the number of variables in a model may simplify the data's trends. Simplified data trends may allow for more efficient processing. Dimensionality reduction may be used, for example, when many (e.g., too many) dimensions obscure (e.g., adversely affect) insights, trends, patterns, outcomes, and / or the like.

[0191] Reducing dimensionality may include using principal component analysis (PCA). PCA may be used to establish principal components that govern the relationships between data points. PCA may focus on simplifying (e.g., simplifying only) the principal components. Dimensionality reduction (e.g., PCA) may be used to maintain the diversity of data groupings within a dataset, but rationalize the number of distinct groups.

[0192] ML techniques may be used to perform data reduction, e.g., linear discriminant analysis (LDA). LDA may refer to a linear decision boundary classifier, which may be created, for example, by fitting class conditional densities to data (e.g., and applying Bayes' rule). LDA may include a generalization of Fisher's Linear Discriminant (e.g., projecting a given dataset into a lower-dimensional space to reduce dimensionality, minimize model complexity, and reduce computational cost). An LDA model (e.g., a standard LDA model) may fit classes with Gaussian densities. An LDA model may assume that classes (e.g., all classes) share a covariance matrix. LDA may be similar to an analysis of variance (ANOVA) process and / or regression analysis. For example, LDA may be used to express a dependent variable as a linear combination of other features and / or measurements.

[0193] ML techniques may be used to perform data reduction, such as, for example, adaptive boosting (e.g., AdaBoost). Adaptive boosting may include creating a classifier (e.g., a powerful classifier). Adaptive boosting may include creating a classifier by combining multiple classifiers (e.g., poorly performing classifiers), for example, to obtain a resulting classifier with high accuracy. AdaBoost may be an adaptive classifier that improves the efficiency of a classifier. AdaBoost may trigger overfitting. AdaBoost may be used (e.g., most commonly used) to improve the performance of decision trees, base estimators, binary classification problems, and / or the like. AdaBoost may be sensitive to noisy data and / or outliers.

[0194] ML techniques may be used to perform data reduction, such as stochastic gradient descent (SGD). SGD may include an iterative process used to optimize a function (e.g., an objective function). SGD may be used, for example, to optimize an objective function with specific smoothness properties. Stochastic may refer to random probability. SGD may be used, for example, to reduce computational load in high-dimensional optimization problems. SGD may be used, for example, to enable faster iterations while trading off a slower convergence rate. Gradient may refer, for example, to the slope of a function that calculates the degree of change of a variable in response to a change in another variable. Gradient descent may refer to a convex function that outputs the partial derivative of a set of its input parameters. For example, α may be a learning rate, and J may be the cost of training examples for the i-th iteration. This formula may represent a stochastic gradient descent weight update method for the j-th iteration. In large-scale ML and sparse ML, SGD may be applied to problems in text classification and / or natural language processing (NLP). SGD can be sensitive to feature scaling (e.g., it may be necessary to use a range of hyperparameters, such as the regularization parameter and number of iterations).

[0195] ML techniques may be used to perform data reduction, such as using outlier detection. An outlier may be a data point that contains information (e.g., useful information) about abnormal behavior of the system described by the data. Outlier detection processes may include univariate and multivariate processes.

[0196] The ML process may be trained, for example, using one or more training methods. For example, the ML process may be trained using one or more of the following training techniques: supervised learning; unsupervised learning; semi-supervised learning; reinforcement learning; and / or others.

[0197] Machine learning can be supervised (e.g., supervised learning). Supervised learning algorithms can create a mathematical model from training data sets (e.g., training data). FIG. 8A illustrates an exemplary supervised learning framework 800. Training data (e.g., training examples 802 as shown in FIG. 8) may consist of a set of training examples (e.g., input data mapped to labeled outputs as shown in FIG. 8). Training examples 802 may include one or more inputs and one or more labeled outputs. The labeled outputs may serve as supervisory feedback. In the mathematical model, training examples 802 may be represented by an array or vector, sometimes called a feature vector. Training data may be represented by rows of the feature vector that form a matrix. Through iterative optimization of an objective function (e.g., a cost function), supervised learning algorithms can learn a function (e.g., a prediction function) that can be used to predict outputs associated with one or more new inputs. A properly trained predictive function (e.g., a trained ML model 808) may determine outputs 804 (e.g., labeled outputs) for one or more inputs 806 that may not be part of the training data (e.g., input data that does not have a mapped labeled output, as shown in FIG. 8). Exemplary algorithms may include linear regression, logistic regression, neural networks, nearest neighbors, naive Bayes, decision trees, SVMs, and / or others. Exemplary problems that can be solved by supervised learning algorithms may include classification, regression problems, etc.

[0198] Machine learning can be unsupervised (e.g., unsupervised learning). FIG. 8B illustrates an exemplary unsupervised learning framework 810. An unsupervised learning algorithm 814 may train on a dataset that may include input 811 and may find structure 812 in the data (e.g., pattern detection and / or descriptive modeling). The structure 812 in the data may resemble groupings or clusterings of data points. Thus, the algorithm 814 may learn from training data that may be unlabeled. Instead of responding to supervised feedback, the unsupervised learning algorithm may identify commonalities in the training data and may react based on the presence or absence of such commonalities in each training data. For example, training may include operating on training input data to generate a model and / or output with a particular energy (e.g., cost function, etc.), and such energy may be used to further refine the model (e.g., to define a model that minimizes the cost function given the training input data). Exemplary algorithms may include the Apriori algorithm, K-means, K-nearest neighbors (KNN), K-medians, etc. Representative problems that can be solved by unsupervised learning algorithms may include clustering problems, anomaly / outlier detection problems, etc.

[0199] Machine learning may be semi-supervised (e.g., semi-supervised learning). Semi-supervised learning algorithms may be used in scenarios where labeling data is costly (e.g., because a skilled expert is required to label the data) and labels for the data are limited. Semi-supervised learning models may take advantage of the idea that while the group membership of unlabeled data is unknown, the data still holds important information about group parameters.

[0200] Machine learning may include reinforcement learning, which may be an area of ​​machine learning that may concern how a software agent can take actions in an environment to maximize some notion of cumulative reward. Reinforcement learning algorithms may not assume knowledge of an exact mathematical model of the environment (e.g., represented by a Markov decision process (MDP)) and may be used when an exact model may not be feasible. Reinforcement learning algorithms may be used in autonomous vehicles or in learning to play games against human opponents. Exemplary algorithms may include Q-learning, temporal difference (TD), deep adversarial networks, and / or others.

[0201] Reinforcement learning may involve an algorithm (e.g., an agent) that continuously learns from an environment in an iterative manner. During the training process, the agent may learn from experience with the environment until the agent has explored the full range of states (e.g., possible states). Reinforcement learning may be defined by the type of problem. Reinforcement learning solutions may be classified as reinforcement learning algorithms. In a problem, an agent may determine an action to select (e.g., a best action) based on the agent's current state. When steps are repeated, the problem may be referred to as an MDP.

[0202] For example, reinforcement learning may include an action step. The action step in reinforcement learning may include an agent observing an input state. The action step in reinforcement learning may include causing an agent to perform an action using a decision-making function. The action step may include an agent receiving a reward and / or reinforcement from the environment (e.g., after an action is performed). The action step in reinforcement learning may include storing state-action pair information related to the reward.

[0203] Machine learning may be part of a technology platform called cognitive computing (CC), which may comprise various fields such as computer science and cognitive science. CC systems may be able to learn at scale, reason purposefully, and interact naturally with humans. Self-teaching algorithms, which may use data mining, visual recognition, and / or natural language processing, may enable CC systems to solve problems and optimize human processes.

[0204] The output of a machine learning training process may be a model for predicting outcomes for new data sets. For example, a linear regression learning algorithm may have a cost function that can minimize the prediction error of a linear prediction function during the training process by adjusting the coefficients and constants of the linear prediction function. If a minimum value can be reached, the linear prediction function with the adjusted coefficients may be considered trained and constitute the model produced by the training process. For example, a neural network (NN) algorithm for classification (e.g., a multilayer perceptron (MLP)) may include a hypothesis function represented by a network of layers of nodes that are assigned biases and interconnected with weighted connections. The hypothesis function may also be a nonlinear function (e.g., a highly nonlinear function) that may include linear and logistic functions nested together, with an outermost layer consisting of one or more logistic functions. The NN algorithm may include a cost function for minimizing classification error by adjusting biases and weights through a process of feedforward propagation and backpropagation. If a global minimum can be reached, the optimized hypothesis function with its adjusted bias and weight layers may be considered trained and constitute the model that the training process produced.

[0205] Data aggregation may be performed for machine learning as a first stage in a machine learning lifecycle. Data aggregation may include steps such as identifying various data sources, collecting data from the data sources, and integrating the data. For example, to train a machine learning model for predicting surgical complications and / or post-surgical recovery rates, data sources including pre-surgical data such as a patient's medical condition and biomarker measurement data may be identified. Such data sources may be a patient's electronic medical record (EMR), a computing system that stores the patient's pre-surgical biomarker measurement data, and / or other similar data stores. Data from such data sources may be retrieved and stored in a central location for further processing in the machine learning lifecycle. Data from such data sources may be linked (e.g., logically linked) and accessed as if they were centrally stored. Surgical data and / or post-surgical data may be similarly identified and collected. Furthermore, the collected data may be integrated. In examples, a patient's pre-surgical medical record data, pre-surgical biomarker measurement data, pre-surgical data, surgical data, and / or post-surgical data may be combined into a patient record, which may be an EMR.

[0206] Data preparation can be performed for machine learning as another stage of the machine learning lifecycle. Data preparation can include data preprocessing steps such as data formatting, data cleaning, and data sampling. For example, collected data may not be in a data format suitable for training a model. Such data records can be converted into a flat file format for model training. Such data can be mapped to numerical values ​​for model training. Such identifying data can be removed before model training. For example, identifying data can be removed for privacy reasons. As another example, data can be removed because there may be more available data than can be used for model training. In such cases, a subset of the available data can be randomly sampled and selected for model training, and the remainder can be discarded.

[0207] Data preparation may include data transformation operations (e.g., after preprocessing), such as scaling and aggregation. For example, the preprocessed data may include data values ​​at various scales. These values ​​may be scaled up or down, e.g., to be between 0 and 1, for model training. For example, the preprocessed data may include data values ​​that become more meaningful when aggregated.

[0208] Model training may be another aspect of the machine learning life cycle. The model training process described herein may depend on the machine learning algorithm used. A model may be considered suitably trained after it has been trained, cross-validated, and tested. Thus, a dataset from the data preparation stage (e.g., an input dataset) may be divided into a training dataset (e.g., 60% of the input dataset), a validation dataset (e.g., 20% of the input dataset), and a test dataset (e.g., 20% of the input dataset). After a model is trained on the training dataset, it may be run on the validation dataset to reduce overfitting. If the model's accuracy is increasing, but decreases when run on the validation dataset, this may indicate an overfitting problem. The test dataset may be used to test the accuracy of the final model to determine whether it is ready for deployment or whether more training may be required.

[0209] Model deployment can be another aspect of the machine learning lifecycle. Models may be deployed as part of a standalone computer program. Models may be deployed as part of a larger computing system. Models may be deployed with model performance parameters. Such performance parameters may monitor model accuracy as it is used to make predictions on a running dataset. For example, such parameters may track false positives and false positives of a classification model. Such parameters may further store false positives and false positives for further processing to improve the accuracy of the model.

[0210] Model updates after deployment may be another aspect of the machine learning cycle. For example, the deployed model may be updated as false positives and / or as false positives are predicted on the production data. In one example, for an MLP model deployed for classification, when a false positive occurs, the deployed MLP model may be updated to increase the probability cutoff for predicting a positive to reduce the false positives. In one example, for an MLP model deployed for classification, when a false negative occurs, the deployed MLP model may be updated to decrease the probability cutoff for predicting a positive to reduce the false negatives. In one example, for an MLP model deployed for classification of surgical complications, when both false positives and false negatives occur, the deployed MLP model may be updated to decrease the probability cutoff for predicting a positive to reduce the false negatives, as predicting a false positive may be less serious than a false negative.

[0211] For example, the deployed model may be updated as more live production data becomes available as training data. In such cases, the deployed model may be further trained, validated, and tested using such additional live production data. In one example, the updated biases and weights of the further trained MLP model may update the biases and weights of the deployed MLP model. Those skilled in the art will recognize that post-deployment model updates may not be a one-time occurrence, but may occur as frequently as is suitable to improve the accuracy of the deployed model.

[0212] ML techniques may be used independently of each other or in combination. Different problems and / or datasets may benefit from using different ML techniques (e.g., combinations of ML techniques). Different training types for models may be more suitable for particular problems and / or datasets. The optimal algorithm (e.g., combinations of ML techniques) and / or training type may be determined for a particular use, problem, and / or dataset. For example, processes may be performed to select one or more of the following: select a data reduction type, select a model and / or algorithm configuration, determine the location of data reduction, determine the efficiency of the reduction and / or results, and / or other.

[0213] For example, an ML technique and / or combination of ML techniques may be determined for a particular problem and / or use case. Multiple data reduction and / or data analysis processes may be performed to determine accuracy, efficiency, and / or compatibility associated with a dataset. For example, a first ML technique (e.g., a first set of combined ML techniques) may be used on a dataset to perform data reduction and / or data analysis. The first ML technique may generate a first output. Similarly, a second ML technique (e.g., a second set of combined ML techniques) may be used on a dataset (e.g., the same dataset) to perform data reduction and / or data analysis. The second ML technique may generate a second output. The first output may be compared to the second output to determine which ML technique produced a more desirable result (e.g., a more efficient result, a more accurate result). Multiple ML techniques may be compared on the same dataset to determine the optimal ML technique to use on future similar datasets and / or problems.

[0214] In an example, in a medical context, a surgeon or medical professional may provide feedback to the ML technique and / or model used on the dataset. The surgeon may input the feedback into the weighted results of the ML model. The feedback may be used as input by the model to determine reduction methods for future analysis.

[0215] In examples, a data analysis method (e.g., an ML technique to be used in the data analysis method) may be determined based on the dataset itself. For example, the origin of the data may influence the type of data analysis method to be used for the dataset. Available system resources may be used to determine the data analysis method to be used for a given dataset. The magnitude of the data may be considered, for example, in determining the data analysis method. For example, the need for an external dataset for local processing levels or magnitude of operational response may be considered (e.g., small device changes may be made using local data, while large device operational changes may require global compilation and validation).

[0216] Such ML techniques may be applied to surgical information (e.g., a combination of the information flow of surgical information in Figure 7) to generate useful ML models.

[0217] With reference to FIG. 9 , an overview of a surgical system may be provided. A surgical instrument may be used in a surgical procedure as part of the surgical system. The surgical computing device / edge computing device may be configured to coordinate information flow to the surgical instrument (e.g., a display on the surgical instrument). For example, the surgical computing device / edge computing device may be described in U.S. Patent Application Publication No. 2019-0200844(A1), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Exemplary surgical instruments suitable for use with the surgical system are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety.

[0218] FIG. 9 shows an example overview of receiving global or regional information and modifying the global or regional information based on local information. A surgical computing device / edge computing device may be used to perform a surgical procedure on a patient. A robotic system may be used as part of a surgical system in a surgical procedure. For example, a robotic system may be described in U.S. Patent Application Publication No. 2019-0200844(A1), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. A robotic hub may be used to process images of the surgical site and then display them to the surgeon through the surgeon's console.

[0219] Other types of robotic systems may be readily adapted for use with the surgical system. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0220] Various examples of cloud-based analytics performed by the cloud and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed December 4, 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," U.S. Patent Application Publication No. 2019-0201119(A1) (U.S. Patent Application No. 15 / 940,694), filed March 29, 2018, entitled "Cloud-based medical analytics for medical facility segmented individualization of instrument function," U.S. Patent Application Publication No. 2019-0201119(A1) (U.S. Patent Application No. 15 / 940,694), filed March 29, 2018, entitled "Cloud-based medical analytics for linking of local usage trends with the resource acquisition behaviors of larger data," and U.S. Patent Application Publication No. 2019-02020109(A1) (U.S. Patent Application No. 2019-0203091), filed March 29, 2018, entitled "Cloud-based medical analytics for linking of local usage trends with the resource acquisition behaviors of larger data." No. 2019-0201144(A1) (U.S. Patent Application No. 15 / 940,679), entitled "Cloud-based medical analytics for customization and recommendations to a user," filed March 29, 2018, and U.S. Patent Application No. 2019-0206555(A1) (U.S. Patent Application No. 15 / 940,660), entitled "Cloud-based medical analytics for customization and recommendations to a user," filed March 29, 2018, the disclosures of which are incorporated herein by reference in their entireties.

[0221] In various aspects, an imaging device may be used in a surgical system and may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors.

[0222] The optics of the imaging device may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0223] The one or more illumination sources can be configured to emit electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or luminous spectrum, is the portion of the electromagnetic spectrum that is visible to (e.g., detectable by) the human eye and is sometimes referred to as visible light or simply light. The typical human eye responds to wavelengths in air between about 380 nm and about 750 nm.

[0224] The invisible spectrum (e.g., non-radiative spectrum) is the portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which constitutes invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which constitutes invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0225] In various aspects, the imaging device may be configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngological-nephroscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0226] The imaging device may use multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and UV light. Spectral imaging can extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is complete to perform one or more of the tests described above on the treated tissue. It is self-evident that any surgical procedure requires strict sterilization of the operating room and surgical equipment. The strict hygiene and sterilization conditions required in an "operating room," i.e., an operating room or procedure room, require the highest possible sterility of all medical devices and equipment. Part of the sterilization process is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device and its accessories and components. It will be understood that the sterile field may be considered a specific area, such as in a tray or on a sterile towel, that is deemed free of microorganisms, or the sterile field may be considered the area immediately surrounding the patient being prepared for the surgical procedure. The sterile field may include appropriately clothed and hand-washed team members, as well as all equipment and fixtures in the area.

[0227] As shown in FIG. 9 , a surgical computing system surgical hub / edge computing device 52500 may be linked to an operating room. In one example, multiple surgical computing devices / edge computing devices may be associated with each operating room. The operating room may include one or more surgical computing devices and one or more surgical instruments or devices 52505 or other modules and / or subsystems that may be utilized during a surgical procedure, for example, as described herein in FIGS. 2 , 3 , and 7B . The surgical computing device or edge computing device may include an analytics subsystem 52530 and a local machine learning (ML) model or subsystem 52515. The surgical device may be used by a medical professional to perform a surgical procedure on a patient. For example, the surgical device may be an endocutter.

[0228] In one example, the surgical computing device and the edge computing device may be two different devices. In such a case, the surgical computing device or the edge computing device may transmit parameters or control algorithms associated with the surgical instrument or other module to the surgical instrument or other module via the surgical computing device (e.g., a surgical hub).

[0229] The surgical device may communicate with the surgical computing device / edge computing edge device 52500. The surgical computing device or computing edge device may be located in the operating room where the surgical procedure is being performed or in the medical facility where the operating room is located. Surgical step and surgical task may be used interchangeably herein. The surgical computing device or computing edge device may transmit one or more algorithms (e.g., control algorithms) or parameters used by the surgical instrument or other module connected to the surgical computing device or computing edge device. The surgical computing device / edge computing device 52500 may instruct the surgical device on information related to the surgical procedure being performed on the patient.

[0230] In one example, the surgical computing device or edge device 52500 may indicate to the surgical instrument 52505 how to set parameters (e.g., patient data, healthcare provider data, surgical instrument data, etc.) to perform a surgical procedure (e.g., or surgical tasks of a surgical procedure), e.g., to autonomously perform a surgical procedure. How surgical instruments operate autonomously is described in more detail in U.S. patent application Ser. No. 17 / 747,806, filed May 18, 2022, under the heading "METHOD OF CONTROLLING AUTONOMOUS OPERATIONS IN A SURGICAL SYSTEM," the disclosure of which is incorporated herein by reference in its entirety. Determining the surgical information (e.g., patient data, healthcare provider data, surgical instrument data, etc.) used to set the parameters may be based on output from a local machine learning model 52515 located within the surgical computing device or edge computing device 52500. In one example, the machine learning model and / or trained machine learning model may be utilized as part of a supervised learning framework. Supervised learning models are described herein in FIG. 8A . Training data (e.g., training examples 802 as illustrated in FIG. 8A ) may consist of a set of training examples (e.g., input data mapped to labeled outputs as shown in FIG. 8A ). The training data used in training the local machine learning model 52515 may include surgical data collected from previous surgical procedures and / or simulated surgical procedures. The training data may include previous control algorithms associated with the surgical instrument (e.g., stored locally or received from an enterprise server 52540). The training data may also include parameters associated with the patient, medical professional, and / or surgical instrument. In one example, the local ML model as output may provide surgical instrument parameters (e.g., surgical instrument firing rate) or control algorithms associated with the surgical instrument.In one example, a surgical instrument parameter or control algorithm associated with the surgical instrument may be utilized to instruct the surgical instrument to set (e.g., autonomously set) a parameter, for example, a firing rate at a particular frequency for performing an anastomosis. The surgical computing device / edge computing device 52500 may set parameters (e.g., patient data, healthcare provider data, surgical instrument data, etc.) of the surgical instrument or device 52505 by sending a message to the surgical instrument. In one example, the message to set the parameter may be in response to the surgical instrument 52505 sending a request message 52520 to the surgical computing device / edge computing device 52500 requesting the parameter.

[0231] Surgical information (e.g., surgical data) related to the surgical procedure may be generated (e.g., by a monitoring module located on the surgical computing device / edge computing device 52500 or locally by the surgical instrument 52505). For example, the surgical information may be based on the performance of the surgical instrument 52505. For example, the surgical information associated with the patient may include physical measurements, physiological measurements, etc. Measurements are described in more detail in U.S. Patent Application No. 17 / 156,28, filed November 10, 2021, under the title "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements," the disclosure of which is incorporated herein by reference in its entirety.

[0232] The surgical computing device / edge computing device 52500 may receive local measurements based on measurements from one or more surgical instruments 52505 located in the operating room in which the surgical computing device / edge computing device 52500 is located. The measurements may relate to a surgical procedure being performed on a patient in the operating room. For example, the surgical procedure may be a colon resection. The surgical computing device / edge computing device / surgical computing device / edge computing device 52500 may have a module that may include a surgical procedure plan 52510. By using the surgical plan 52510, the surgical computing device / edge computing device 52500 may determine surgical tasks to be performed, which may be part of the surgical procedure, for example, as described herein in FIG. 7D . For example, the surgical procedure may be a lung segmentectomy. In such a case, the surgical tasks may include surgical tasks 1 through K. For example, surgical task 1 may include retrieving electronic medical records associated with the patient, and surgical task K may include terminating anesthesia and removing all monitors. While surgical tasks 1-K are being performed by the medical professional, surgical instruments 52505 in the operating room, along with other devices capable of measuring data related to the surgical procedure, may transmit data (e.g., related to the surgical procedure) to the surgical computing device / edge computing device 52500.

[0233] When highly sensitive surgical information associated with a patient is transmitted to a remote entity (e.g., an enterprise cloud server) located (physically or virtually) outside the protected boundary 52525, it may first be anonymized. Anonymizing patient data may include one or more of the following actions: editing, randomizing, converting the data to a shorter format (e.g., summarizing, or averaging). Editing may include, for example, removing data from a dataset before transmitting the dataset to a remote server (e.g., an enterprise cloud server). Randomizing may include applying a random value to the data, which may be restored if the receiver receives the private key. Converting the data to a shorter format may include summarizing and / or averaging. Summarizing may include, for example, representing patient data by ranges and transmitting the data ranges representing the data. Averaging may include representing data by average values ​​instead of exact values.

[0234] An analysis subsystem 52530 within the surgical computing device or edge computing device may be used by the surgical computing device / edge computing device 52500 to collect and / or analyze surgical data associated with the surgical procedure. The surgical data may include data associated with the surgical procedure plan 52500 (e.g., including a set of surgical tasks), patient-related data, medical professional-related data, and / or other data (e.g., metrics associated with various surgical devices and / or instruments utilized during the surgical procedure). The analysis subsystem 52530 may determine whether to request global or regional surgical information 52535 from the global cloud enterprise server 52540 based on the surgical data associated with the surgical procedure. For example, during the surgical procedure (e.g., at the start of the surgical procedure), the analysis subsystem 52530 associated with the surgical computing device / edge computing device 52500 may determine to send a request to the global cloud enterprise 52540 to receive recommendations regarding surgical information (e.g., default parameters, control algorithms, etc.) related to the surgical procedure. The global cloud enterprise 52540 may be located outside the protected perimeter 52525. In such cases, information located on the surgical computing device / edge computing device 52500 (e.g., in a database accessible by the surgical hub / edge device) that is sent to the cloud server 52540 outside the protected perimeter 52525 may be anonymized (e.g., redacted, randomized, summarized, averaged, etc.) as described herein.In determining whether a request can be sent to the enterprise global server 52540, the surgical computing device / edge computing device 52500 (e.g., via the analysis subsystem 52530) may consider one or more of surgical information (e.g., metrics) linked to the surgical task, the surgical task itself, the overall surgical procedure plan 52510, performance criteria related to the surgical task (e.g., the overall latency required for the endocutter to successfully perform (e.g., autonomously perform) the anastomosis), the capabilities of the surgical computing device / edge computing device 52500 and the global cloud enterprise server 52540, the type of surgical data, etc.

[0235] The request message 52520 may include one or more of an indication of the surgical procedure being performed, the current surgical task (e.g., if the request is sent during a surgical procedure), the surgical data with which the request is associated (e.g., parameters associated with various surgical instruments and / or devices, and metrics collected by the surgical computing device / edge computing device 52500 during the surgical task), anonymized patient-related information, etc. The request sent to the global cloud enterprise server 52540 may be for one or more global algorithms or default parameters that may be used for the various surgical instruments and devices associated with the current surgical procedure being performed.

[0236] In one example, information collected by the surgical computing device / edge computing device 52500 and related to one or more surgical tasks of the surgical procedure and / or algorithms used by the local surgical system may be transmitted to the enterprise cloud server 52540 before or after transmitting the request message 52520. The enterprise cloud server 52540 may train a global machine learning subsystem using surgical information received from various globally distributed surgical computing devices / edge computing devices, as described with respect to FIG. 10. The global machine learning subsystem may learn what global or regional surgical information 52535 to transmit (e.g., recommend) to the surgical computing device / edge computing device 52500 based on receiving as input a particular dataset related to a particular surgical task.

[0237] The surgical computing device / edge computing device 52500 may anonymize (e.g., edit, randomize, summarize, average, etc.) at least a portion of the data before transmitting it to the enterprise cloud server 52540. The surgical computing device / edge computing device 52500 may perform the anonymization of the data based on the rules (e.g., privacy rules) of the location where the surgical computing device / edge computing device 52500 is located. When transmitting data outside the protected perimeter 52525 (e.g., outside the protected network perimeter), the surgical computing device / edge computing device 52500 may determine that the data needs to be modified based on the rules. The surgical computing device / edge computing device 52500 may anonymize (e.g., edit, randomize, summarize, average, etc.) the data based on a set of rules. In an example, a subset of the data (e.g., a subset of the data that is likely to be linked to the patient) may be anonymized, while another subset of the data may be sent in a non-anonymized form to the enterprise cloud server 52540 or any other device in the surgical system hierarchy for processing, as described, for example, in U.S. patent application having attorney docket number END9438USNP12, the entire disclosure of which is incorporated herein by reference.

[0238] An enterprise cloud server 52540 located outside the protected boundary 52525 may receive the request message 52520 with patient surgical information and / or surgical instrument information related to the surgical procedure. The enterprise cloud server 52540 may maintain a global or regional data structure (e.g., a global or regional database) of information associated with surgical procedures performed globally. In one example, the enterprise cloud server 52540 may compare the received information associated with the surgical procedure with one or more entries present in the data structure (e.g., entries already present in a database). Based on the comparison, the enterprise cloud server 52540 may generate global or regional surgical information 52535 (e.g., algorithms and / or recommendations) that is sent to the surgical computing device / edge computing device 52500. The surgical information stored in the enterprise cloud server 52540 may include diverse surgical information received from medical facilities across the world or geographic region. The global or regional surgical information 52535 provided by the global enterprise cloud server 52540 may include algorithms and parameters (e.g., patient data, healthcare provider data, surgical instrument data, etc.) of the surgical instruments 52505 performing autonomously configured surgical tasks. For example, the global or regional surgical information 52535 may include algorithms to be pushed to the surgical instruments / devices (e.g., smart surgical instruments / devices). The global or regional surgical information 52535 may also include an identification of the model of surgical instrument / device used and / or the settings used by the surgical instrument / device. For example, the identified surgical instrument may be a particular model of endocutter device, such as for performing an anastomosis in a surgical procedure. The setting used for the endocutter may be the firing rate setting.

[0239] In one example, the global or regional surgical information 52535 may include coordinates of a starting position of the surgical instrument 52505. In one example, the global or regional surgical information 52535 may include a set of coordinates that may be transmitted to the surgical computing device / edge computing device 52500. The surgical computing device / edge computing device 52500 may take into account when setting parameters associated with the movement of the surgical instrument 52505 (e.g., patient data, healthcare provider data, surgical instrument data, etc.).

[0240] In one example, machine learning may be used by the enterprise cloud server 52540 to generate global or regional surgical information 52535, for example, using a global machine learning model or subsystem 52517. In one example, the machine learning model 52517 (e.g., using deep learning) may use a surgical task and surgical information (e.g., surgical information associated with the surgical task) as input to predict a set of parameters to be used in the surgical procedure (e.g., parameters associated with patient information, healthcare provider information, surgical instrument information, etc.). The machine learning may also provide global or regional algorithms that may be pushed to surgical instruments and / or devices via the surgical computing device or edge computing device 52500. The machine learning predictions may be based on multiple (e.g., numerous) diverse datasets associated with surgical procedures that may have been performed on various patients across various globally diverse locations. The global machine learning model 52517 may use a global machine learning model, and / or a globally trained machine learning model may be utilized as part of a supervised learning framework, for example, as described herein in FIG. 8A . The training data (e.g., training examples 802 as shown in FIG. 8A) may include a set of training examples (e.g., input surgical information mapped to labeled outputs as shown in FIG. 8A). The training data used in training the global machine learning model may include surgical information collected from surgical procedures and / or simulated surgical procedures from across the world or region. The training data may include previous control algorithms associated with surgical instruments (e.g., stored globally and / or received from various medical facilities across the world or region). The training data may also include parameters associated with patients, medical professionals, and / or surgical instruments. In one example, the global ML model as output may provide a control algorithm and / or surgical instrument parameters associated with the surgical instrument (e.g., the firing rate of the surgical instrument).

[0241] The surgical computing device / edge computing device 52500 may analyze (e.g., using the analysis subsystem 52530) the global surgical information 52535 received from the enterprise cloud server. When evaluating the global surgical information 52535, the surgical computing device / edge computing device 52500 may access and / or consider local information. The local information may include information that was anonymized before being transmitted to the enterprise cloud server 52540. As described with respect to Figures 10 and 12, the surgical computing device / edge computing device 52500 may, for example, use the local information to modify the received global surgical information 52535.

[0242] In one example, the surgical computing device / edge computing device 52500 may have access to local surgical information, including information that may have been anonymized (e.g., edited, randomized, summarized, averaged, etc.) before transmission to the enterprise cloud server 52540 (e.g., enterprise cloud server). For example, the local data may be associated with the patient's fat percentage. This data may have been anonymized from the dataset transmitted to the remote server 52540 (e.g., enterprise cloud server) due to privacy rules (e.g., Health Insurance Portability and Accountability Act (HIPAA), Article 9 General Data Protection Regulation (GDPR), or data protection laws in the UK). Privacy rules may be used to protect health data, which is a special category of personal data and therefore receives a higher level of protection than other personal data.

[0243] After the surgical computing device / edge computing device 52500 receives global or regional surgical information 52535 related to performing a surgical task, the surgical computing device / edge computing device 52500 may consider local data related to the patient's fat percentage. The surgical computing device / edge computing device 52500 may adjust the global or regional surgical information 52535 based on the patient's fat percentage. The global or regional surgical information 52535 may include recommendations for setting one or more parameters of the surgical instrument 52505 (e.g., patient data, healthcare provider data, surgical instrument data, etc.) to particular values. For example, the surgical computing device / edge computing device 52500 may receive global or regional surgical information 52535 associated with setting an endocutter to a recommended firing rate. Taking into account the fat percentage (e.g., a high fat percentage that was not transmitted to the remote server), the surgical computing device / edge computing device 52500 may increase the firing rate before transmitting the firing rate as a parameter (e.g., patient data, healthcare provider data, surgical instrument data, etc.) to the surgical instrument 52505 (e.g., as a local surgical information message 52545). Modifying the global or regional surgical information 52535 may include adding a weight (e.g., a coefficient). For example, as shown in FIG. 9, A may be a constant value of 1.2, which may increase the firing rate of X by 0.2 or 20%.

[0244] In one example, the surgical computing device / edge computing device 52500 may override (e.g., completely override) global or regional information (e.g., global recommendations or algorithm changes) based on additional local information that may be anonymized and therefore not available to the enterprise cloud server. For example, the surgical computing device / edge computing device 52500 may determine that one of the patient-related parameters (e.g., the patient's blood pressure) has been sent to the enterprise cloud server 52540 in an edited form. The surgical computing device / edge computing device 52500 may also determine that the population where the surgical procedure is being performed is known to have a fat percentage that differs from the global average. Based on one or more of these determinations, the surgical computing device / edge computing device 52500 may determine that the global or regional surgical information 52535 received from the enterprise cloud server associated with the firing rate of the surgical instrument may not be suitable for the patient and may, for example, pose a serious risk to the patient. Accordingly, the surgical computing device / edge computing device 52500 may revise the surgical information provided by the enterprise cloud server 52540. The surgical computing device / edge computing device 52500 may then update the surgical information and, for example, change the firing rate or update the algorithm based on local patient information and / or demographic factors. In such cases, the surgical computing device / edge computing device 52500 may override the recommended firing rate with its own firing rate, which may be based on data private to the enterprise cloud server 52540 (e.g., data that has been anonymized before sending to the cloud). In one example, overriding the global recommendation or algorithm change may be made based on a mismatch between the global recommendation and a value generated by local machine learning within the surgical computing device / edge computing device 52500.

[0245] The parameters (e.g., patient data, healthcare provider data, surgical instrument data, etc.) or modified parameters may be transmitted from the surgical computing device / edge computing device 52500 to the surgical instrument 52505 in order for the surgical instrument to perform the surgical task (e.g., autonomously perform the surgical task). This may include a local machine learning model 52515 located locally on the surgical computing device / edge computing device 52500 or on the surgical instrument 52505. The machine learning model 52515 may use the parameters (e.g., patient data, healthcare provider data, surgical instrument data, etc.) to set instructions for the surgical instrument 52505.

[0246] The request message 52520 may be sent at the start of executing a surgical task (e.g., each of the surgical tasks). For example, the surgical computing device / edge computing device 52500 may recognize a transition stage from a first surgical task to a second surgical task and determine, via the analysis subsystem 52530, to send the request message 52520. In an example, the request message 52520 may be sent at periodic intervals throughout the execution of the surgical task. Sending the request message 52520 may be based on a trigger. For example, the error may be determined by the surgical computing device / edge computing device based on the performance of a surgical instrument. Determining the error is described in more detail in U.S. Patent Application No. 17 / 747,806, filed May 18, 2022, under the heading "METHOD OF CONTROLLING AUTONOMOUS OPERATIONS IN A SURGICAL SYSTEM," the disclosure of which is incorporated herein by reference in its entirety. Simulations can be used to determine thresholds (e.g., ideal thresholds). The simulation framework may be described in U.S. Patent Application No. 17 / 332,593, filed May 27, 2021, entitled "Method for Surgical Simulation," the disclosure of which is incorporated herein by reference in its entirety. If the error exceeds a threshold (e.g., a configured threshold), the surgical computing device / edge computing device 52500 may trigger a request message 52520 to be sent to a remote server 52540 (e.g., an enterprise cloud server). A cost analysis of the value of sending the request message 52520 and receiving globally sourced recommendations may be considered by the surgical computing device / edge computing device 52500. The surgical computing device / edge computing device 52500 may weight the benefits and costs of sending the request message 52520 and receiving global or regional surgical information 52535.The global or regional surgical information 52535 may be more accurate due to being generated from a global machine learning model with a more diverse training set.

[0247] In one example, the surgical computing device / edge computing device 52500 may take the recommendations received from the enterprise cloud server 52540 and modify (e.g., customize) them with patient-specific, population-specific, or surgeon-specific needs based on individualized data available within the protected network (e.g., local surgical data as described herein).

[0248] In one example, the local machine learning model 52515 may be capable of making local modifications (e.g., customizations) to globally provided recommendations or algorithms, for example, by adjusting a surgical instrument or surgical device based on local processing and local data.

[0249] In one example, the surgical computing device / edge computing device 52500 may have access to private interaction data of patients, staff, and other sensitive information. The surgical computing device / edge computing device may use that data to review and modify (e.g., customize) more global or regional algorithms that are supplied to it before the modified algorithms are pushed to the local surgical instruments or surgical devices 52505. In such cases, the global algorithms can benefit from the local private data without the data having to leave the protected local boundary 52525.

[0250] Global recommendations or algorithm changes may have pre-identified parameters or variables that can benefit from local procedure modifications, specific surgeon techniques, or subgroup patient data. These parameters may be identified within a pushed algorithm that includes the necessary programs or methods to compile local private data and insert them into a global algorithm update. For example, during a colon resection surgical procedure, the surgical computing device / edge computing device 52500 may identify that it will perform a defined procedure. As part of the surgical procedure, the surgical computing device / edge computing device 52500 may access the enterprise cloud server 52540 to request surgical information to be used (e.g., required) during the surgical procedure and one or more sets of default parameters associated with one or more surgical instruments or surgical devices. The surgical computing device / edge computing device 52500 may also obtain local parameters specific to the patient and / or demographics or the procedure or supply / inventory availability of the local medical facility. Such parameters may include characteristics that may be unique due to the demographics associated with the patient. Such parameters may also be unique due to procedures employed by local medical facilities and / or supplies / inventory available at those medical facilities.

[0251] As described herein, the surgical computing device / edge computing device 52500 may override, adjust, or modify global or regional information or parameters received from the enterprise cloud server 52540 with local variables. The global or regional information or parameters may be modified, for example, based on laws, procedures, technologies, and / or devices available within the medical facility. In one example, device targets / limits may be changed based on demographics and / or other patient information associated with a patient to modify (e.g., change / shift) initial or default settings of a surgical instrument or surgical device. Global / regional parameters may be set based on surgical information collected from surgical procedures performed worldwide or across a region. The surgical computing device / edge computing device 52500 may modify (e.g., shift, weight, or change) global variables using locally available information, for example, to optimize performance.

[0252] In one example, the surgical computing device / edge computing device 52500 may provide anonymized surgical information (e.g., a data set) to the enterprise cloud server 52540. Based on the surgical information provided by various surgical computing devices / edge computing devices around the world or region, such surgical information may enable the enterprise cloud server to determine that there are patterns and relationships, for example, between the orientation of two linear staple lines relative to one another for the next step of circular staple approximation and firing. This relationship may be highlighted in the surgical procedure plan 52510 or approach for colorectal leak rates and increasing circular device firing force (FTF) or clamp force (FTC). By considering additional surgical information (e.g., annotated video), the system may determine staple line patterns that correlate well with firing force abnormalities that may correlate with elevated leak rates.

[0253] The enterprise cloud server 52540 may determine that additional factors, in addition to alignment, may contribute to the results (e.g., due to statistical probability explaining some of the variance in the results). In such cases, the enterprise cloud server 52540 may determine staple line alignment (e.g., as seen through the scope) as well as recommendations (e.g., new recommendations) regarding firing force thresholds and responses from the smart circular staples. The enterprise cloud server 52540 may push parameter values ​​and / or control algorithm updates to the surgical computing device / edge computing device 52500 to push or forward them to smart surgical instruments or smart surgical devices that may be connected with the surgical computing device or edge computing device, or as they connect with the surgical computing device or edge computing device. The enterprise cloud server 52540 may indicate to the surgical computing device or edge computing device 52500 that there may be relevant data that the server may not have considered. The enterprise cloud server 52540 may look for the causes of these problems and make recommendations to the surgical or edge computing device 52500 to correct or adjust them (e.g., if possible).

[0254] A surgical or edge computing device 52500 located within a medical facility's network may identify additional relationships between various parameters that may be part of the non-anonymized surgical information. The non-anonymized surgical information may include more complete patient medical record access than is available to an enterprise cloud server (e.g., a compiled patient medical record sent to the cloud). In one example, the surgical or edge computing device 52500 may determine that a combination of surgical information associated with a patient (e.g., the patient's blood pressure) and a medical professional's technique regarding colonic mobility may be correlated with an outcome. In one example, the surgical or edge computing device 52500 may modify or adjust global parameters or control algorithm adjustments with additional local updates, for example, based on their use or population, resulting in local modification (e.g., customization) of the pushed algorithm.

[0255] In one example, the medical facility may identify advanced conditions that may result in local modification or alteration of received global or regional surgical parameter value updates and / or control algorithm updates. In one example, the surgical computing device / edge computing device 52500 may send the modified (e.g., customized) control algorithm to the enterprise cloud server 52540 without including private patient information. The enterprise cloud system may then push it to other surgical computing devices / edge computing devices (e.g., automatically or upon request). In one example, the enterprise cloud system may compare the modified or changed surgical information or control algorithm with what was previously pushed to determine additional modifications (e.g., customizations) and initiate a learning process that looks for these correlations with data accessible to the enterprise cloud system.

[0256] 10 illustrates an example of a message sequence diagram showing the communication (e.g., receiving and / or transmitting) and modification / customization / alteration of global or regional information at a local device, for example, a surgical computing device / edge computing device 52500 located within a protected boundary 52525. Global or regional information and globally or regionally provided information may be used interchangeably herein.

[0257] As shown in Figure 10, a surgical computing device / edge computing device 52500 may be provided that may be the same as the surgical computing device / edge computing device 52500 described with respect to Figure 9. The surgical computing device / edge computing device 52500 may be located within the hospital's internal network 52525, which is protected (e.g., under HIPAA rules as described herein).

[0258] The surgical instrument 52505 associated with the surgical computing device / edge computing device 52500 may be used to perform a surgical procedure (e.g., autonomously perform a surgical procedure). The surgical instrument 52505 may also be located within a protected boundary 52525, as described herein. The enterprise cloud server 52540 may be located outside the protected boundary 52525. Surgical information (e.g., surgical information associated with a patient, a medical professional, or a surgical instrument) transmitted to the enterprise cloud server 52565 may be vulnerable to misuse. Such data within the protected network 52525 (e.g., data exchanged between the surgical instrument 52505 and the surgical computing device / edge computing device 52500) may be exchanged without modification, while data transmitted outside the protected boundary 52525 may be modified. For example, as described with respect to Figure 10, the surgical information sent to an entity (e.g., enterprise cloud server 52540) may be anonymized (e.g., redacted, summarized, etc.), randomized, encrypted, and / or manipulated. The surgical information may be anonymized so that the data cannot be traced back to the patient.

[0259] At 52550, the surgical computing device or surgical edge computing device 52500 may establish an authentication session with the enterprise cloud server 52540. To establish the authentication session, the surgical computing device or surgical edge computing device 52500 may register with and perform authentication with 52540. In one example, authentication may be performed by using message hash model-based encryption to achieve desired network latency and security during surgical information exchange between the surgical computing device or surgical edge computing device 52500 and the enterprise cloud server 52540. In one example, the surgical computing device or surgical edge computing device 52500 may be pre-configured with authentication information, thus minimizing end-to-end delay to create a secure communication interface between the devices.

[0260] At 52552, the surgical computing device / edge computing device 52500 (e.g., surgical computing devices / edge computing devices distributed across a region or the world) may transmit surgical information to the enterprise cloud server 52540. The surgical information may include surgical information associated with one or more surgical instruments / devices 52505, patient-related surgical information, medical professional-related surgical information, etc. In one example, the surgical computing device / edge computing device 52500 may transmit the surgical information periodically, for example, based on a configured period of time. In one example, the surgical computing device / edge computing device 52500 may transmit the surgical information aperiodically, for example, as an update based on newly acquired local surgical information, for example, parameters or control program algorithms associated with a surgical instrument or device related to the outcome of the surgical procedure. In one example, the surgical computing device / edge computing device 52500 may transmit the surgical information aperiodically, for example, based on a request from the enterprise cloud server 52540.

[0261] At 52575, the surgical computing device / edge computing device 52500 may generate a request to receive recommendations regarding surgical information related to the surgical procedure (e.g., default parameters, control algorithms, etc.). The surgical computing device / edge computing device 52500 may generate the request as part of the surgical procedure (e.g., as a first step of the surgical procedure). At 52576, the surgical computing device / edge computing device 52500 may send the request to the enterprise cloud server 52540. The request may include an identification of the surgical task, the surgical instrument.

[0262] At 52577, the surgical computing device / edge computing device 52500 may receive recommendations regarding surgical information (instrument / device configuration parameters) related to the surgical procedure from the enterprise cloud server 52540. In one example, the recommendations may be received in response to a request sent by the surgical computing device / edge computing device 52500 or may be pushed autonomously (e.g., periodically) by the enterprise cloud server 52540.

[0263] At 52580, the surgical computing device / edge computing device 52500 may modify / alter the received recommendation based on the local surgical information as described herein. At 52582, the surgical computing device / edge computing device 52500 may transmit the modified / altered recommendation to one or more of the surgical instruments / devices 52505.

[0264] Figure 11 illustrates an example of the relationship between a surgical computing device / edge computing device 52500 and an enterprise cloud server 52540 (e.g., an enterprise cloud server). As shown in Figure 11, the surgical computing device / edge computing device 52500 may include, among other things, a processor 52620, a memory 52600 (e.g., non-removable memory and / or removable memory), an analysis subsystem 52530, a local machine learning model 52515, and / or a local storage subsystem 52610. It will be understood that the surgical computing device / edge computing device 52500 may include any sub-combination of the foregoing elements / subsystems while remaining consistent with an embodiment.

[0265] The processor 52620 in the surgical computing device / edge computing device 52500 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 52620 may perform data processing, authentication, input / output processing, and / or any other function that may enable the surgical computing device / edge computing device 52500 to operate in an environment suitable for performing a surgical procedure. The processor 52620 may be coupled to a transceiver (not shown). The processor 52620 may use a transceiver (not shown in the drawings) to communicate with the enterprise cloud server 52540.

[0266] The processor 52620 in the surgical computing device / edge computing device 52500 may access information from and store data in any type of suitable memory (e.g., non-removable and / or removable memory). Non-removable memory may include random-access memory (RAM), read-only memory (ROM), a hard disk, a solid-state drive, or any other type of memory storage device. Removable memory may include secure digital memory.

[0267] The processor 52620 in the surgical computing device / edge computing device 52500 may access information from and store data in the extended storage 52610 (e.g., non-removable memory and / or removable memory). In one example, the processor 52620 may access information from and store data in memory that is not physically located on the surgical computing device / edge computing device 52500, such as on a server or a secondary edge computing system (not shown).

[0268] 11 , enterprise cloud server 52540 may include, among other things, processor 52650, memory 52625 (e.g., non-removable memory and / or removable memory), analysis subsystem 52630, global machine learning model 52517, and / or storage subsystem 52660. It will be understood that enterprise cloud server 52540 may include any sub-combination of the foregoing elements / subsystems while remaining consistent with an embodiment.

[0269] The processor 52650 in the enterprise cloud server 52540 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 52650 may perform data processing, authentication, input / output processing, and / or any other function that may enable the enterprise cloud server 52540 to operate in an environment suitable for performing a surgical procedure. The processor 52650 in the enterprise cloud server 52540 may be coupled with a transceiver (not shown). The processor 52650 in the enterprise cloud server 52540 may use the transceiver to communicate with the surgical computing device / edge computing device 52500, for example, via a secure interface as described herein.

[0270] The processor 52650 in the enterprise cloud server 52540 may access information from and store data in any type of suitable memory (e.g., non-removable and / or removable memory). Non-removable memory may include random-access memory (RAM), read-only memory (ROM), a hard disk, a solid-state drive, or any other type of memory storage device. Removable memory may include secure digital memory.

[0271] The processor 52650 in the enterprise cloud server 52540 may access information from and store data in the extended storage 52660 (e.g., non-removable memory and / or removable memory). In one example, the processor 52650 in the enterprise cloud server 52540 may access information from and store data in memory that is not physically located on the enterprise cloud server 52540, such as on a server or a secondary edge computing system (not shown).

[0272] 11 , surgical information (e.g., including surgical instrument setting parameter values, control program algorithms, and / or updates associated with control program algorithms) may be transmitted to and / or received from the surgical computing device / edge computing device 52500 to the enterprise cloud server 52540. In examples, the surgical information may pass through an application programming interface 52595 (API), which may be available after establishing a secure interface between the surgical computing device / edge computing device 52500 and the enterprise cloud server 52540, for example, as described herein. The surgical information may include measurements obtained from sensors, actuators, robotic movements, biomarkers, surgeon biomarkers, visual aids, and / or others. The surgical information may also include medical professional-related information and / or patent-related information, for example, obtained from a billing subsystem or database. Wearables are described in more detail in U.S. Patent Application No. 17 / 156,28, filed November 10, 2021, under the title "Monitoring of adjusting a surgical parameter based on biomarker measurements," the disclosure of which is incorporated herein by reference in its entirety.

[0273] 12 shows an example of a flowchart for a surgical computing device / edge computing device 52500 coordinating or modifying global or regional surgical information provided by an enterprise cloud server 52540. The surgical computing device / edge computing device 52500 may be located inside a protected network (e.g., a HIPAA protected network) and the enterprise cloud server 52540 may be located outside the protected network.

[0274] At 52662, the surgical computing device / edge computing device 52500 may receive global or regional surgical information associated with the surgical procedure (e.g., one or more surgical tasks of the surgical procedure) from the enterprise cloud server 52540. In one example, the surgical computing device / edge computing device 52500 may receive the global or regional surgical information in response to a request message sent by the surgical computing device / edge computing device 52500 to the enterprise cloud server 52540. The request message may be generated based on the occurrence of a trigger event.

[0275] At 52664, the surgical computing device / edge computing device 52500 may obtain local surgical information (e.g., from the surgical instrument). The local surgical information may be associated with the patient and / or the patient's location. The local surgical information may include at least one of demographics, local medical procedures, supplies or inventory status, or control algorithms associated with the surgical instrument. The local surgical data may be based on characteristics of the local surgical procedure.

[0276] At 52666, the surgical computing device / edge computing device 52500 may adjust or modify at least a portion of global or regional surgical information associated with the local surgical procedure and / or patient. In one example, adjusting or modifying the portion of the global or regional surgical information may include adjusting or modifying a global control algorithm using at least one local update. In one example, the portion of the global or regional surgical information may be adjusted or modified based on at least one of privacy laws, procedures, techniques, or device availability within the medical facility where the surgical procedure is being performed. In one example, adjusting at least a portion of the global or regional surgical information may be based on a neural network analysis of the global or regional surgical information, local surgical data, and / or patient-related data. The neural network may be trained using the global or regional surgical information, local surgical information, and patient-related surgical information to determine how to adjust at least a portion of the global or regional surgical information.

[0277] The surgical computing device / edge computing device 52500 may transmit the adjusted global or regional surgical information to the surgical instrument at 52668. In one example, the adjusted global or regional control algorithm received from the enterprise server 52540 may be transmitted to the surgical instrument.

[0278] As used herein, the term "local surgical information" refers to surgical information generated within a particular hospital or medical facility. Local surgical information may include surgical information protected by local protection rules. Local surgical information may originate and / or be stored / processed within the protected boundaries or network of a medical facility. Thus, local surgical information may include surgical information protected by local protection rules (e.g., General Data Protection Regulation (GDPR), Health Insurance Portability and Accountability Act (HIPAA), etc.). Local surgical information may also include patient information associated with the patient's location (e.g., based on population characteristics in the region).

[0279] As used herein, the term "regional surgical information" refers to information generated within a geographic region, e.g., within hospitals / healthcare facilities within a given county or country, continent, or region of a continent. The regional surgical information may or may not originate within a protected network or healthcare facility and / or may or may not be stored / processed within a protected network or healthcare facility. This depends on whether the computing devices used to process the regional surgical information are located within or outside the protected boundaries or network of the healthcare facility.

[0280] As used herein, the term "global surgical information" refers to data generated from hospitals / healthcare facilities located anywhere in the world, for example, from multiple countries. Global surgical information may include surgical information that is not protected by local protection rules.

[0281] As used herein, a "surgical procedure" includes a series of surgical steps or tasks. The terms "steps" and "tasks" are used interchangeably herein.

[0282] The following is a non-exhaustive list of embodiments that may or may not be claimed. 1. A surgical computing device comprising: a processor, the processor comprising: receiving global or regional surgical information associated with the surgical procedure; obtaining local surgical information associated with the surgical procedure, the local surgical information being associated with a patient and a position of the patient; adjusting a portion of the global or regional surgical information, the portion of the global or regional surgical information being adjusted based on the local surgical information; A surgical computing device configured to transmit a coordinated portion of the global or regional surgical information to a surgical instrument associated with a surgical procedure. 2. The processor: generating a request message requesting global or regional surgical information; Sending a request message to the enterprise cloud server; 10. The surgical computing device of embodiment 1, further configured to receive global or regional surgical information from an enterprise cloud server in response to a request message. 3. A surgical computing device as described in embodiment 2, wherein the request message includes a request for a set of default parameters or a control algorithm update used by at least one surgical instrument associated with the surgical procedure. 4. A surgical computing device as described in embodiment 2, wherein the request message is generated based on the occurrence of a trigger event, the trigger event being a transition stage from a first surgical step of the surgical procedure to a second surgical step of the surgical procedure. 5. A surgical computing device as described in embodiment 2, wherein the surgical computing device is located inside a protected network and the corporate cloud server is located outside the protected network. 6. The surgical computing device of embodiment 5, wherein the protected network is protected based on local privacy laws associated with the patient's location. 7. A surgical computing device as described in embodiment 1, wherein the local surgical information includes at least one of demographics, local medical procedures, or supply or inventory status. 8. A surgical computing device as described in embodiment 1, wherein a portion of the global or regional surgical information is further tailored based on at least one of privacy laws, procedures, techniques, or device availability within the medical facility where the surgical procedure is being performed. 9. A surgical computing device as described in embodiment 1, wherein being configured to adjust at least a portion of the global or regional surgical information includes being configured to adjust a global control algorithm using at least one local update. 10. A method implemented by a surgical computing device, the method comprising: receiving global or regional surgical information associated with the surgical procedure; obtaining local surgical information associated with the surgical procedure, the local surgical information being associated with a patient and a location of the patient; adjusting a portion of the global or regional surgical information, the portion of the global or regional surgical information being adjusted based on the local surgical information; and transmitting the coordinated portion of the global or regional surgical information to a surgical instrument associated with the surgical procedure. 11. generating a request message requesting global or regional surgical information; sending a request message to an enterprise cloud server; 11. The method of embodiment 10, further comprising receiving global or regional surgical information from an enterprise cloud server in response to the request message. 12. The method of embodiment 11, wherein the request message includes a request for a set of default parameters or a control algorithm update used by at least one surgical instrument associated with the surgical procedure. 13. The method of embodiment 11, wherein the request message is generated based on the occurrence of a trigger event, the trigger event being a transition stage from a first surgical step of the surgical procedure to a second surgical step of the surgical procedure. 14. The method of embodiment 11, wherein the surgical computing device is located inside the protected network and the corporate cloud server is located outside the protected network. 15. The method of embodiment 14, wherein the protected network is protected based on local privacy laws associated with the patient's location. 16. The method of embodiment 10, wherein the local surgical information includes at least one of demographics, local medical procedures, or supply or inventory status. 17. The method of embodiment 10, wherein a portion of the global or regional surgical information is further tailored based on at least one of privacy laws, procedures, techniques, or device availability within the medical facility where the surgical procedure is being performed. 18. The method of embodiment 10, wherein adjusting a portion of the global or regional surgical information includes adjusting a global control algorithm using at least one local update.

[0283] [Embodiment] (1) A method implemented by a processor of a first surgical computing device configured to couple to a surgical device for performing surgical tasks of a surgical procedure and to a second surgical computing device, the method comprising: receiving, from the second surgical computing device, parameter values ​​and / or control algorithms for the surgical device based on the surgical task of the surgical procedure; obtaining patient information, said patient information including parameters associated with a patient and details of a surgical procedure performed / being performed on said patient; adjusting the parameter values ​​and / or control algorithms based on the patient information; and transmitting the adjusted parameter values ​​and / or control algorithms to the surgical device. (2) The method of embodiment 1, further comprising updating the surgical device by setting the adjusted parameter values ​​and / or control algorithms. (3) The method of any one of claims 1 to 2, further comprising receiving an indication of pre-identified parameters or variables of the control algorithm that may be adjusted by the processor, wherein the processor is configured to adjust the pre-identified parameters or variables of the control algorithm based on the patient information. (4) transmitting a portion of the patient information to the second surgical computing device; or enabling the second surgical computing device to access a portion of the patient information; A method according to any one of claims 1 to 3, wherein the second surgical computing device is configured to generate the parameter values ​​and / or control algorithms based on the portion of the patient information. (5) A method according to any one of embodiments 1 to 4, wherein the first surgical computing device is located within a privacy protection boundary or protected network, and the second surgical computing device is located outside the privacy protection boundary or protected network.

[0284] (6) A method according to any one of embodiments 1 to 5, wherein the received parameter values ​​and / or control algorithms are based on a regional or global analysis of past treatments of patients who have undergone the surgical procedure. (7) generating a request message requesting parameter values ​​and / or control algorithms; sending the request message to the second surgical computing device; A method according to any one of embodiments 1 to 6, further comprising receiving the parameter values ​​and / or control algorithms in response to the request message. (8) The method of embodiment 7, wherein the request message includes a redacted or anonymized form of the local information. (9) receiving image / video data from a visualization device used during said surgical procedure; and A method as described in any one of embodiments 1 to 8, further comprising transmitting the image / video data to the second surgical computing device, wherein the parameter values ​​and / or control algorithms are based on interpretation of the image / video data. (10) The method described in embodiment 7 or 8, wherein the request message is generated based on the occurrence of a trigger event, the trigger event being a transition stage from a first surgical task of the surgical procedure to a second surgical task of the surgical procedure.

[0285] (11) The method of embodiment 5, wherein the protected network is protected based on local privacy laws associated with the patient's location. (12) The method of any one of claims 1 to 11, wherein the patient information includes at least one of demographics, medical treatment of the patient, or supplies or inventory for the patient treatment. (13) A method according to any one of embodiments 1 to 12, wherein the parameter values ​​and / or control algorithms are further adjusted based on at least one of privacy laws, procedures, techniques, or device availability within the medical facility where the surgical procedure is being performed. (14) A method according to any one of embodiments 1 to 13, further comprising transmitting the adjusted parameter values ​​and / or control algorithms to the second surgical computing device. (15) A first computing system configured to couple with a surgical device and a second surgical computing device for performing surgical tasks of a surgical procedure, the first surgical computing device comprising: A first computing system comprising a processor configured to execute a method according to any one of embodiments 1 to 14.

[0286] (16) A computing program that, when executed by a processor, causes the processor to perform the method described in any one of embodiments 1 to 14.

Claims

1. 1. A method implemented by a processor of a first surgical computing device configured to couple to a surgical device for performing surgical tasks of a surgical procedure and to a second surgical computing device, the method comprising: receiving, from the second surgical computing device, parameter values ​​and / or control algorithms for the surgical device based on the surgical task of the surgical procedure; obtaining patient information, said patient information including parameters associated with a patient and details of a surgical procedure to be performed / being performed on said patient; adjusting the parameter values ​​and / or control algorithms based on the patient information; transmitting the adjusted parameter values ​​and / or control algorithms to the surgical device.

2. The method of claim 1 , further comprising updating the surgical device by setting the adjusted parameter values ​​and / or control algorithms.

3. 10. The method of claim 1, further comprising receiving an indication of pre-identified parameters or variables of the control algorithm that may be adjusted by the processor, the processor being configured to adjust the pre-identified parameters or variables of the control algorithm based on the patient information.

4. transmitting a portion of the patient information to the second surgical computing device; or enabling the second surgical computing device to access a portion of the patient information; The method of claim 1 , wherein the second surgical computing device is configured to generate the parameter values ​​and / or control algorithms based on the portion of the patient information.

5. 10. The method of claim 1, wherein the first surgical computing device is located within a privacy-protecting boundary or protected network and the second surgical computing device is located outside the privacy-protecting boundary or protected network.

6. The method of claim 1 , wherein the received parameter values ​​and / or control algorithms are based on a regional or global analysis of past procedures of patients who have undergone the surgical procedure.

7. generating a request message requesting parameter values ​​and / or control algorithms; sending the request message to the second surgical computing device; The method of claim 1 , further comprising: receiving the parameter values ​​and / or control algorithms in response to the request message.

8. The method of claim 7 , wherein the request message includes a redacted or anonymized form of the local information.

9. receiving image / video data from a visualization device used during said surgical procedure; 10. The method of claim 1, further comprising transmitting the image / video data to the second surgical computing device, wherein the parameter values ​​and / or control algorithms are based on interpretation of the image / video data.

10. 8. The method of claim 7, wherein the request message is generated based on the occurrence of a trigger event, the trigger event being a transition stage from a first surgical task of the surgical procedure to a second surgical task of the surgical procedure.

11. The method of claim 5 , wherein the protected network is protected based on local privacy laws associated with the patient's location.

12. The method of claim 1 , wherein the patient information includes at least one of demographics, a medical treatment of the patient, or supplies or inventory for the patient treatment.

13. 10. The method of claim 1, wherein the parameter values ​​and / or control algorithms are further adjusted based on at least one of privacy laws, procedures, techniques, or device availability within a medical facility where the surgical procedure is being performed.

14. The method of claim 1 , further comprising transmitting the adjusted parameter values ​​and / or control algorithms to the second surgical computing device.

15. 1. A first computing system configured to couple with a surgical device and a second surgical computing device for performing surgical tasks of a surgical procedure, the first surgical computing device comprising: A first computing system comprising a processor configured to perform the method of any one of claims 1 to 14.

16. A computing program which, when executed by a processor, causes said processor to carry out a method according to any one of claims 1 to 14.

Images