Automated compilation, annotation, and distribution of surgical data to a system for predicting related automated actions

Abstract

A system, method, and means are disclosed for automatically compiling, annotating, and distributing surgical data to a system to anticipate related automated actions. Surgical data can be selectively transmitted to a surgical system, for example, to perform autonomous tasks (e.g., without being requested for data). A surgical computing system can obtain surgical data from the surgical system. The surgical computing system can annotate the surgical data using surgical context data. The surgical computing system can determine data needs associated with subsequent target system tasks associated with the target system. The surgical computing system can determine the target system, for example, based on the annotated surgical data. The surgical computing system can generate a data package associated with the data needs, the data package can include a portion of the annotated surgical data, and the surgical computing system can transmit the data package to the target system.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application is related to the following applications filed simultaneously, the content of each of which is incorporated herein by reference. · Attorney Docket No. END9430USNP1, titled "METHOD OF CONTROLLING AUTONOMOUS OPERATIONS IN A SURGICAL SYSTEM". · Attorney Docket No. END9430USNP8, titled "AUTOMATIC COMPILATION, ANNOTATION, AND DISSEMINATION OF SURGICAL DATA TO SYSTEMS TO ANTICIPATE RELATED AUTOMATED OPERATIONS". · Attorney Docket No. END9430USNP9, titled "AGGREGATION OF PATIENT, PROCEDURE, SURGEON, AND FACILITY PRE - SURGICAL DATA AND POPULATION AND ADAPTATION OF A STARTING PROCEDURE PLAN TEMPLATE". · Attorney Docket No. END9430USNP10, titled "IDENTIFICATION OF IMAGES SHAPES BASED ON SITUATIONAL AWARENESS OF A SURGICAL IMAGE AND ANNOTATION OF SHAPES OR PIXELS".

Background Art

[0002] Surgical operations are typically performed in an operating room or room within a medical facility such as a hospital. Various surgical devices and systems are utilized in the performance of surgical operations. In the digital information age, medical systems and facilities often implement systems or surgeries using newer and improved technologies more slowly due to the general desire to maintain patient safety and conventional practices.

Summary of the Invention

Means for Solving the Problems

[0003] Systems, methods, and means are disclosed for the automatic compilation, annotation, and distribution to a system of surgical data to anticipate related automated actions. Surgical data can be selectively transmitted to a surgical system, for example, to perform autonomous tasks (e.g., without being requested for data). A surgical computing system can obtain surgical data from the surgical system. The surgical computing system can annotate the surgical data using surgical context data. The surgical computing system can determine data needs associated with subsequent target system tasks associated with the target system. The surgical computing system can determine the target system, for example, based on the annotated surgical data. The surgical computing system can generate a data package associated with the data needs (e.g., selectively differentiated data), and the data package can include a portion of the annotated surgical data. The surgical computing system can transmit the data package to the target system.

[0004] The surgical computing system may be configured to determine surgical context data based on, for example, surgical data. The surgical computing system may be configured to transmit an instruction indicating the current surgical step to the target system.

[0005] The surgical computing system may be configured to determine a risk level based on the surgical data. The data needs may be determined, for example, based on the risk level.

[0006] A surgical computing system may be configured to perform edits on at least a portion of surgical data and / or annotated surgical data. For example, the surgical computing system may determine a classification associated with the surgical data. The classification may indicate private and / or confidential information (e.g., regarding HIPAA). The surgical computing system may determine, for example, based on the type of the target device and / or the geographical location associated with the target device (e.g., outside of a HIPAA boundary), that the target system may not receive confidential information. The surgical computing system may perform edits on a portion of the surgical data, for example, before transmitting a data package to the target system.

[0007] A computer-implemented method, comprising: obtaining surgical data associated with a surgical procedure from a plurality of surgical systems; annotating the surgical data using surgical context data related to the surgical procedure; determining data needs for subsequent target surgical system tasks associated with a target surgical system, at least partially based on the annotated surgical data; generating a data package associated with the data needs, the data package including a subset of the annotated surgical data; and transmitting the data package to the target surgical system. Annotating the surgical data with the surgical context data may mean pairing the surgical data with the surgical context data so that context is provided to the surgical data. In one example, the computer can process the annotated surgical data to read out the surgical data and an indication of the context to which the surgical data relates. In one example, the data needs correspond to the data used for the task. In one example, the data needs for future target surgical system tasks are the data used for that task (associated with the target surgical system). At the broadest level, the target surgical system is simply the surgical system associated with (e.g., performing or monitoring) each surgical task. A subsequent target surgical system task can be a future surgical task to be performed after the current task of the surgical procedure. In one example, generating the data package includes selectively discriminating the annotated data, for example, by decompiling the annotated data so that only a subset associated with the data needs remains. In other words, a second subset of the annotated data not associated with the data needs is removed / ignored. Advantageously, annotating the surgical data to provide context to the data enables the processor to more quickly identify a given context, e.g., a subset of that data related to a particular surgical step.Advantageously, by pre - determining the data requirements for a surgical task and by sending a data package that includes data that meets those requirements (rather than all of the acquired data), the target surgical system associated with that task is provided with relevant data in advance and without extra data. This reduces the delay in performing the surgical task because it is no longer necessary to analyze all of the data acquired to find the relevant subset. The process of determining the data requirements itself is made more efficient by the previous annotation step, and thus there is a cumulative time - efficiency benefit across all that is claimed.

[0008] The method may further include determining surgical context data based on surgical data. The surgical context data may include an indication of the current surgical step in the surgery. The method may further include determining an indication of a subsequent target system task associated with the target surgical system. Advantageously, determining surgical context data based on surgical data eliminates the need for a medical expert to enter context, thereby automating the surgery, increasing its efficiency, and reducing delays. This then improves the surgical outcome. In one example, it may be determined that the current surgical step is a major vessel management step in a lobectomy and the subsequent task is to cut a fissure.

[0009] The method may further include determining a subsequent target system task based on annotated surgical data. Advantageously, determining a subsequent task based on annotated data enables the data needs to be determined without input from a medical expert, thereby automating the surgery, increasing its efficiency, and reducing delays. This then improves the surgical outcome.

[0010] Determining the data needs may include determining the data needs based on at least one of prioritization data, system usage data, or hierarchical segmentation information.

[0011] The method may further include determining a risk level based on the obtained surgical data. Determining data needs may include determining data needs based on the determined risk level.

[0012] The subset of annotated surgical data may be a first subset of the annotated surgical data. The method may further include determining a classification associated with a second subset of the annotated surgical data. The method may further include performing an edit on the second subset of the annotated surgical data based at least on the classification associated with the second subset of the annotated surgical data.

[0013] The subset of annotated surgical data may be a first subset of the annotated surgical data. The method may further include determining a classification associated with a second subset of the annotated surgical data, wherein the first subset of the data at least partially includes the second subset of the data, and the classification is associated with private data or outlier data. The method may further include performing an edit on the second subset of the surgical data based at least on the classification associated with the second subset of the annotated surgical data.

[0014] The subset of annotated surgical data may be a first subset of the annotated surgical data. The method may further include determining whether the second subset of the annotated surgical data is private data. The method may further include determining whether the target system is outside the patient data privacy protection boundary. The method may further include performing an edit on the second subset of the surgical data based on the condition that the second subset of the annotated surgical data is private data and the target system is outside the patient data privacy protection boundary.

[0015] The method may further include transmitting a second subset of the annotated surgical data to local storage. The second subset of the annotated surgical data may be stored before the editing is performed.

[0016] The plurality of surgical systems may include one or more of surgical instruments, monitoring systems, surgical sensors, or surgical devices.

[0017] The target system may be a surgical instrument. Subsequent target system tasks may be associated with using the surgical instrument. The data needs may include data associated with the use of the surgical instrument. Advantageously, automating the delivery of relevant data for subsequent surgical tasks performed by the instrument to the surgical instrument reduces delays in the surgery.

[0018] The target system may be a facility system. Subsequent target system tasks may be facility maintenance. The data needs may include one or more of the surgical instruments used, the surgical consumables used, the surgical plan, or the surgical schedule.

[0019] A surgical computer system comprising a processor configured to perform any of the methods described above is described.

[0020] A computer program comprising instructions that, when the program is executed by a controller, cause the controller to perform any of the methods described above is described.

[0021] A computer-readable medium comprising instructions that, when executed by a controller, cause the controller to perform any of the methods described above is described.

[0022] Systems, methods, and means for situation recognition of surgical images and identification of image shapes based on annotation of shapes or pixels are disclosed. Surgical videos associated with a surgical procedure can be acquired. Surgical context data for a surgical procedure can be acquired. Elements within a video frame can be identified using image processing based on the surgical context data. Annotation data can be determined and generated for a video frame based on, for example, the surgical context data and the identified element(s).

[0023] Video frames within a surgical video can include pixels. The identified element(s) can correspond to respective groups of pixels. The identified element(s) can be separated into sub-elements each including an element. Sub-elements can be composed of sub-groups of pixels associated with each sub-element.

[0024] Annotation data may be inserted into a video frame. For example, annotation data can be inserted at the pixel level (e.g., each annotation data can be associated with each pixel and / or group of pixels). For example, annotation data can include one or more of element identification information, element grouping information, element sub-grouping information, element type information, element description information, element condition information, surgical step information, surgical task information, surgical event information, surgical instrument information, surgical device information, and / or the like.

[0025] Surgical context data can be refined and / or updated based on the surgical video. For example, surgical context data can be refined and / or updated based on the identified element(s) within a video frame. Elements within another video frame can be identified using, for example, the refined and / or updated surgical context data (e.g., using image processing). Annotation data for a second video frame can be determined using the refined surgical context data and the identified element(s) within the second video frame.

[0026] Tracking information associated with a surgical video can be determined, for example, by analyzing a plurality of video frames. Elements identified within a first video frame and elements identified within a second video frame can be used, for example, to determine tracking data. The tracking data may be associated with element behavior, element movement, and / or outcome information (e.g., associated with the element). Annotation data may include the determined tracking data.

[0027] The tracking data and / or information may be verified, for example, using predicted tracking information. The predicted tracking information may be obtained, for example, based on surgical context data and / or a surgical plan. The annotation data may include an indication as to whether the elements and / or the tracking data have been verified. For example, the annotation data can indicate that a subset of the identified elements has been verified. The annotation data can indicate that a subset of the identified elements has not been verified.

[0028] Jobs, outcomes, and / or constraints may be determined for a surgery, for example, based on a surgical video. Jobs, outcomes, and / or constraints may be determined based on surgical context data. Surgical task information associated with a first video frame can be determined using the surgical context data and the identified elements within the first video frame. Comorbidity information may be determined, for example, in relation to the determined surgical tasks using the surgical context data, the surgical task information, and the identified elements within a first surgical video frame. Outcome information associated with a surgical task may be determined, for example, based on the surgical context data, the surgical task information, and the comorbidity information.

[0029] Change and / or control parameters can be determined for a surgical procedure by analyzing a surgical video. The change and / or control parameters may be determined, for example, based on surgical task information, complication information, and / or outcome information. A determination can be made to change parameters associated with the surgical procedure. The parameters can include parameters associated with operating a surgical instrument, a surgical device, and / or the like. Based on a determination to change parameters associated with the surgical procedure, the change and / or control parameters can be determined. A control signal may be generated. The control signal can include an instruction indicating the determined control parameters. The control signal may be transmitted, for example, to a surgical control system that transmits parameter information to a surgical system (e.g., a surgical instrument, a surgical device, etc.). The change and / or control parameters can be used to perform the surgical procedure.

[0030] A computer-implemented method is described. The method includes obtaining surgical context data related to a surgical procedure, obtaining a surgical video of the surgical procedure including one or more surgical video frames, using image processing based on the obtained surgical context data to identify one or more elements within a first surgical video frame, wherein the one or more elements each include respective groups of pixels, and generating annotation data for the first surgical video frame based on the surgical context data and the one or more elements within the first surgical video frame, wherein the annotation data includes respective element annotation data associated with each of the identified one or more elements. Advantageously, the surgical context data is used to identify elements within the video frame and generate annotation data associated with those elements to flag relevant elements of the surgical video with data related to the surgical context. This helps a healthcare provider more quickly / easily identify important features within the video and understand their context simultaneously. This then enables more rapid decision-making and better surgical outcomes. The surgical context data related to the surgical procedure may indicate a surgical procedure, surgical event, surgical step, surgical phase, surgical task, surgical device, complication, and / or the like.

[0031] The method may further include inserting the annotation data for the first surgical video frame into the surgical video, wherein the respective element annotation data is attached to respective groups of pixels. Advantageously, inserting the annotation data into the video itself, and in particular attaching the annotation data directly to groups of pixels of the elements, enables a healthcare provider to more quickly / easily identify and understand the context of important features within the video as the relevant elements are flagged / labeled at their positions within the video image.

[0032] The method may further include using image processing to identify a plurality of sub-elements associated with a first element identified within a first surgical video frame based on surgical context data. Each sub-element may include a respective sub-group of pixels. The element annotation data may include respective sub-element annotation data associated with the plurality of sub-elements. The method may further include attaching each sub-element annotation data to a respective sub-group of pixels.

[0033] The annotation data may include one or more of element identification information, element grouping information, element sub-grouping information, element type information, element description information, element condition information, surgical step information, surgical task information, surgical event information, surgical instrument information, or surgical device information.

[0034] The method may further include refining surgical context data based on one or more elements identified within the first surgical video frame. The method may further include using image processing to identify one or more elements within a second surgical video frame based on the refined surgical context data. The method may further include generating annotation data for the second surgical video frame based on the refined surgical context data and one or more elements within the second surgical video frame. Advantageously, refining the surgical context data improves the identification of relevant elements as well as the content and relevance of the annotation data.

[0035] The method may further include using image processing to identify one or more elements within a second surgical video frame based on the acquired surgical context data. The method may further include determining tracking data regarding the second surgical video frame based on one or more elements within a first surgical video frame and one or more elements within the second video frame, the tracking data being associated with one or more of element behavior, element movement, or outcome information. The method may further include generating annotation data for the second surgical video frame based on the surgical context data and one or more elements within the second surgical video frame, the annotation data including tracking data regarding the second surgical video frame.

[0036] The method may further include obtaining predicted tracking information associated with the surgical context indicated by the surgical context data. The method may further include validating one or more identified elements within the second surgical video frame based on the predicted tracking information. The annotation data for the second surgical video frame may include an indication as to whether one or more identified elements within the second surgical video frame are validated. Advantageously, validation confirms or disproves the (predicted) identification of an element, e.g., its classification, type, behavior, or movement.

[0037] The predicted tracking information can be obtained based on a surgical plan.

[0038] The method may further include using surgical context data and one or more identified elements within a first surgical video frame to determine surgical task information associated with the first surgical video frame. The method may further include generating control parameters for a surgical procedure based on the determined surgical task information. The method may further include sending a control signal to a surgical control system, the control signal including an instruction indicating the determined control parameters to the surgical control system. Advantageously, the accuracy of determining correct surgical task information is improved by basing this determination on a combination of context data and identified elements within the image. This then improves the suitability of the control signal for controlling the next step in the surgical procedure, thereby improving the surgical outcome.

[0039] The method may further include using surgical context data and one or more identified elements within a first surgical video frame to determine surgical task information associated with the first surgical video frame. The method may further include using the surgical context data, the surgical task information, and one or more identified elements within the first surgical video frame to determine complication information associated with the determined surgical task. The method may further include generating control parameters for a surgical procedure based on the determined complication information. The method may further include sending a control signal to a surgical control system, the control signal including an instruction indicating the determined control parameters to the surgical control system. Advantageously, the accuracy of determining correct surgical task information is improved by basing this determination on a combination of context data and identified elements within the image. Using this to determine that a complication has occurred in a surgical task improves the confidence that a particular complication has occurred. Further, it is advantageous for a control signal for implementing a corrective measure to be automatically generated and sent. The improved accuracy and confidence in earlier steps improves the suitability of the control signal for controlling the most effective corrective measure, thereby improving the surgical outcome.

[0040] The method may further include determining surgical task information associated with a first surgical video frame using surgical context data and one or more identified elements within the first surgical video frame. The method may further include determining complication information associated with the determined surgical task using the surgical context data, the surgical task information, and the one or more identified elements within the first surgical video frame. The method may further include determining outcome information associated with the surgical task based on the surgical context data, the surgical task information, and the complication information. The method may further include determining to change a parameter associated with the surgical procedure based on the surgical task information, the complication information, and the outcome information. The method may further include generating control parameters for the surgical procedure based on the determination to change the parameter. The method may further include transmitting a control signal including an instruction indicative of the determined control parameters to a surgical control system. Advantageously, the accuracy of determining the correct surgical task information is improved by basing this determination on a combination of the context data and the identified elements within the image. Using this to determine that a complication has occurred in a surgical task improves the confidence that a particular complication has occurred. Further, it is advantageous that a control signal for implementing a corrective measure is automatically generated and transmitted. The improved accuracy and confidence in earlier steps improves the suitability of the control signal for controlling the most effective corrective measure, thereby improving the surgical outcome.

[0041] A computing system is described that includes a processor configured to execute any of the methods described above.

[0042] A computer program including instructions that, when the program is executed by a controller, cause the controller to execute any of the methods described above is described.

[0043] A computer-readable medium including instructions which, when executed by a controller, cause the controller to perform any of the foregoing methods is described.

[0044] Systems, methods, and means for aggregating preoperative data and the input and adaptation of an initial surgical plan template for a patient, surgery, surgeon, and / or facility are disclosed. The preoperative data can be used to input a patient-specific surgical plan. A surgical system (e.g., a surgical hub) can be configured to obtain patient-specific preoperative data for a patient-specific surgery. The surgical system may obtain a surgical template including surgical steps. The surgical system may determine surgical task options for the surgical steps, for example, based on patient-specific preoperative data. The surgical system may determine the characterization (e.g., predicted outcome, risk level, efficiency) of the surgical task options. For example, an input patient-specific surgical plan can be generated that may include the determined surgical steps and their respective characterizations.

[0045] The surgical system may select a set of surgical tasks from the surgical task options, for example, preselect the set, based on their respective characterizations. The selected surgical task options may include tasks with the best outcome success. The surgical system may generate a preselected input surgical plan including the selected set of surgical tasks. The preselected input surgical plan can be used as a recommendation for the surgery.

[0046] The surgical system may obtain information separate from the preoperative data, such as facility information. The facility information may include information associated with surgical tools, surgical equipment, availability of healthcare professionals (HCPs), availability of facility rooms, etc. The patient-specific surgical plan can be generated based on the facility information.

[0047] The surgical system may determine that surgical task options have been determined using incomplete and / or conflicting information. The surgical system may indicate (e.g., in the entered surgical plan) that surgical task options may be determined using incomplete information and / or conflicting information. The surgical system may prompt the user to enter information that may be used to clarify incomplete data and / or conflicting data. The surgical system may re-determine surgical task options and / or characterizations associated with the surgical task options by considering the entered information.

[0048] A computer-implemented method for generating a patient-specific surgical plan for a patient-specific surgery is described. The method includes obtaining patient-specific pre-surgery data, obtaining a surgical template including surgical steps, determining one or more surgical task options for a first surgical step and a second surgical step based at least in part on the patient-specific pre-surgery data, determining a respective characterization for each of the one or more surgical task options based on the patient-specific pre-surgery data, each respective characterization including at least one of a predicted result of each surgical task, a predicted result probability of each surgical task, a complication of each surgical task, a risk level of each surgical task, and an efficiency assessment of each surgical task, and generating an input patient-specific surgical plan including the one or more surgical task options for the first surgical step and the second surgical step and the respective characterizations associated with the one or more surgical task options. Advantageously, the computer-implemented method for generating a patient-specific surgical plan creates a plan tailored to the patient's specific needs rather than simply a general plan template, saving the time of medical professionals (compared to creating such a plan manually). Further, by including the respective characterizations associated with the surgical task options, medical professionals can review the advantages and disadvantages associated with each surgical task option. This enables medical professionals to make more informed choices for the surgery, particularly choices tailored to the specific needs of the patient. This leads to better decision-making and ultimately improved surgical outcomes. Optionally, the method further includes displaying the generated patient-specific surgical plan on a display.

[0049] The method may further include selecting a set of surgical tasks from surgical task options for a first surgical step based on respective characterizations associated with each surgical task option. The method may further include generating a preselected input surgical plan based on the selected set of surgical tasks for the first surgical step. Advantageously, providing a preselected surgical plan saves the medical professional even more time because when determining which to perform at each surgical step, the medical provider no longer has to review and compare as many surgical task options. Instead, the computer selects a set of surgical task options, and thus the medical professional has fewer options to review and compare (or, in some scenarios, the method selects only a single option and the medical professional does not have to review any options). At the same time, by making the selection based on respective characterizations, the most effective option can be selected. Thus, the most effective option is selected in less time and can thus be implemented more quickly, which further improves surgical outcomes. As an example, the result of tissue resection can correspond to the integrity of the seal line, and a seal with less leakage is equivalent to a more positive result than a seal with more leakage. As another example, a positive result of surgery can also correspond to a reduction in the risk of complications that occur during surgery, including, for example, instrument malfunction, leakage of the seal line, misfiring of the staple line, and the like.

[0050] The method may further include selecting a set of surgical tasks from surgical task options for a first surgical step based on respective characterizations associated with each surgical task option. The method may further include generating a preselected input surgical plan based on the selected set of surgical tasks for the first surgical step. The preselected input surgical plan may include one or more alternative surgical task options from one or more surgical task options for the first surgical step and respective characterizations associated with the one or more alternative surgical task options. Advantageously, providing a preselected surgical plan saves the medical professional even more time because when determining which to perform at each surgical step, the medical provider no longer has to review and compare as many surgical task options. Instead, the computer selects a set of surgical task options, and thus the medical professional has fewer options to review and compare (or, in some scenarios, the method selects only a single option and the medical provider does not have to review any options). At the same time, by making the selection based on respective characterizations, the most effective option can be selected. Thus, the most effective option is selected in less time and can thus be implemented more quickly, which further improves surgical outcomes. By including alternatives, the medical professional is prompted to immediately have or consider a range of appropriate alternative options in case the primary option is less suitable or not possible.

[0051] The method may further include obtaining facility information. Determining one or more surgical task options for a first surgical step may be further determined based on the facility information. Advantageously, determining surgical task options based on the facility information ensures that only surgical task options that are possible given the facility constraints are included in the plan. This reduces the amount of irrelevant and inaccurate information presented to the medical professional, so that the medical professional can make a clinical decision about the surgery more quickly, and that decision is actually possible. Thus, the surgical outcome is optimized taking into account the constraints of the facility.

[0052] The facility information may include availability information associated with one or more of a surgical device, a surgical instrument, a healthcare provider, or a facility room.

[0053] Facility information may include availability information associated with one or more of a surgical device or surgical instrument. The availability information may indicate whether the surgical device or surgical instrument has been selected for an independent surgical procedure. If the surgical device or surgical instrument has been selected for an independent surgical procedure, the availability information may further indicate a scheduling time window associated with the independent surgical procedure. The method may further include determining whether there is a conflict between a patient-specific surgical procedure and an independent surgical procedure. The determination may include determining whether the surgical device or surgical instrument can be used for both the patient-specific surgical procedure and the independent surgical procedure. Advantageously, the clinical determination of whether the surgical device / instrument can be used for both surgical procedures informs the healthcare professional of the conflict. Thus, the healthcare professional can be quickly apprised of the conflict and make a clinical determination as to which surgical procedure should be prioritized based on patient-specific data. Optionally, if there is a conflict, the method may further include determining which surgical procedure to reschedule based on an assessment of the urgency of each procedure. The urgency assessment may be determined based on respective patient-specific data. Advantageously, making a clinical determination as to which procedure to prioritize improves the overall surgical outcome across both patients.

[0054] One or more surgical task options may include multiple access points to the patient. The characterization associated with the first access point may be a first outcome success probability. The characterization associated with the second access point may be a second outcome success probability. Advantageously, identifying the probability of success of a surgery associated with using various access points (e.g., access port locations to the patient) enables the healthcare professional to select and use that which is associated with the best likelihood of success (e.g., as it enables improved access to the surgical site), thereby improving the surgical outcome.

[0055] One or more surgical task options may include multiple instrument positionings. The characterization associated with a first instrument positioning may be a first resultant success probability. The characterization associated with a second instrument positioning may be a second resultant success probability. Advantageously, identifying the probability of success of a surgery associated with using various instrument positionings enables a medical professional to select and use that which is associated with the best likelihood of success (e.g., reducing the likelihood of collision with another instrument), thereby improving the surgical outcome.

[0056] The method may further include determining that a subset of data from patient-specific surgical data is incomplete. If the characterization of a first surgical task option is determined based at least in part on the subset of data, the method may further include tagging the input patient-specific surgical plan with an indication that the characterization was determined using incomplete data. Advantageously, indicating that a characterization was determined using incomplete data means that a medical professional knows that the results associated with the plan are less certain and perhaps inaccurate. The medical professional is not given a false impression of the suitability of the plan. Thus, the medical professional can proceed with appropriate caution (correspondingly reducing the likelihood of making an error in the surgery) or can decide to obtain more data to improve the deficiency.

[0057] The indication can further instruct the user to input additional data corresponding to the incomplete data. The method may further include obtaining additional data corresponding to the incomplete data. The method may further include updating the first surgical task option based on the additional data. Advantageously, prompting a medical professional to obtain missing data and updating a surgical task option based thereon means that the final recommendation for the surgical task is more likely to be appropriate for the patient, e.g., associated with an improved surgical outcome or reduced risk.

[0058] The characterization of each surgical task option can be determined based on past surgical results. Advantageously, basing the characterization on past surgical results means that they are based on actual data that is directly relevant, rather than on theoretical assumptions that may be inaccurate or overly simplistic. This improves the reliability of the determined characterization and ultimately improves surgical outcomes.

[0059] Optionally, the past surgical results may be an aggregation of patient outcome data from multiple surgeries performed on other patients, executed by a cloud analysis system, and the outcome data record is the medical resources used in each surgery (optionally, how those medical resources were used during the surgery, i.e., the surgical tasks performed by specific surgical instruments), and an indication of whether the outcome was successful or failed. The cloud analysis system may determine a correlation between positive outcomes from the patient outcome data and the medical resources used in a particular type of surgery (and optionally, how those resources were used). (This step may be performed prior to the method of this embodiment.) Optionally, generating a patient-specific surgical plan based on the past surgical results and / or correlations may include determining to use a specific surgical instrument, operating parameters for operating the surgical instrument (e.g., tissue clamping force of a surgical stapler, which is a change in device settings in the surgical instrument), changes in orientation regarding how the surgical instrument is handled during the patient's surgery, changes when the surgical instrument is used during the patient's surgery, changes to the control algorithm of the surgical instrument, or exchanging the surgical instrument for a second surgical instrument during the patient's surgery. Optionally, this determination may be output to (and performed by) the surgical instrument.

[0060] Optionally, the result data further records an indication of historical and / or physiological characterization for each patient. The cloud analysis system determines a correlation between positive results from the patient result data and the medical resources used for a particular type of surgery and patient-specific historical and / or physiological characterization (and optionally how those resources were used), i.e., the correlation is determined from the result data grouped by the particular historical and / or physiological characterization, and may generate surgical recommendations based on the patient-specific historical and / or physiological characterization for which a patient-specific surgical plan has been created. Generating the input patient-specific surgical plan may include determining historical characterization or physiological differences and patient-specific surgical tasks associated with the patient-specific pre-surgical data. Based on this, the method may include determining at least one of a change to a device setting within the surgical instrument, a change in orientation regarding how the surgical instrument is handled during the patient's surgery, a change when the surgical instrument is used during the patient's surgery, a change to the control algorithm of the surgical instrument, or exchanging the surgical instrument for a second surgical instrument during the patient's surgery. Optionally, this determination may be output to (and implemented by) the surgical instrument.

[0061] The method may further include instructing one or more of a surgical instrument, a surgical system, and a surgical robot to autonomously perform one or more of the surgical tasks of the surgical plan.

[0062] The method may further include determining one or more operating parameters for one or more of the surgical instruments, surgical systems, and surgical robots used during the surgery, based on the input patient-specific surgical plan. The operating parameters may be, for example, the clamping force of a surgical stapler and the ultrasonic energy level applied by a cutter / sealing device. The method may include determining at least one of a change in device settings in the surgical instrument, a change in orientation regarding how the surgical instrument is handled during the patient's surgery, a change when the surgical instrument is used during the patient's surgery, a change in the control algorithm of the surgical instrument, or exchanging the surgical instrument for a second surgical instrument during the patient's surgery, based on the input patient-specific surgical plan. Optionally, this determination may be output to (and may be implemented by) the surgical instrument.

[0063] A surgical system is described that includes a processor configured to execute any of the foregoing methods.

[0064] A computer program including instructions that, when the program is executed by a controller, cause the controller to execute any of the foregoing methods is described.

[0065] A computer-readable medium including instructions that, when executed by a controller, cause the controller to execute any of the foregoing methods is described.

Brief Description of the Drawings

[0066]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

[0067] FIG. 1 is a block diagram of a computer-implemented surgical system 20000. An exemplary surgical system such as surgical system 20000 can include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. For example, each surgical system 20002 can include a computer-implemented bidirectional surgical system. For example, surgical system 20002 can include a surgical hub 20006 and / or a computing device 20016 that communicate with a cloud computing system 20008 as described, for example, in FIG. 2. The cloud computing system 20008 can include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Exemplary surgical systems 20002, 20003, or 20004 can include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human interface system 20012, etc. The human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 can include one or more HCP sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 can include, for example, one or more devices used to measure one or more environmental attributes as further described, for example, in FIG. 2. The robotic system 20013 can include, for example, a plurality of devices used to perform a surgical procedure as further described, for example, in FIG. 2.

[0068] The surgical system 20002 may communicate with a remote server 20009 that may be part of a cloud computing system 20008. In one example, the surgical system 20002 may communicate with the remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, the patient sensing system may communicate directly with the remote server 20009. The surgical system 20002 and / or its components may use one or more of the cellular protocols of GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), long term evolution (LTE) or 4G, LTE-Advanced (LTE-A), new radio (NR) or 5G to communicate with the remote server 20009 via a cellular transmission / reception point (TRP) or base station.

[0069] The surgical hub 20006 may have a cooperative interaction with one of more means for displaying images from the laparoscope scope and information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 may interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 may be configured to collect measurement data from one or more sensing systems 20011 and transmit notification or control messages to one or more sensing systems 20011. The surgical hub 20006 may transmit and / or receive information, including notification information, to and from a human interface system 20012. The human interface system 20012 may include one or more human interface devices (HIDs). The surgical hub 20006 may transmit and / or receive acoustic devices, notification or control information to a display, and / or control information to various devices that communicate with the surgical hub.

[0070] For example, as discussed in FIG. 1, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and one or more patient sensing systems) and an environmental sensing system 20015. One or more sensing systems 20001 may measure data regarding various biomarkers. One or more sensing systems 20001 may use one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc., to measure biomarkers. One or more sensors may use one of more of the sensing techniques such as photoplethysmography, electrocardiogram examination, electroencephalogram examination, colorimetric analysis, impedancemetry, potential difference measurement, current measurement, etc., to measure biomarkers as described herein.

[0071] Biomarkers measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood potential of hydrogen, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, airway bacteria, edema, mental aspect, sweat, circulating tumor cells, autonomic nervous tension, circadian rhythm, and / or menstrual cycle.

[0072] Biomarkers can be related to physiological systems, including but not limited to the behavioral and psychological, cardiovascular, renal, skin, nervous, gastrointestinal, respiratory, endocrine, immune, tumor, musculoskeletal, and / or reproductive systems. Information from biomarkers may be determined and / or used, for example, by a computer-implemented patient and surgical system 20000. Information from biomarkers may be determined and / or used by a computer-implemented patient and surgical system 20000 to, for example, improve the above system and / or improve patient outcomes. One or more sensing systems 20001, biomarkers 20005, and physiological systems are described in detail by U.S. Patent Application No. 17 / 156,287, filed on January 22, 2021, entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS" (Attorney Docket No. END9290USNP1), the disclosure of which is incorporated herein by reference in its entirety.

[0073] Figure 2 shows an example of a surgical system 20002 in an operating room. As illustrated in Figure 2, a patient is operated on by one or more healthcare professionals (HCPs). The HCPs are monitored by one or more HCP sensing systems 20020 worn by the HCPs. The HCPs, and the environment surrounding the HCPs, may also be monitored by one or more environmental sensing systems, including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors deployed in the operating room. The HCP sensing systems 20020 and the environmental sensing systems communicate with a surgical hub 20006 and may further communicate with one or more cloud servers 20009 of a cloud computing system 20008, as shown in Figure 1. The environmental sensing system may be used to measure one or more environmental attributes, such as the position of the HCPs in the operating room, the movement of the HCPs, the ambient noise in the operating room, the temperature / humidity in the operating room, etc.

[0074] As illustrated in FIG. 2, the main display 20023 and one or more audio output devices (e.g., speaker 20019) are placed within the sterile field so as to be visible to the operator on the operating table 20024. Additionally, the visualization / notification tower 20026 is placed outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human interface device (HID) 20027 and a second non-sterile HID 20029 facing opposite each other. The HID may be a display, or may be a display having a touch screen that enables a human to directly interface with the HID. The human interface system guided by the surgical hub 20006 may be configured to utilize the HIDs 20027, 20029, and 20023 to regulate the information flow to the operators inside and outside the sterile field. In one example, by the surgical hub 20006, the HID (e.g., the main HID 20023) may display notifications and / or information regarding the patient and / or surgical steps. In one example, the surgical hub 20006 may prompt and / or receive input from a person within the sterile field or non-sterile area. In one example, by the surgical hub 20006, the HID may display a snapshot of the surgical site recorded by the imaging device 20030 on the non-sterile HID 20027 or 20029 while maintaining a live video of the surgical site on the main HID 20023. The snapshot on the non-sterile display 20027 or 20029 may, for example, permit a non-sterile operator to perform diagnostic steps related to the surgery.

[0075] In one aspect, the surgical hub 20006 may be configured to send diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 to the main display 20023 within the sterile field so that it can be viewed by the sterile operator on the operating table. In one example, the input may be in the form of a modification to a snapshot displayed on the non-sterile display 20027 or 20029 that can be sent by the surgical hub 20006 to the main display 20023.

[0076] Referring to FIG. 2, the surgical instrument 20031 is used as part of a surgical system 20002 in a surgical operation. The hub 20006 can also be configured to regulate the information flow to the display of the surgical instrument 20031. For example, in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. Diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 20026 are sent by the hub 20006 to the surgical instrument display within the sterile field, where they can be viewed by the operator of the surgical instrument 20031. Exemplary surgical instruments suitable for use with the surgical system 20002 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0077] FIG. 2 illustrates an example of a surgical system 20002 used to perform surgery on a patient lying on an operating table 20024 within an operating room 20035. A robotic system 20034 can be used as part of the surgical system 20002 in a surgical operation. The robotic system 20034 can include a surgeon's console 20036, a patient-side cart 20032 (surgical robot), and a surgical robot hub 20033. While the surgeon views the surgical site through the surgeon's console 20036, the patient-side cart 20032 can manipulate at least one removably coupled surgical tool 20037 through a minimally invasive incision in the patient's body. An image of the surgical site can be acquired by a medical imaging device 20030 that can be manipulated by the patient-side cart 20032 to change the orientation of the imaging device 20030. The robotic hub 20033 can be used to process an image of the surgical site and then display it to the surgeon through the surgeon's console 20036.

[0078] Other types of robotic systems can be readily adapted for use with the surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019 / 0201137 (A1) (U.S. Patent Application No. 16 / 209,407), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed on Dec. 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety.

[0079] Various examples of cloud-based analysis methods implemented by a cloud computing system 20008 and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569 (A1) (U.S. Patent Application No. 16 / 209,403), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," filed on Dec. 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety.

[0080] In various embodiments, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide Semiconductor (CMOS) sensors.

[0081] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors may be capable of receiving light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0082] The one or more illumination sources may be configured to irradiate electromagnetic energy within the visible spectrum as well as the invisible spectrum. The visible spectrum is, in some cases, also referred to as the optical spectrum or emission spectrum and is a portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye) and may be referred to as visible light or simply light. Typically, the human eye responds to wavelengths of approximately 380 nm to approximately 750 nm in air.

[0083] The invisible spectrum (e.g., non-emission spectrum) is a portion of the electromagnetic spectrum that is located below and above the visible spectrum (i.e., wavelengths less than approximately 380 nm and greater than approximately 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than approximately 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than approximately 380 nm are shorter than the violet spectrum and become invisible ultraviolet rays, x-rays, and gamma-ray electromagnetic radiation.

[0084] In various aspects, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0085] The imaging device may employ multispectral monitoring to distinguish topography from the underlying structure. A multispectral image captures image data within a specific wavelength range from across the electromagnetic spectrum. The wavelength can be separated by a filter or by using an instrument having sensitivity to a specific wavelength, including frequencies beyond the visible light range, such as IR, and light from ultraviolet. Spectral imaging enables extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is detailed under the heading “Advanced Imaging Acquisition Module” in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385), filed Dec. 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after completion of a surgical task for performing one or more of the above-described tests on the treated tissue. It is understood that strict sterilization of the operating room and surgical instruments is required during any surgical procedure. The strict hygiene and sterilization conditions required in the “operating room,” i.e., the operating or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process is the need to sterilize anything that comes into contact with the patient or enters the sterile field, including the imaging device 20030 and its accessories and components. It will be understood that the sterile field can be considered a specific area considered to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding a patient prepared for surgery. The sterile field can include properly attired and scrubbed team members, as well as all equipment and fixtures within that area.

[0086] The wearable sensing system 20011 shown in FIG. 1 may include one or more sensing systems, such as the HCP sensing system 20020 as shown in FIG. 2. The HCP sensing system 20020 may include a sensing system for monitoring and detecting a set of physical and / or physiological states of a healthcare provider (HCP). The HCP may generally be one or more healthcare providers assisting a surgeon or other healthcare service provider. In one example, the sensing system 20020 may measure a set of biomarkers to monitor the heart rate of the HCP. In one example, the sensing system 20020 (e.g., a watch or a wristband) worn on the wrist of a surgeon may use an accelerometer to detect hand movement and / or shaking and determine the magnitude and frequency of tremors. The sensing system 20020 may transmit measurement data associated with the set of biomarkers and data associated with the physical state of the surgeon to the surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing device may include a camera 20021 for detecting the hand / body position of the HCP. The environmental sensing device may include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include devices such as a thermometer for measuring temperature and a hygrometer for measuring ambient humidity in the operating room. The surgical hub 20006 may, alone or in communication with a cloud computing system, use the surgeon biomarker measurement data and / or environmental sensing information to, for example, modify the control algorithm of a handheld instrument or the average latency of a robot interface to minimize tremors. In one example, the HCP sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and transmit the measurement data associated with the surgeon biomarkers to the surgical hub 20006.The HCP awareness system 20020 may use one or more of the RF protocols of Bluetooth (registered trademark), Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), and Wi-Fi to communicate with the surgical hub 20006. The surgeon biomarkers may include one or more of stress, heart rate, etc. The environmental measurements from the operating room may include ambient noise levels related to the movement of the surgeon or patient, surgeon and / or staff, and the attention level of the surgeon and / or staff.

[0087] The surgical hub 20006 may adaptively control one or more surgical instruments 20031 using the surgeon biomarker measurement data associated with the HCP. For example, the surgical hub 20006 may send a control program to the surgical instrument 20031 to control its actuator to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may send the control program based on situation recognition and / or circumstances regarding the importance or criticality of the task. The control program may instruct the instrument to change its operation to provide more control when control is needed.

[0088] Figure 3 shows an exemplary surgical system 20002 having a surgical hub 20006. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014 via a modular control unit. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain aspects, as illustrated in FIG. 3, the hub 20006 further includes an exhaust smoke module 20054 and / or a suction / irrigation module 20055. The various modules and systems can be connected to the modular control unit directly or via the communication module 20056 via a router. Operating room devices can be coupled to cloud computing resources and data storage via the modular control unit. The human interface system 20012 can include a display subsystem and a notification subsystem.

[0089] The modular control unit may be connected to a non-contact sensor module. The non-contact sensor module may use ultrasonic, laser type, and / or similar non-contact measurement devices to measure the dimensions of the operating room and generate a map of the operating room. Other distance sensors can be used to determine the boundaries of the operating room. In U.S. Provisional Patent Application No. 62 / 611,341, filed on December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," which is hereby incorporated by reference in its entirety, as described under the heading "Surgical Hub Spatial Awareness Within an Operating Room" in the same document, an ultrasonic-based non-contact sensor module can scan the operating room by transmitting ultrasonic bursts and receiving the echoes when the ultrasonic bursts are reflected by the outer wall of the operating room. The sensor module may be configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. A laser-based non-contact sensor module can scan the operating room, for example, by transmitting laser light pulses, receive the laser light pulses reflected by the outer wall of the operating room, compare the phase of the transmitted pulses with the received pulses to determine the size of the operating room, and adjust the Bluetooth pairing distance limit.

[0090] During surgery, applying energy to tissue for sealing and / or cutting is generally associated with smoke evacuation, aspiration of excess fluid, and / or perfusion of tissue. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgery. Valuable time can be lost in dealing with this problem during surgery. To untangle the lines, it may be necessary to remove the lines from their corresponding modules, which may require resetting the modules. The hub module type enclosure 20060 provides an integrated environment for managing power lines, data lines, and fluid lines, reducing the frequency of such line entanglements. Aspects of the present disclosure present a surgical hub 20006 for use in surgeries involving the application of energy to tissue at the surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combined generator module slidably receivable within the docking station of the hub enclosure 20060. The docking station includes data contacts and power contacts. The combined generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed within a single unit. In one aspect, the combined generator module also includes a smoke evacuation component, at least one energy supply cable for connecting the combined generator module to a surgical instrument, at least one smoke evacuation component configured to discharge smoke, fluid, and / or particulates generated by the application of therapeutic energy to the tissue, and a fluid line extending from a remote surgical site to the smoke evacuation component. In one aspect, the fluid line may be a first fluid line, and a second fluid line may extend from the remote surgical site to an aspiration and perfusion module 20055 slidably receivable within the hub enclosure 20060. In one aspect, the hub enclosure 20060 may include a fluid interface. Certain surgeries may require applying two or more energy types to the tissue.One type of energy may be more beneficial for cutting tissue, while a different type of energy may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut the sealed tissue. Aspects of the present disclosure present a solution where a hub module enclosure 20060 is configured to house different generators and facilitate bidirectional communication between them. One advantage of the hub module enclosure 20060 is that it allows for the quick removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is also slidably movable to disengage from the electrical engagement with the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue different from the first energy, and a second docking station having a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power contacts and data contacts, and the second energy generator module is also slidably movable to disengage from the electrical engagement with the second power contacts and the second data contacts. Additionally, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port configured to facilitate communication between the first energy generator module and the second energy generator module.Referring to FIG. 3, aspects of the present disclosure are presented regarding a generator module 20050, a smoke exhaust module 20054, and a hub module enclosure 20060 that enables modular integration of the aspiration / irrigation module 20055. The hub module enclosure 20060 further facilitates two-way communication between the module 20059, the module 20054, and the module 20055. The generator module 20050 may include integrated monopolar components, bipolar components, and ultrasonic components supported within a single housing unit slidably insertable into the hub's modular enclosure 20060. The generator module 20050 may be configured to connect to a monopolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the hub module enclosure 20060. The hub module enclosure 20060 can be configured to facilitate the insertion of multiple generators and two-way communication between the generators docked to the hub module enclosure 20060 such that the multiple generators function as a single generator.

[0091] FIG. 4 illustrates a surgical data network having a set of communication hubs configured to connect to a cloud, a set of sensing systems, an environmental sensing system, and a set of other modular devices disposed in one or more operating rooms, patient recovery rooms, or rooms within a medical facility specially equipped for surgery within a medical facility, according to at least one aspect of the present disclosure.

[0092] As illustrated in FIG. 4, the surgical hub system 20060 may include a modular communication hub 20065 configured to connect modular devices disposed within a medical facility to a cloud-based system (e.g., a cloud computing system 20064 that may include a remote server 20067 connected to a remote storage 20068). The modular communication hub 20065 and the devices may be connected in a room within a medical facility specially equipped for surgical procedures. In one aspect, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicate with a network router 20066. The modular communication hub 20065 may also be connected to a local computer system 20063 and provide local computer processing and data manipulation.

[0093] The computer system 20063 may include a processor and a network interface 20100. The processor may be coupled via a system bus to a communication module, storage, memory, non-volatile memory, and an input / output (I / O) interface. The system bus can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures, examples of which include a 9-bit bus, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus, but is not limited thereto.

[0094] The processor may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex by Texas Instruments. In one aspect, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM) and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, the details of which are available in the product datasheet.

[0095] In one example, the processor may include a safety controller with two controller-based families such as TMS570 and RM4x, also known by the trade name Hercules ARM Cortex R4 from Texas Instruments. The safety controller may be configured specifically for safety-critical applications of IEC61508 and ISO26262, among others, while providing scalable performance, connectivity, and memory options and providing a high level of integrated safety mechanisms.

[0096] It should be understood that computer system 20063 may include software that functions as an intermediary between the described user and basic computer resources in a suitable operating environment. Such software may include an operating system. An operating system that may be stored on disk storage may function to control and allocate the resources of the computer system. System applications may utilize the resource management by the operating system via program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0097] A user can input commands or information into the computer system 20063 via an input device connected to the I / O interface. Examples of input devices include, but are not limited to, pointing devices such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, gamepad, satellite broadcast receiving antenna, scanner, TV tuner card, digital camera, digital video camera, webcam, etc. These and other input devices are connected to the processor 20102 through the system bus via an interface port. Examples of interface ports include serial ports, parallel ports, game ports, and USB. Output devices use some of the same type of ports as input devices. Thus, for example, a USB port may be used to provide input to the computer system 20063 and output information from the computer system 20063 to an output device. Output adapters may be provided, among other output devices that may require special adapters, to illustrate that several output devices such as monitors, displays, speakers, and printers can exist. Examples of output adapters include video and sound cards that provide connection means between the output device and the system bus, but this is for illustration only and not limiting. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0098] The computer system 20063 can operate in a networked environment that uses logical connections to one or more remote computers, such as a cloud computer, or a local computer. The remote cloud computer can be, for example, a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, or other common network nodes, but typically includes many or all of the elements described with respect to the computer system. For the sake of brevity, only a memory storage device is illustrated along with the remote computer. The remote computer can be logically connected to the computer system via a network interface and subsequently physically connected via a communication connection. The network interface can include communication networks such as a local area network (LAN) and a wide area network (WAN). Examples of LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE802.3, Token Ring / IEEE802.5, etc. Examples of WAN technologies include circuit-switched networks such as point-to-point links, Integrated Services Digital Network (ISDN) and its variants, packet-switched networks, and Digital Subscriber Line (DSL), but are not limited thereto.

[0099] In various examples, computer system 20063 may include an image processor, an image processing engine, a media processor, or any special digital signal processor (DSP) used for processing digital images. The image processor can enhance speed and efficiency using parallel computing with single instruction, multiple data (SIMD), or multiple instruction, multiple data (MIMD) technologies. The digital image processing engine can perform various tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0100] The communication connection part may refer to the hardware / software used to connect a network interface to a bus. For the sake of exemplary clarity, the communication connection part is shown inside computer system 20063, but the communication connection part may be outside computer system 20063. For illustrative purposes only, the hardware / software required for connecting to a network interface can include modems such as ordinary telephone grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, as well as internal and external technologies such as Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0101] The surgical data network associated with the surgical hub system 20060 may be configured as passive, intelligent, or switching. A passive surgical data network functions as a conduit for data, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through the surgical data network under surveillance and includes additional features that configure each port within the network hub 20061 or network switch 20062. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0102] The modular devices 1a - 1n arranged in the operating room can be connected to the modular communication hub 20065. The network hub 20061 and / or the network switch 20062 can be connected to the network router 20066 to connect the devices 1a - 1n to the cloud computing system 20064 or the local computer system 20063. The data associated with the devices 1a - 1n may be transferred via the router to a cloud - based computer for remote data processing and operation. The data associated with the devices 1a - 1n can also be transferred to the local computer system 20063 for local data processing and operation. The modular devices 2a - 2m arranged in the same operating room may also be connected to the network switch 20062. The network switch 20062 can be connected to the network hub 20061 and / or to the network router 20066 to connect the devices 2a - 2m to the cloud 20064. The data associated with the devices 2a - 2m can be transferred via the network router 20066 to the cloud computing system 20064 for data processing and operation. The data associated with the devices 2a - 2m may also be transferred to the local computer system 20063 for local data processing and operation.

[0103] The wearable sensing system 20011 can include one or more sensing systems 20069. The sensing system 20069 can include an HCP sensing system and / or a patient sensing system. One or more sensing systems 20069 can communicate with the computer system 20063 of the surgical hub system 20060 or the cloud server 20067 directly via one of the network routers 20066 or via the network hub 20061 or the network switching 20062 that communicates with the network router 20066.

[0104] The sensing system 20069 can be connected to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 can be transferred to the cloud computing system 20064 via the network router 20066 for data processing and operation. Data associated with the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and operation.

[0105] As illustrated in FIG. 4, the surgical hub system 20060 can be extended by interconnecting a plurality of network hubs 20061 and / or a plurality of network switches 20062 with a plurality of network routers 20066. The modular communication hub 20065 can be housed within a modular control tower configured to receive a plurality of devices 1a - 1n / 2a - 2m. The local computer system 20063 may also be housed within the modular control tower. The modular communication hub 20065 can be connected to a display 20068 to display images obtained by some of the devices 1a - 1n / 2a - 2m, for example, during a surgical procedure. In various aspects, the devices 1a - 1n / 2a - 2m can include various modules such as an imaging module connected to an endoscope, a generator module connected to an energy-based surgical device, a smoke evacuation module, a suction / irrigation module, a communication module, a processor module, a storage array, a surgical device connected to a display, and / or a non-contact sensor module, among other modular devices that can be connected to the modular communication hub 20065 of a surgical data network.

[0106] In one aspect, the surgical hub system 20060 illustrated in FIG. 4 may include a combination of a network hub(s), network switch, and network router(s) that connect devices 1a-1n / 2a-2m, or sensing system 20069, to the cloud-based system 20064. One or more of devices 1a-1n / 2a-2m or sensing system 20069 connected to network hub 20061 or network switch 20062 may collect data in real time and transfer the data to a cloud computer for data processing and operation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term "cloud" may be used as a metaphor for the "Internet", but this term is not so limited. Thus, the term "cloud computing" can be used herein to refer to "one type of Internet-based computing", in which case various services such as servers, storage, and applications are distributed to a modular communication hub 20065 and / or computer system 20063 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 20065 and / or computer system 20063 via the Internet. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services may perform a number of calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to connect to a computer that communicates with cloud computing resources and storage.

[0107] By applying cloud computing data processing technology to the data collected by devices 1a to 1n / 2a to 2m, the surgical data network can provide improvements in surgical outcomes, cost reduction, and patient satisfaction. After tissue sealing and cutting procedures, at least some of devices 1a to 1n / 2a to 2m can be used to observe the state of the tissue to evaluate leakage or perfusion of the sealed tissue. At least some of devices 1a to 1n / 2a to 2m can be used to examine data including images of samples of body tissue for diagnostic purposes to identify pathologies such as the effects of diseases using cloud-based computing. This can include tissue localization, margin confirmation, and phenotype. At least some of devices 1a to 1n / 2a to 2m can be used to identify the anatomical structures of the body using various sensors integrated with the imaging device and techniques such as overlaying images captured by multiple imaging devices. The data collected by devices 1a to 1n / 2a to 2m, including image data, can be transferred to cloud computing system 20064 or local computer system 20063 or both for data processing and operations including image processing and manipulation. The data may be analyzed to improve the results of surgery by determining whether further treatments such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and precision robotics can be performed on tissue-specific sites and conditions. Such data analysis may further employ prognostic analysis processing, and using standardized methods can provide useful feedback either to confirm surgical treatment and surgeon behavior or to propose modifications to surgical treatment and surgeon behavior.

[0108] When cloud computer data processing technology is applied to the measurement data collected by the sensing system 20069, the surgical data network can result in improved surgical outcomes, improved recovery outcomes, reduced costs, and improved patient satisfaction. At least some of the sensing system 20069 may be used to evaluate the physiological state of a surgeon operating on a patient, a patient being prepared for surgery, or a patient recovering after surgery. The cloud-based computing system 20064 can monitor biomarkers associated with a surgeon or patient in real time, generate a surgical plan based at least on the measurement data collected before surgery, supply control signals to surgical instruments during surgery, and be used to notify a patient of complications during the postoperative period.

[0109] The operating room devices 1a to 1n can be connected to the modular communication hub 20065 via a wired channel or a wireless channel, depending on the configuration of the devices 1a to 1n with respect to the network hub 20061. In one aspect, the network hub 20061 may be implemented as a local network broadcast device that functions on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to the devices 1a to 1n located within the same operating room network. The network hub 20061 can collect data in the form of packets and transmit them to the router in half-duplex mode. The network hub 20061 cannot store any media access control / Internet Protocol (MAC / IP) for transferring device data. Only one of the devices 1a to 1n can transmit data at a time via the network hub 20061. The network hub 20061 cannot have a routing table or intelligence regarding the destination of information and broadcasts all network data across each connection and to the remote server 20067 of the cloud computing system 20064. The network hub 20061 can detect basic network errors such as collisions, but broadcasting all information to multiple ports can pose a security risk and cause bottlenecks.

[0110] The operating room devices 2a to 2m can be connected to the network switch 20062 via a wired channel or a wireless channel. The network switch 20062 functions within the data link layer of the OSI model. The network switch 20062 may be a multicast device for connecting the devices 2a to 2m arranged in the same operating room to the network. The network switch 20062 transmits data in the form of frames to the network router 20066 and can function in full-duplex mode. A plurality of devices 2a to 2m can transmit data simultaneously via the network switch 20062. The network switch 20062 stores and uses the MAC addresses of the devices 2a to 2m for transferring data.

[0111] The network hub 20061 and / or the network switch 20062 can be connected to the network router 20066 to connect to the cloud computing system 20064. The network router 20066 functions within the network layer of the OSI model. The network router 20066 creates a route for transmitting data packets received from the network hub 20061 and / or the network switch 20062 to cloud-based computer resources for further processing and operation of the data collected by any one or all of the devices 1a to 1n / 2a to 2m and the wearable sensing system 20011. The network router 20066 may be used to connect two or more different networks located at different positions, such as different operating rooms in the same medical facility or different networks in different operating rooms of different medical facilities. The network router 20066 transmits data in the form of packets to the cloud computing system 20064 and can function in full-duplex mode. A plurality of devices can transmit data simultaneously. The network router 20066 can use IP addresses for transferring data.

[0112] In one example, the network hub 20061 may be implemented as a USB hub that enables a plurality of USB devices to be connected to a host computer. The USB hub can expand a single USB port into several levels so that there are more available ports for connecting devices to the host system computer. The network hub 20061 may include a wired function or a wireless function for receiving information via a wired channel or a wireless channel. In one aspect, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a to 1n and devices 2a to 2m located in the operating room.

[0113] In an example, the operating room devices 1a - 1n / 2a - 2m and / or the sensing system 20069 may communicate with the modular communication hub 20065 via the Bluetooth wireless technology standard to exchange data over a short distance from fixed and mobile devices (using short - wavelength UHF radio waves in the 2.4 - 2.485 GHz ISM band) and to construct a personal area network (PAN). The operating room devices 1a - 1n / 2a - 2m and / or the sensing system 20069 may communicate with the modular communication hub 20065 via some wireless communication standards or wired communication standards or protocols, such as Bluetooth, Low - Energy Bluetooth, near - field communication (NFC), Wi - Fi (IEEE802.11 family), WiMAX (IEEE802.16 family), IEEE802.20, New Radio (NR), Long Term Evolution (LTE), as well as Ev - DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and Ethernet derivatives thereof, and not limited to these, any other wireless protocol and wired protocol specified for 3G, 4G, 5G, and beyond. The computing module may include a plurality of communication modules. For example, the first communication module may be dedicated to short - distance wireless communication such as Wi - Fi and Bluetooth, Low - Energy Bluetooth, Bluetooth Smart, etc., and the second communication module may be dedicated to long - distance wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev - DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, etc.

[0114] The modular communication hub 20065 functions as a central connection for one or more of the operating room devices 1a - 1n / 2a - 2m and / or the sensing system 20069 and can handle data types known as frames. The frames can carry data generated by the devices 1a - 1n / 2a - 2m and / or the sensing system 20069. When a frame is received by the modular communication hub 20065, the frame is amplified and sent to the network router 20066, which can transfer this data to the cloud computing system 20064 or the local computer system 20063 by using a number of wireless communication standards or wired communication standards or protocols as described herein.

[0115] The modular communication hub 20065 may be used as a stand - alone device or may be connected to compatible network hubs 20061 and network switches 20062 to form a larger network. Since the modular communication hub 20065 is generally easy to install, configure, and maintain, it can be a good option for networking the operating room devices 1a - 1n / 2a - 2m.

[0116] FIG. 5 illustrates a logic diagram of a control system 20220 for a surgical instrument or surgical tool according to one or more aspects of the present disclosure. The surgical instrument or surgical tool may be configurable. The surgical instrument may be at hand, such as an imaging device, a surgical stapler, an energy device, an end cutter device, and may include surgical supplies specific to the procedure. For example, the surgical instrument may include any of an electric stapler, an electric stapler generator, an energy device, a high energy device, a high energy Joe device, an end cutter clamp, an energy device generator, an in - operating - room imaging system, a smoke evacuation device, a suction irrigation device, a pneumoperitoneum system, etc. The system 20220 may comprise a control circuit. The control circuit may include a microcontroller 20221 comprising a processor 20222 and a memory 20223. For example, one or more of sensors 20225, 20226, 20227 provide real - time feedback to the processor 20222. A motor 20230 driven by a motor driver 20229 is operably coupled to a longitudinally movable displacement member to drive an I - beam knife element. A tracking system 20228 may be configured to determine the position of the longitudinally movable displacement member. The position information may be provided to a processor 20222 that may be programmed or configured to determine the position of the longitudinally movable drive member as well as the position of the firing member, the firing bar, and the I - beam knife element. Additional motors may be provided to a tool driver interface to control the firing of the I - beam, the movement of the closure tube, the rotation of the shaft, and the articulation movement. A display 20224 may display various operating states of the instrument and may include a touch - screen function for data input. Information displayed on the display 20224 may be overlaid with an image acquired via an endoscope imaging module.

[0117] The microcontroller 20221 may be any single-core or multi-core processor, such as those known by the product names of ARM Cortex from Texas Instruments. In one aspect, the main microcontroller 20221 may be, for example, a 256KB single-cycle flash memory or other non-volatile memory on-chip memory with a maximum of 40MHz, the details of which are available in the product datasheet, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle SRAM, an internal ROM with StellarisWare (registered trademark) software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels, which may be an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments.

[0118] The microcontroller 20221 may also include a safety controller with two controller-based families such as TMS570 and RM4x known by the product names of Hercules ARM Cortex R4 from Texas Instruments. The safety controller may be configured specifically for safety-critical applications of IEC61508 and ISO26262, among others, to provide a scalable performance, connectivity, and memory options while providing a high-level integrated safety mechanism.

[0119] The microcontroller 20221 may be programmed to perform various functions such as precise control over the speed and position of the knife and the articulation movement system. In one aspect, the microcontroller 20221 may include a processor 20222 and a memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the articulation movement part or the knife system. In one aspect, the motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers may be easily substituted for use in the tracking system 20228 with an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, published on October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT", which is hereby incorporated by reference in its entirety.

[0120] The microcontroller 20221 may be programmed to provide accurate control over the speed and position of the displacement member and the articulation movement system. The microcontroller 20221 may be configured to calculate a response within the software of the microcontroller 20221. The calculated response may be compared with the measured response of the actual system to obtain an "observed" response, which is used for actual feedback decision-making. The observed response may be a suitably adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0121] The motor 20230 may be controlled by a motor driver 20229 and can also be used by a surgical instrument or a tool firing system. In various forms, the motor 20230 may be a brushed DC drive motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 20230 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may include, for example, an H-bridge driver with field effect transistors (FETs). The motor 20230 may be powered by a power supply assembly releasably attached to a handle assembly or a tool housing to supply control power to a surgical instrument or tool. The power supply assembly may include a battery that may include a number of battery cells connected in series that can be used as a power source to power the surgical instrument or tool. In certain situations, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be lithium-ion batteries that can be connectable to and separable from the power supply assembly.

[0122] The motor driver 20229 may be the A3941 available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal-oxide semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 20229 may include an inherent charge pump regulator, which supplies a full (>10V) gate drive to a battery voltage up to 7V, enabling the A3941 to operate with a reduced gate drive up to 5.5V. A bootstrap capacitor may be used to supply the above battery supply voltage required for the N-channel MOSFET. The internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in high-speed or low-speed decay modes using diodes or synchronous rectification. In the low-speed decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FET can be protected from shoot-through by a resistor-adjustable dead time. The integrated diagnostics indicate low voltage, over-temperature, and abnormalities in the power bridge and can be configured to protect the power MOSFET under most short-circuit conditions. Other motor drivers may be easily substituted for use in the tracking system 20228 with an absolute positioning system.

[0123] The tracking system 20228 can comprise a controlled motor drive circuit configuration with a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for an absolute positioning system can supply a unique position signal corresponding to the position of a displacement member. In some examples, the displacement member can represent a longitudinally movable drive member with a rack of drive teeth for meshing engagement with a corresponding drive gear of a gear reduction assembly. In some examples, the displacement member can represent a firing member adapted and configured to include a rack of drive teeth. In some examples, the displacement member can represent a firing bar or an I-beam, each of which can be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member can generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, a firing member, a firing bar, an I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member can be coupled to a firing member, a firing bar, and an I-beam. Thus, the absolute positioning system can effectively track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member can be coupled to any position sensor 20225 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, a firing member, a firing bar, or an I-beam, or a combination thereof, can be coupled to any suitable linear displacement sensor. The linear displacement sensor can include a contact displacement sensor or a non-contact displacement sensor.The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system including a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system including a fixed magnet and a series of linearly arranged movable Hall effect sensors, an optical detection system including a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system including a fixed light source and a series of linearly arranged movable photodiodes or photodetectors, or any combination thereof.

[0124] The electric motor 20230 may include a rotatable shaft that operably interfaces with a gear assembly mounted to mesh with a set of drive teeth or a rack on the displacement member. The sensor element may be operably coupled to the gear assembly such that one revolution of the position sensor 20225 element corresponds to some linear longitudinal translation of the displacement member. The configuration of the gear ring and the sensor may be connected to a linear actuator by a rack and pinion configuration or to a rotary actuator by spur gears or other connections. A power supply may supply power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing with a corresponding drive gear of a gear reduction assembly. The displacement member may represent a longitudinally movable emitter member, emitter bar, I-beam, or a combination thereof.

[0125] One rotation of the sensor element associated with the position sensor 20225 can correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance that the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor device can be connected via a gear reduction device in which the position sensor 20225 completes one or more rotations with respect to the full stroke of the displacement member. The position sensor 20225 may complete multiple rotations with respect to the full stroke of the displacement member.

[0126] To provide a unique position signal for two or more rotations of the position sensor 20225, a series of switches (where n is an integer greater than 1) may be used alone or in combination with a gear reduction device. The state of the switch can be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1 + d2 +... dn of the displacement member. The output of the position sensor 20225 is supplied to the microcontroller 20221. The position sensor 20225 of the sensor device may comprise a magnetic sensor, an analog rotational sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0127] The position sensor 20225 may comprise any number of magnetic sensing elements such as a magnetic sensor classified, for example, by measuring the total magnetic field or a vector component of the magnetic field. The technologies used to produce both types of magnetic sensors can encompass many aspects of physics and electronics. Technologies used for sensing magnetic fields include, among others, search coils, fluxgates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetic impedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and microelectromechanical systems-based magnetic sensors.

[0128] The position sensor 20225 of the tracking system 20228 with an absolute positioning system may include a magnetic rotary absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT single-chip magnetic rotary position sensor available from Austria Microsystems, AG. The position sensor 20225 is connected to the microcontroller 20221 to implement an absolute positioning system. The position sensor 20225 is a low-voltage and low-power component and may include four Hall effect elements in the area of the position sensor 20225 that can be arranged above the magnet. Also, a high-resolution ADC and a smart power management controller may be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the border algorithm, may be provided to implement a concise and efficient algorithm for calculating hyperbolic and trigonometric functions that only require addition, subtraction, bit shifting, and table reference operations. The angular position, alarm bit, and magnetic field information may be transmitted to the microcontroller 20221 via a standard serial communication interface such as a serial peripheral interface (SPI) interface. The position sensor 20225 may provide a resolution of 12 bits or 14 bits. The position sensor 20225 may be an AS5055 chip provided in a small QFN16-pin 4×4×0.85 mm package.

[0129] Tracking system 20228 with an absolute positioning system may include and / or be programmed to implement a feedback controller such as a PID, state feedback, and adaptive controller. The power supply converts a signal from the feedback controller into a physical input to the system, in this case a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by position sensor 20225, other sensors (sensors) may be provided to measure physical parameters of the physical system. In some embodiments, other sensors (sensors) include U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM", which is hereby incorporated by reference in its entirety, and U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM", which is hereby incorporated by reference in its entirety, and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT", which is hereby incorporated by reference in its entirety. Sensor arrangements such as those described may be included. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include a comparison and combination circuit to combine the calculated response with the measured response using algorithms such as weighted averages and theoretical control loops that drive the calculated response towards the measured response. The calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, inductive resistance, etc. to predict how the state and output of the physical system will change by knowing the input.

[0130] An absolute positioning system can provide the absolute position of a displacement member upon power-up of the instrument without having to retract or advance the displacement member to a reset (zero or home) position, which may require a conventional rotary encoder that simply counts the number of forward or backward steps taken by the motor 20230 to estimate the position of a device actuator, drive bar, knife, etc.

[0131] For example, a sensor 20226, such as a strain gauge or a micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as the amplitude of the strain exerted on the anvil during a clamping operation, which can indicate, for example, the closing force applied to the anvil. The measured strain can be converted into a digital signal and provided to the processor 20222. Instead of or in addition to the sensor 20226, a sensor 20227, such as a load cell, may measure the closing force applied to the anvil by a closing drive system. For example, a sensor 20227, such as a load cell, may measure the firing force applied to the I-beam during a firing stroke of a surgical instrument or tool. The I-beam is configured to engage a wedge thread, which is configured to cam the staple driver upward to eject staples and deformably contact the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue when the I-beam is advanced distally by a firing bar. Alternatively, a current sensor 20231 may be used to measure the current consumed by the motor 20230. The force required to advance the firing member can correspond, for example, to the current drawn by the motor 20230. The measured force can be converted into a digital signal and provided to the processor 20222.

[0132] For example, a strain gauge sensor 20226 may be used to measure the force applied to tissue by an end effector. To measure the force exerted by the end effector on the tissue being treated, the strain gauge may be coupled to the end effector. A system for measuring the force applied to tissue grasped by the end effector may include a strain gauge sensor 20226, such as a micro strain gauge, configured to measure one or more parameters of the end effector. In one aspect, the strain gauge sensor 20226 can measure the amplitude or magnitude of the strain exerted on the jaw members of the end effector during a clamping operation, which can indicate tissue compression. The measured strain can be converted to a digital signal and supplied to a processor 20222 of a microcontroller 20221. A load sensor 20227 may measure, for example, the force used to operate a knife element to cut tissue captured between an anvil and a staple cartridge. A magnetic field sensor may be used to measure the thickness of the captured tissue. The measurements of the magnetic field sensor may also be converted to a digital signal and provided to the processor 20222.

[0133] The measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, respectively measured by sensors 20226, 20227, may be used by the microcontroller 20221 to characterize corresponding values of a selected position of the firing member and / or the velocity of the firing member. In one embodiment, the memory 20223 may store techniques, equations, and / or look-up tables that may be used by the microcontroller 20221 during evaluation.

[0134] The control system 20220 of the surgical instrument or tool may also include a wired or wireless communication circuit for communicating with a surgical hub 20065 as shown in FIG. 4.

[0135] FIG. 6 illustrates an exemplary surgical system 20280 according to the present disclosure, which may include a surgical instrument 20282 that can communicate with a console 20294 or a portable device 20296 through a local area network 20292 and / or a cloud network 20293 via a wired and / or wireless connection. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 such that the adapter 20285 transmits force from the drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein for measuring the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be a multi-firing loading unit (MFLU) that allows a clinician to fire a plurality of fasteners multiple times without removing the loading unit 20287 from the surgical site to reload the loading unit 20287.

[0136] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue therebetween, fire a fastener through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include an exchangeable multi-firing fastener cartridge that includes a plurality of fasteners (e.g., staples, clips, etc.) that can be fired two or more times before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener as the fastener is ejected from the multi-firing fastener cartridge.

[0137] The handle 20297 may include a motor coupled to the drive shaft so as to act on the rotation of the drive shaft. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanism or user interface, which may be engaged by a clinician to activate the motor.

[0138] The control interface of the handle 20297 may communicate with the controller 20298 of the handle 20297 to selectively activate the motor and act on the rotation of the drive shaft. The controller 20298 may be disposed within the handle 20297 and may be configured to receive inputs from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 may analyze the inputs from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 may also include a display visible to the clinician during use of the handle 20297. The display may be configured to display portions of the adapter or loading unit data before, during, or after firing of the instrument 20282.

[0139] The adapter 20285 may include an adapter identification device 20284 disposed therein, and the loading unit 20287 may include a loading unit identification device 20288 disposed therein. The adapter identification device 20284 may communicate with the controller 20298, and the loading unit identification device 20288 may communicate with the controller 20298. It will be understood that the loading unit identification device 20288 may communicate with the adapter identification device 20284 that relays or passes the communication from the loading unit identification device 20288 to the controller 20298.

[0140] Adapter 20285 may also include a plurality of sensors 20286 (one is shown) disposed therearound to detect various states of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of firings of the adapter 20285, the peak force of the adapter 20285 during firing, the total amount of force applied to the adapter 20285, the peak recoil force of the adapter 20285, the number of rest periods of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide an input to the adapter identification device 20284 in the form of a data signal. The data signals of the plurality of sensors 20286 may be stored within the adapter identification device 20284 or may be used to update the adapter data stored within the adapter identification device 20284. The data signals of the plurality of sensors 20286 may be analog or digital. The plurality of sensors 20286 may include a force gauge for measuring the force exerted on the loading unit 20287 during firing.

[0141] The handle 20297 and the adapter 20285 may be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting (e.g., inductively transmitting) energy and signals therebetween. It is also contemplated that the adapter identification device 20284 and the controller 20298 may wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0142] The handle 20297 may include a transceiver 20283 configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., the LAN 20292, the cloud 20293, the console 20294, or the portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 20280. For example, the controller 20298 may transmit instrument data including the serial number of a mounting adapter (e.g., adapter 20285) attached to the handle 20297, the serial number of a loading unit (e.g., loading unit 20287) attached to the adapter 20285, and the serial numbers of a plurality of firing fastener cartridges loaded into the loading unit to the console 20294. Thereafter, the console 20294 may reply to the controller 20298 with data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively. The controller 20298 may display a message on the local instrument display or transmit a message via the transceiver 20283 to the console 20294 or the portable device 20296 to display the message on the display 20295 or the portable device screen, respectively.

[0143] FIG. 7 illustrates a diagram of a situation awareness surgical system 5100 according to at least one aspect of the present disclosure. The data source 5126 can include, for example, a modular device 5102 (which can include sensors configured to detect parameters associated with a patient, an HCP, and the environment, and / or the modular device itself), a database 5122 (e.g., an EMR database including patient records), a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor), an HCP monitoring device 35510, and / or an environment monitoring device 35512. The surgical hub 5104 can be configured to derive context information regarding a surgical procedure from the data, for example, based on a particular combination of the received data or the particular order in which data is received from the data source 5126. The context information inferred from the received data can include, for example, the type of surgical procedure being performed, a particular step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity that is the subject of the procedure. This function, according to some aspects of the surgical hub 5104 for deriving or inferring information regarding a surgical procedure from the received data, can be referred to as “situation awareness.” For example, the surgical hub 5104 can incorporate a situation awareness system, which is the hardware and / or programming associated with the surgical hub 5104 that derives context information regarding a surgical procedure from received data and / or surgical planning information received from an edge computing system 35514 or an enterprise cloud server 35516.

[0144] The situation recognition system of the surgical hub 5104 can be configured to derive context information from data received from various different data sources 5126. For example, the situation recognition system can include a pattern recognition system, or a machine learning system (such as an artificial neural network) trained with training data to correlate various inputs (e.g., data from database 5122, patient monitoring device 5124, modular device 5102, HCP monitoring device 35510, and / or environmental monitoring device 35512) with corresponding context information regarding the surgical procedure. The machine learning system can be trained to accurately derive context information regarding the surgical procedure from the provided inputs. In the example, the situation recognition system can include a lookup table that stores pre-characterized context information regarding the surgical procedure in association with one or more inputs (or a range of inputs) corresponding to that context information. In response to a query with one or more inputs, the lookup table can return the corresponding context information of the situation recognition system to control the modular device 5102. In the example, the context information received by the situation recognition system of the surgical hub 5104 can be associated with a specific control adjustment, or a series of control adjustments, of one or more modular devices 5102. In the example, the situation recognition system can include a further machine learning system, lookup table, or other such system that generates or reads one or more control adjustments of one or more modular devices 5102 when the context information is provided as an input.

[0145] The surgical hub 5104 incorporating the situation awareness system can provide many advantages to the surgical system 5100. One advantage can include providing improved interpretation of sensed and collected data, which can improve the processing accuracy during the surgical procedure and / or the use of the data. Returning to the previous example, the situation awareness surgical hub 5104 can determine which type of tissue is being operated on, and thus, if an unexpectedly high force is detected to close the end effector of the surgical instrument, the situation awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the tissue type.

[0146] The type of tissue being operated on can affect the adjustments made to the compression speed and load threshold of a surgical stapling and cutting instrument for specific tissue gap measurements. The situation awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic procedure or an abdominal procedure, whereby the surgical hub 5104 can determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (in the case of a thoracic procedure) or stomach tissue (in the case of an abdominal procedure). The surgical hub 5104 can then appropriately adjust the compression speed and load threshold of the surgical stapling and cutting instrument according to the type of tissue.

[0147] The type of body cavity being operated on during a insufflation procedure can affect the function of the smoke evacuation device. The situation awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure utilizes insufflation) and can determine the type of procedure. Generally, since a certain type of procedure can be performed within a specific body cavity, the surgical hub 5104 can appropriately control the motor speed of the smoke evacuation device according to the body cavity being operated on. Thus, the situation awareness surgical hub 5104 can provide a consistent amount of smoke evacuation for both thoracic and abdominal procedures.

[0148] The type of procedure being performed can affect the energy level optimal for the operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, in arthroscopic procedures, the end effector of an ultrasonic surgical instrument or an RF electrosurgical instrument is immersed in fluid, which may require a higher energy level. The Situational Awareness Surgical Hub 5104 can determine whether the surgical procedure is an arthroscopic procedure. The Surgical Hub 5104 can then adjust the RF power level of the generator or the ultrasonic amplitude (e.g., "energy level") to compensate for the fluid-filled environment. In connection therewith, the type of tissue being operated on can affect the energy level optimal for the operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The Situational Awareness Surgical Hub 5104 can determine which type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument, respectively, according to the tissue profile expected for the surgical procedure. Further, the Situational Awareness Surgical Hub 5104 can be configured to adjust the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument not only for each procedure but also over the course of the surgical procedure. The Situational Awareness Surgical Hub 5104 can determine which step of the surgical procedure is being performed or will be performed next and then update the control algorithm of the generator and / or the ultrasonic surgical instrument or the RF electrosurgical instrument to set the energy level to an appropriate value for the type of tissue expected according to the step of the surgical procedure.

[0149] In an example, the surgical hub 5104 can derive data from additional data sources 5126 to improve conclusions derived from one data source 5126. The situation awareness surgical hub 5104 can enhance data received from the modular device 5102 with context information constructed from other data sources 5126 regarding the surgical procedure. For example, the situation awareness surgical hub 5104 can be configured to determine whether hemostasis has occurred (e.g., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. The surgical hub 5104 can be further configured to make a determination regarding the integrity of a staple line or tissue weld by comparing physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device communicatively coupled to the surgical hub 5104). The situation awareness system of the surgical hub 5104 can provide additional context when analyzing visualization data in consideration of physiological measurement data. The additional context can be useful when the visualization data may not be conclusive or may be incomplete by itself.

[0150] For example, if it is determined in a subsequent step of the procedure that the use of an instrument is required, the situation awareness surgical hub 5104 can actively activate the generator to which the RF electrosurgical instrument is connected. Actively activating the energy source can enable the instrument to be ready for use as soon as the preceding step of the procedure is completed.

[0151] The situation-aware surgical hub 5104 can determine whether the current or subsequent steps of a surgical procedure require different views or magnifications on the display according to the features of the surgical site that the surgeon is expected to view. The surgical hub 5104 can actively change the displayed view (e.g., supplied from a medical imaging device for a visualization system) as appropriate, such that the display automatically adjusts throughout the surgical procedure.

[0152] The situation-aware surgical hub 5104 can determine which step of a surgical procedure is being performed or will be performed next and whether specific data or a comparison between data is required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up a data screen based on the step of the surgical procedure being performed without waiting for the surgeon to request specific information.

[0153] During the setup of a surgical procedure or during the surgical procedure itself, errors can be checked. For example, the Situation Awareness Surgical Hub 5104 can determine whether the operating room is appropriately or optimally set up for the surgical procedure to be performed. The Surgical Hub 5104 can determine the type of surgical procedure being performed, read out the corresponding checklist, product positions, or setup requirements (e.g., from memory), and then be configured to compare the current operating room layout to the standard layout for the type of surgical procedure that the Surgical Hub 5104 has determined is being performed. In some examples, the Surgical Hub 5104 can compare a list of items for the procedure and / or a list of devices paired with the Surgical Hub 5104 to the recommended or expected manifest of items and / or devices for a given surgical procedure. If there is any discrepancy between the lists, the Surgical Hub 5104 can provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, HCP monitoring device 35510, environmental monitoring device 35512, and / or other surgical supplies are missing. In some examples, the Surgical Hub 5104 can determine, for example, the relative distance or relative position of the modular device 5102 and the patient monitoring device 5124 via a proximity sensor. The Surgical Hub 5104 can compare the relative position of the devices to the layout recommended or expected for a particular surgical procedure. If there is any discrepancy between the layouts, the Surgical Hub 5104 can be configured to provide an alert indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0154] The situation awareness surgical hub 5104 can determine whether a surgeon (or other HCP) is making a mistake or deviating from a series of actions expected during a surgical procedure. For example, the surgical hub 5104 can determine the type of surgical procedure being performed, read a corresponding list of steps or order of device use (e.g., from memory), and then compare the steps being taken or devices being used during the surgical procedure to the steps or devices expected for the type of surgical procedure that the surgical hub 5104 has determined is being performed. The surgical hub 5104 can provide an alert indicating that an unexpected action is being taken at a particular step in the surgical procedure or that an unexpected device is being utilized.

[0155] Surgical instruments (and other modular devices 5102) can be adjusted to suit the specific context of each surgical procedure (such as adjustment to different tissue types) and can verify actions during the surgical procedure. The next steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the procedure.

[0156] A vast amount of surgical data is generated during a surgical procedure. Surgical tasks performed during and / or after the surgical procedure are continuously completed using the generated surgical data. The surgical system may request specific task-related data for a surgical task before performing the task, or the surgical system may receive wholesale surgical data and analyze the complete data set to find the specific task-related data required for the task. This can cause delays or use unnecessary resources when performing surgical tasks. Analyzing wholesale surgical data to find task-specific data for a task is inefficient and raises bandwidth issues as the wholesale data is communicated to surgical systems that only need a portion of the data.

[0157] Systems, methods, and means are disclosed for automatically compiling, annotating, and distributing surgical data to a surgical system and / or device to anticipate related automated actions (e.g., prior to, before, in preparation for). A surgical computing system may be configured to obtain data associated with, for example, a patient being treated in an operating room, a healthcare professional (HCP) participating in a surgical procedure, a surgical device and / or instrument used in the surgical procedure, a surgical sensor, a monitoring system in the operating room, a surgical hub, and / or the like (e.g., surgical procedure data). The surgical procedure data may be compiled, for example, by the surgical computing system. The surgical computing system may be configured to annotate the surgical procedure data. The annotation may indicate a surgical context and / or a surgical step. The surgical computing system may identify, for example, a surgical system that may use the annotated data for a related surgical task. The surgical computing system may determine data needs (e.g., data to be used for a task) for the identified surgical system (e.g., a target surgical system). The computing system may generate, for example, selectively differentiated data (e.g., a data package, a data stream) for the target surgical system by reverse-compiling the annotated surgical procedure data. The selectively differentiated data may include at least a portion of the annotated surgical procedure data (e.g., the complete annotated surgical procedure data). The surgical computing system may transmit, for example, the selectively differentiated data to the target surgical system for use by the target surgical system in a subsequent task.

[0158] A surgical computing system can determine data needs for a target surgical system based on surgical data, for example, using situation awareness (such as described herein with respect to FIG. 7). The surgical computing system can determine, for example, the current surgical context and / or the current surgical step in a surgical procedure based on the surgical data. Based on the current surgical context and / or the current surgical step, the surgical computing system can determine a subsequent surgical context and / or a subsequent surgical step (e.g., the next surgical context and / or surgical step that occurs after the current surgical context and / or surgical step). The surgical computing system can determine tasks associated with the subsequent surgical context and / or the subsequent surgical step to be performed by the surgical system. The surgical computing system can determine data needs based on the determined tasks associated with the subsequent surgical context and / or the subsequent surgical step. For example, the surgical computing system can anticipate data to be used for tasks in the subsequent surgical context and / or the subsequent surgical step.

[0159] A surgical computing system can perform selective editing on surgical data, annotated surgical data, and / or data packages. The surgical computing system may perform editing, for example, on pre-identified data and / or conditional aspects of the data. For example, the surgical computing system may perform editing on the confidential aspects of surgical data (e.g., in accordance with the Health Insurance Portability and Accountability Act (HIPAA)). Editing can include removing the confidential aspects of surgical data, for example, by removing confidential data, replacing the data (e.g., with generic values and / or default values), scrambling to make the confidential aspects unreadable, encrypting, etc. The surgical computing system can edit abnormal data and / or unexpected data. Editing may be time and / or geopence controlled.

[0160] FIG. 8 shows an exemplary flow diagram of a surgical computing system that automatically performs selective distribution of annotated surgical data to a surgical system. As shown at 50020 in FIG. 8, the surgical computing system 50010 can acquire surgical data. The surgical data can be acquired, for example, from a surgical system 50030. At 50035, the surgical computing system 50010 can annotate the surgical data. As shown at 50040 in FIG. 8, the surgical computing system 50010 can identify / determine a target system (e.g., a target surgical system). At 50045, the surgical computing system 50010 can determine the data need(s) associated with the target system. The data need(s) can be associated, for example, with tasks to be performed by the target system (e.g., in subsequent / future / next surgical steps). The data need(s) can be associated with the data used (e.g., required) when performing the task. As shown at 50050, the surgical computing system 50010 can generate, for example, a data package (e.g., a data stream) for the target system. The data package can be generated based on the annotated surgical data and / or the data need(s) associated with the target system. As shown at 50055, the surgical computing system 50010 can be configured to perform editing. The editing can be performed on the acquired surgical data, the annotated surgical data, the data package, and / or the like. At 50060, the surgical computing system 50010 can send the data package to the target system. The target system can be, for example, a surgical hub, a surgical instrument, a surgical device, a system associated with a facility department, a billing system, and / or the like.

[0161] The surgical computing system 50010 can obtain surgical data from, for example, systems / devices within the surgical system 50030. For example, the surgical computing system 50010 can obtain surgical data from one or more of the monitoring system 50021, the surgical instrument 50022, the surgical procedure sensor 50023, the surgical device 50024, the surgical hub(s) 50025, and / or the like.

[0162] The surgical data may be data associated with a monitoring system, a surgical procedure sensor (e.g., a biomarker sensor, an HCP sensor, etc.), a surgical instrument, a surgical device, a surgical tool, the staff for a surgical procedure, the OR setup / layout, the surgical steps, the consumables used in the surgical procedure, the electronic medical record, the imaging scan and / or results, the surgical outcome, etc. For example, the surgical data may include raw data from the surgical system from which the data is obtained. The surgical data may be processed data (e.g., from the surgical hub 50025). For example, the surgical hub 50025 may process the obtained surgical procedure sensor data. The processed raw data may indicate surgical events, surgical steps, timing, surgical outcomes, device utilization, and / or the like. The surgical computing system 50010 can obtain the processed surgical data.

[0163] The surgical computing system 50010 can compile the acquired surgical data. In an example, the surgical computing system 50010 can obtain the compiled surgical data. For a surgical operation, the surgical hub 50025 can obtain data from a surgical system (e.g., an operating room (OR)) associated with the surgical hub. The surgical hub 50025 can compile the acquired surgical data and transmit the compiled surgical data to the surgical computing system 50010. The surgical computing system 50010 may use the acquired and compiled surgical data for annotation and / or distribution to a target system.

[0164] The surgical computing system 50010 may annotate surgical data (e.g., compiled surgical data). For example, the surgical computing system 50010 may annotate the surgical data based on situation recognition (as described herein with reference to, for example, FIG. 7). The surgical computing system 50010 can determine, for example, surgical context data that can indicate surgical events, times, surgical steps, etc. based on the surgical data. The annotation can be associated with the determined surgical context, surgical events, times, surgical steps, etc. The annotation can be used, for example, to provide context to the surgical data when the surgical data is decompiled.

[0165] As shown at 50040, the target system(s) can be identified. For example, the target system can include a surgical system (e.g., within an OR and / or within a facility), a facility department system, surgical equipment, a surgical hub, a billing system, and / or the like. The target system may be performing a task associated with the surgical operation (e.g., a surgical task). For example, the target system (e.g., a facility department system) can be configured to schedule the replacement, repair, and / or cleaning of the OR and consumables (e.g., materials used during the surgical operation).

[0166] The target system can execute tasks using surgical data. For example, surgical tasks in subsequent surgical steps can be autonomously executed based on surgical data associated with previous surgical tasks and / or surgical steps. For example, the surgical data associated with the first surgical step can be used by the target system to execute a surgical task in the second surgical step. For example, the surgical data can include the patient's surgical sensor data associated with patient biomarkers. The patient biomarker information can be used by the target system to execute a surgical task in subsequent surgical steps / phases. For example, the target system can change the parameters associated with the task based on the patient biomarker information.

[0167] In 50045, the surgical computing system 50010 can determine the data needs (s) associated with the identified target system. The data needs may be a data set used to execute a surgical task (e.g., by the target system). The surgical computing system 50010 can determine the data needs based on surgical tasks configured to be executed by the surgical system (e.g., configured to be executed at a later time or in subsequent surgical steps). For example, the data needs can be predicted by the surgical computing system 50010.

[0168] Surgical computing system 50010 can determine (e.g., predict) tasks (e.g., to be performed by a target system) based on situation awareness using, for example, surgical data and / or annotated surgical data. For example, surgical computing system 50010 can determine the current surgical context and / or surgical steps based on the surgical data. The current surgical context and / or surgical steps can be used to determine subsequent surgical context and / or surgical steps. Subsequent surgical context and / or surgical steps can be determined (e.g., predicted) using the surgical plan and the current surgical context and / or surgical steps.

[0169] For example, the surgical computing system may determine that the surgery is currently at a first surgical step (e.g., based on surgical data). The surgical computing system can identify a second surgical step as the surgical step that follows the first surgical step in the surgical plan. The surgical computing system can determine the surgical task associated with the second surgical step. Using the knowledge of the surgical task, the data needs associated with the surgical task can be determined. The surgical computing system can predict the data needs before the second surgical step is performed. The surgical computing system can determine the data needs associated with the target surgical system, for example, without the target system sending a request indicating the data needs.

[0170] Various examples of surgical steps and corresponding data needs are suitable for use with the present disclosure and are described in U.S. Patent Application No. 17 / 156,287, entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS," filed on January 22, 2021, the disclosure of which is incorporated herein by reference in its entirety. For example, a thoracic surgery such as lobectomy can be performed. The surgical computing system may determine that the lobectomy is in a surgical step associated with the management of large blood vessels. The surgical computing system may, for example, determine that the surgery is in a large blood vessel management step based on surgical data (e.g., determine that the current surgical task associated with the large blood vessel management step is being performed). For example, the surgical computing system may determine that the current surgical task is ligating the patient's pulmonary artery (e.g., based on surgical data). The surgical computing system may determine that the lobe removal surgical step is a subsequent surgical step (e.g., the next surgical step) in the lobectomy. The surgical computing system may determine that the lobe removal surgical step follows the large blood vessel management step. The surgical computing system may determine that the lobe removal step includes using a surgical stapler (e.g., a linear stapler) for a surgical task. For example, the surgical stapler may be used in a surgical task associated with the transection of the fissure. The surgical stapler may be adaptively controlled, for example, using dynamic parameters for operating the surgical stapler based on surgical data. Surgical data from previous surgical steps and / or tasks may be used to determine the parameters used to operate the surgical stapler. The surgical computing system may determine the data needs of the surgical stapler used in the fissure transection. The surgical computing system may selectively distinguish data that can be used for the fissure transection from the surgical data.A surgical computing system can, for example, anticipate a task (e.g., before a surgical step involving a surgical stapler) and selectively transmit differentiated data to the surgical stapler. The selectively differentiated data can be transmitted to the surgical stapler, for example, before a request for specific data is received.

[0171] A data package / stream (e.g., including selectively differentiated data) can be generated, for example, for a target system. The surgical computing system 50010 can generate a data package / stream for the target system using annotated surgical data based on determined data needs associated with the target system. The data package / stream can be generated, for example, using automatic selective differentiation (e.g., selecting portions of an annotated surgical data set for distribution to an appropriate system). For example, the data package / stream can include data relevant to the data needs and exclude data not used by the target system. Automatic selective differentiation can enable the transmission of discrete chunks of data (e.g., rather than wholesale communication of all compiled surgical data) for communicating with the target system. Automatic selective differentiation can minimize bandwidth issues and / or storage issues.

[0172] In an example, the surgical computing system can select a portion of the annotated surgical data to generate a data package / stream. The surgical computing system can reverse-compile relevant data from a complete set of annotated surgical data. The reverse-compiled data can be tagged (e.g., time-tagged) using situation awareness and / or surgical context. A portion of the annotated surgical data can be a subset of the complete set of annotated surgical data. For example, the data package / stream can include a subset of surgical steps, a subset of resources used in the surgery (e.g., consumables used, costs, recorded time, etc.), redacted information, and the like.

[0173] Surgical computing system 50010 can perform editing on surgical operation data. For example, surgical computing system 50010 can perform editing on surgical operation data (e.g., acquired surgical operation data), annotated surgical operation data (e.g., based on annotations), data packages / streams, etc.

[0174] In 50060, the data package / stream can be sent to a target system. The target system can include a surgical system, a facility system, and / or the like. For example, the target system can include a device maintenance scheduling system (e.g., for determining when to inspect / repair surgical tools / devices). The target system can include a surgical hub where a surgical operation is being performed. The surgical hub can acquire the data package / stream and use the data package / stream for subsequent surgical operation steps.

[0175] In an example, the computing system can perform an automatic selective discrimination of a data set (e.g., a data set automatically annotated), and send (e.g., distribute) a specific data package to different systems. The computing system can send a data package including data associated with the data needs of the target system. The computing system can decompile the acquired surgical operation data and annotate the decompiled data (e.g., using situation recognition, e.g., using surgical context and / or time tags). The computing system can send the decompiled data (e.g., discrete chunks of data) to the target system. The computing system can refrain from sending the compiled surgical operation data (e.g., the entire set of surgical operation data) by just sending the decompiled data. Sending the decompiled data can minimize bandwidth problems and / or storage problems.

[0176] In an example, the decompiled data (e.g., a selected packet of data) can be separated. The decompiled data packets can be separated based on characterization, risk level, prioritization, magnitude of change from what is expected (e.g., outlier data), hierarchical segmentation, system utilization, and / or the like.

[0177] In an example, surgical data can be used during automated selective discrimination (e.g., as described herein). For example, if a portion of the surgical data is different from what is expected (e.g., significantly different and / or different beyond a threshold), a portion of the different surgical data (e.g., outlier data, unexpected data) can be included in the data package. For example, a surgical computing system can obtain statistical data associated with a performed surgery. The statistical data can include expected data and / or average data. The statistical data can include an expected deviation (e.g., a threshold) associated with the expected data and / or average data. The surgical computing system can determine that the surgical data deviates from the expected data and / or average data, e.g., beyond a threshold. The surgical computing system can flag the surgical data, and the flag can indicate the deviation. In an example, a portion of the different (e.g., deviating from an expected value) surgical data can be tagged with an indication (e.g., indicating that it is different from what was expected). The computing system can include in the data package surgical data that constitutes a threshold amount of data (e.g., a non-conforming amount of data). The surgical data that constitutes the threshold amount of data can be tagged with an indication (e.g., indicating that the data constitutes the threshold amount of data).

[0178] Figure 9 shows an example of generating a data package and transmitting it to a target system. Surgical data (e.g., compiled surgical data) can be acquired. The surgical data may include data related to the surgical operation (e.g., all data). The target system may use a part of the surgical data, for example, to perform a surgical task. The target system may not require the entire compiled surgical data, or may not require it in some cases. Transmitting the compiled surgical data (e.g., the entire set of surgical data) may cause unnecessary bandwidth and / or storage problems.

[0179] As shown at 50075 in Figure 9, surgical data can be acquired. Annotations can be added to the acquired surgical data and / or annotations can be added to the acquired surgical data. The annotated surgical data may include data sets (e.g., multiple data sets) such as data set 1 50080a, data set 2 50080b, data set N 50080c, etc.

[0180] As shown at 50085, the target system(s) can be determined / identified. The target system(s) may include a system that receives surgical data to perform tasks such as surgical tasks, data storage, facility management, etc. Target systems such as target system A 50090a, target system B 50090b, and / or target system C 50090c can be determined / identified by a surgical computing system. For example, the target system(s) can be identified based on the current surgical context (e.g., the current surgical step). The target system(s) can be identified as one that performs tasks in the current surgical context and / or subsequent surgical contexts.

[0181] As shown in 50095, the data needs of the target system(s) can be determined. Different target systems can be associated with different data needs. For example, the first data need can be associated with target system A 50090a. As shown in 50100a, the first data need can be associated with data set 1 50080a and / or data set N 50080c. The second data need can be associated with target system B 50090b. As shown in 50100b, the second data need can be associated with data set 2 50080b. The third data need can be associated with target system C 50090c. As shown in 50100c, the third data need can be associated with data set 1 50080a, data set 2 50080b, and / or data set N 50080c. The data needs can be determined, for example, based on annotated surgical data (such as described herein).

[0182] As shown in 50105, data package(s) can be generated. The data package(s) can be generated for the target system(s). The data package(s) can include data associated with the data needs of the target system(s) (e.g., only data associated with the data needs). The target system(s) can receive a portion of the surgical data (e.g., a portion of the complete version of the annotated surgical data). For example, the data package can include edited information, a subset of the surgical steps, and / or a subset of the collected data (e.g., consumables used, cost, recorded time, etc.).

[0183] As shown in FIG. 9, the generated package A 50110a may include data associated with the data needs associated with the target system A 50090a. For example, the package A 50110a may include the data set 1 50080a and / or the data set N 50080c. The generated package B 50110b may include data associated with the data needs associated with the target system B 50090b. For example, the package B 50110b may include the data set 2 50080b. The generated package C 50110c may include data associated with the data needs associated with the target system C 50090c. For example, the package C 50110c may include the data set 1 50080a, the data set 2 50080b, and / or the data set N 50080c. The data package may be transmitted to the target system(s).

[0184] The surgical data can be edited. The editing may be performed on the compiled surgical data, the annotated surgical data, the data package, the individual data sets, etc. The editing may be performed multiple times. The selective editing may be automated. For example, the selective editing may be automated based on the pre-identified and / or conditional aspects of the data. For example, the surgical context, events, images, data sets, etc. may be used (e.g., as a filter) to select a part of the data for removal. A part of the data may be selected for removal and edited from the generated data package.

[0185] For example, the pre - defined portion of surgical data can follow a set of rules for inclusion and / or exclusion in a data package. The data may be edited based on meeting the exclusion criteria from a data set. A portion of the data within the data set may be edited (e.g., only a portion of the data set may be edited). The data may be edited, for example, if it is associated with confidential information (e.g., HIPAA information). The data may be edited, for example, if the data is abnormal and / or unexpected (e.g., different from the expected value by exceeding a threshold). The data may be flagged as abnormal and / or unexpected (e.g., not edited). The data may be edited based on, for example, time conditions and / or location conditions (e.g., geofence controlled). For example, the conditions for editing the data can be associated with whether a certain amount of time has elapsed, when the data exits the network / system, when the surgical procedure is completed, when the data is transferred to a protected archive, and / or one or more of the like. For example, the data may be edited after a predetermined amount of time has elapsed (e.g., only after it has elapsed).

[0186] The edited data may be flagged, for example, by tagging with an indication that the data has been edited. The edited data may be flagged to indicate that information has been removed (e.g., automatically removed). The edited data may be flagged to indicate the condition(s) associated with the edit.

[0187] For example, editing can be performed on data based on classification. For example, the classification can be determined for surgical data (e.g., a part of surgical data). For example, the surgical data can include private and / or confidential information (e.g., in accordance with HIPAA guidelines). A surgical computing system can determine a private classification of surgical data that includes private and / or confidential information. Editing can be performed on the surgical data that is determined to have a private classification. The surgical computing system can refrain from performing editing on surgical data that does not include private information and / or confidential information.

[0188] For example, editing may be performed on data based on a target system (e.g., to which the data is to be sent). Surgical data may be edited based on the surgical data and classifications associated with the target system. For example, the target system may be a surgical system within or outside of a patient privacy protection boundary. The target system may be a surgical system at a geographical location, network-based location, or organization-based location that can indicate whether the system is inside or outside of the patient privacy protection boundary. The geographical location may be outside of the HIPAA boundary. A surgical computing system may determine to edit surgical data classified as private information based on the geographical location of the target system (e.g., because it is outside of the HIPAA boundary). Confidential information (e.g., regarding HIPAA) may not be permitted to be transmitted outside of the HIPAA boundary. The target system may be a surgical system at a geographical location within the HIPAA boundary. A surgical computing system may determine to refrain from re-editing surgical data classified as private information. For example, data storage may be located within a cloud network (e.g., outside of the HIPAA boundary). A surgical computing system may edit confidential information before sending the data to the cloud network data storage. For example, a patient's electronic medical record storage may be located within a facility (e.g., within the HIPAA boundary). A surgical computing system can refrain from performing an edit before sending the data to the patient's electronic medical record storage.

[0189] In an example, a computing system can annotate (e.g., automatically annotate) surgical data (e.g., a video feed) during a surgical procedure. The annotation can indicate the surgical step(s) and / or time scale associated with the surgical procedure. The computing system can flag the annotation (e.g., and / or the video feed) in terms of different coding modalities. For example, a first coding modality can be associated with a computing system that maintains the integrated surgical data (e.g., the video feed) as a whole. A second coding modality can be associated with a computing system that edits the data after secure storage of the surgical procedure (e.g., primary secure storage) is completed (e.g., after the surgical procedure is finished). A third coding modality can be associated with a computing system that edits a portion of the data (e.g., the aspect of the data) before transmission of the data package to a target system (e.g., after the data package exits the primary secure storage).

[0190] In an example, secure storage can be composed of an amount of time. Secure storage can, for example, eliminate the stored data after the configured amount of time. Secure storage may perform automatic deletion of the data. The automatic deletion may be associated with a condition. The automatic deletion can be modified based on the condition (e.g., the amount of time before deletion can be extended / shortened, and / or the automatic deletion can be cancelled). The condition can be associated with an event that may occur after storage, such as, for example, a patient's complication, readmission, nosocomial infection, legal / claim issuance, and / or the like.

[0191] Figure 10 shows an example of selectively editing data within a dataset. As shown at 50120, surgical data (e.g., annotated surgical data) can be obtained. The surgical data can include a plurality of datasets (multiple possible). At 50125, a classification can be determined for the surgical data. The surgical data, the datasets (multiple possible) within the surgical data, a portion of the surgical data, a subset of the surgical data, etc. can be associated with the classification.

[0192] For example, a subset of surgical data can be classified as confidential information (e.g., with respect to HIPAA). For example, a subset of surgical data can be classified as inaccurate, erroneous, outliers, and / or the like.

[0193] At 50130, for example, a first edit can be performed on the surgical data based on the determined classification. Confidential information can be edited, for example, before being sent to a different system (e.g., a target device). A subset of surgical data can be edited (e.g., may be required to be edited) before sending the data, for example, based on the classification that the subset of surgical data is confidential information (e.g., with respect to HIPAA). A subset of surgical data can be edited, for example, based on the classification that the subset of surgical data is inaccurate. The computing system can determine, for example, to refrain from editing inaccurate surgical data in order to notify the HCP about the inaccurate data. For example, if the HCP recognizes inaccurate data, the HCP can change the surgical steps or use a different surgical device.

[0194] At 50135, data packages (multiple possible) can be generated for the associated target system(s) (multiple possible). The data package(s) can include the edited data.

[0195] At 50140, a second edit may be performed on at least a portion of the data within the data package. For example, the second edit may be performed based on a target system associated with the data package. For example, the target system may be outside of the HIPAA boundary and may be subject to confidentiality rules. Confidential information (e.g., regarding HIPAA) may be edited before sending the data to a target system outside of the HIPAA boundary. The target system may be within the HIPAA boundary, where the same confidential information may not need to be edited (e.g., may not need to be edited).

[0196] Surgical data determined to be edited may be stored, for example, before performing the edit. For example, surgical data determined to be confidential information (e.g., regarding HIPAA) may be edited before being sent outside of the HIPAA boundary. A surgical computing system may be able to store the surgical data to be edited in local storage, for example, to save the surgical data. The surgical data to be edited may be stored, for example, as a backup and / or if the surgical data to be edited is needed later.

[0197] An automated data package may be generated for a facility system. The automated data package may be sent to the facility system, for example, to schedule device / tool exchange, repair, and / or cleaning in an operating room, replenishment of consumables in the facility room, etc.

[0198] For example, a data package can be sent to a facility product reordering system. The data package can include data associated with products used in a surgical procedure. The data associated with products used during a surgical procedure can indicate consumable resources that were used in a previous surgical procedure and lost in the operating room for a planned surgical procedure. The data package may, for example, based on facility information, indicate that the product to be replenished is out of stock at the facility. The data package can indicate a request to the facility product reordering system and show when the planned surgical procedure can be performed soonest (e.g., due to a delay in replenishing depleted consumables), and / or can show alternative instruments / consumables that can be used to perform the planned surgical procedure (e.g., to perform the surgery sooner).

[0199] The data package can be sent to a facility system associated with cleaning and maintenance. For example, the data package can include data related to a planned surgical procedure, tools / devices / instruments used in the planned surgical procedure, tools / devices / instruments currently stored in the operating room, etc. The data package can include data to verify that surgical tools / devices / instruments are present and ready for use for a planned surgical procedure. The data package can include data indicating that the operating room is not ready for a planned surgical procedure and / or data indicating the time when the operating room can be made ready.

[0200] A data package can be sent to a facility system associated with the sterilization of (e.g., surgical instruments and / or tools in an operating room). For example, the data package can include data indicating the verification of instruments / devices / equipment to be used in a planned surgical procedure. The data package can include data indicating that the surgical instruments for a planned surgical procedure are sterilized and / or that preparations for use in the planned surgical procedure are ready. The data package can include data indicating that the surgical instruments have not been cleaned. The data package can include data indicating that the surgical instruments are in the process of being cleaned. The data package can include data indicating the prioritization of specific surgical devices / tools / equipment (e.g., currently not cleaned or unavailable) for sterilization.

[0201] A data package can be sent to a facility system associated with staff allocation and the timing of a surgical procedure. For example, the data package can include data indicating the planned surgical procedure timing. The data package can include data associated with the HCPs scheduled for the surgical procedure. The data package can include identification information associated with the staff. For example, a specific surgical task can be performed by a specific HCP role in a surgical procedure. The data package may indicate the staff suitable for the surgical task. The data package can indicate the staff available for the surgical procedure. The data package may indicate the staff based on classifications such as surgeon preference, OR setup, and / or the like.

[0202] The data package(s) can be sent to the target system(s), for example, for surgical documentation. The data package(s) can be sent for automated input for surgical documentation. The data package(s) may be sent to a system database (e.g., a linked system database within a facility). The data package(s) can be associated with the update, annotation, and / or transcription of information related to a surgical procedure.

[0203] One or more data packages may be sent to a target system(s), for example, to claim inputs, annotations, and / or classifications. The data package(s) may include information related to surgical steps performed during a surgical procedure. Information related to surgical steps performed during a surgical procedure may be linked to claim codes (e.g., diagnosis-related group claim codes). The data package(s) may include data used by a facility billing system to track claim information. For example, a computing system can determine the data to include in a data package for the billing system. The computing system can further annotate the data within the data package (e.g., tag with metadata) and classify the data, for example, with related claim codes. The annotated data can be used to update a data set to be used by the target system(s). The data package(s) may be sent to the target system(s), for example, for a billing task.

[0204] One or more data packages may be sent to a target system(s), for example, for inventory maintenance (e.g., consumable maintenance, surgical tool maintenance, surgical device maintenance, and / or the like). For example, the data package(s) may include information indicating products used (e.g., retracted) during a surgical procedure. The data package(s) may include information indicating serial numbers associated with the products used. The data package(s) may include information indicating the number of times surgical tools, surgical devices, and / or surgical equipment were used during a surgical procedure. The data package(s) may include information indicating that surgical tools, surgical devices, and / or surgical equipment are due for maintenance, replacement, repair, disposal, return, sterilization, cleaning, and / or the like. The data package(s) may include information indicating any problems and / or notes related to product use during a surgical procedure (e.g., if there were problems using the product and / or if the product malfunctioned).

[0205] Data package(s) can be sent to a target system(s), for example, for input into an electronic medical record (EMR) database. For example, data package(s) can include information related to a patient and a surgical procedure. Data package(s) can include information associated with details, annotations, records, surgical notes, etc. from a surgical procedure. Data package(s) may be used to update (e.g., automatically update) an EMR database regarding a patient record. Data package(s) can include information associated with a surgical procedure modality, surgical instruments used, surgical tasks performed, alternative treatments performed, and / or the like. Data package(s) can include surgical video(s) associated with a surgical procedure (e.g., annotated surgical video(s)). Surgical video(s) can be used as a record of a surgical procedure and of surgical steps and / or surgical tasks performed during the surgical procedure.

[0206] Distribution of information / data (e.g., automatic distribution of information / data) may be documented / annotated. For example, automation steps performed in relation to one or more of acquiring surgical data, annotating surgical data, determining a target system(s), determining data needs associated with the target system(s), generating a data package for the target system(s), editing a portion of the data, sending the data package to the target system(s), etc. may be documented (e.g., annotated). For example, a computing system can document and / or record autonomous operations associated with the distribution of surgical data. A surgical computing system can document and / or record user responses (e.g., override, verification, confirmation, and / or the like) to autonomous operations.

[0207] A computing system can document responses to and actions taken with respect to a data set (e.g., when performing automated distribution of data to a target system). Machine learning data can be generated, for example, based on the response of the computing system to a data set. The machine learning data can be used to train an artificial intelligence (AI) model. The AI model can be trained and / or used to perform subsequent automated tasks. The machine learning data can be associated with one or more of a failure of a surgical device / tool, a degradation in performance (e.g., associated with a surgical device / tool, a user technical action, and / or the like), data regarding a user biomarker, a surgical procedure step, a staff interaction, a user behavior (e.g., associated with an unexpected event), and / or the like.

[0208] A computing system may generate machine learning data associated with a failure of a surgical device during a surgical procedure. For example, the computing system may determine that the surgical device was unable to perform properly and / or generated inaccurate data. The computing system may determine, for example, a failure of the surgical device based on other surgical data acquired during the surgical procedure. The computing system may determine, for example, to adjust the magnitude of inaccurate data associated with the surgical device based on other surgical data acquired during the surgical procedure. The computing system may generate machine data associated with recalibration (e.g., adjusting the magnitude of inaccurate data). The computing system may determine the type of failure associated with the surgical device. The type of failure may be used, for example, to escalate the magnitude of associated data (e.g., peripheral data generated by an associated device) that may be attached to the failure data. The machine learning data may indicate a device associated with a surgical device failure (e.g., providing and / or using data associated with a surgical device failure). The machine learning data may indicate, for example, that an associated device needs to be enhanced (e.g., adjusted) and / or recorded in order to limit (e.g., minimize) the propagation of failure and / or improve the reliability between devices.

[0209] A computing system may generate machine learning data associated with performance degradation, such as machine learning data associated with user actions, surgical instruments, and / or the like. For example, machine learning data indicating that the performance was below a threshold may be generated. The performance may be degraded, for example, based on a user's technical actions and / or a surgical instrument. The machine learning data may be generated based on tracking user control interactions coupled to a surgical instrument. The machine learning data may include, for example, recorded data indicating repeated user control actions that may result in functional degradation. Applying the machine learning data to an AI model can prevent, for example, the execution of user control actions coupled to a surgical instrument that results in functional degradation. The AI model can be improved to prevent subsequent surgical procedures from using actions associated with performance and / or functional degradation.

[0210] A computing system may generate machine learning data associated with improved performance (e.g., unexpectedly improved performance). For example, the computing system may determine that a user technique associated with a surgical instrument is being performed above a threshold. The computing system can record / document surgical data (e.g., from a surgical hub) for example to investigate performance (e.g., to determine the cause of the improved performance). The recorded surgical data may include information associated with events that occurred in the OR during the surgery during which the improved performance occurred. The computing system may obtain diagnostic data for a system (e.g., an internal system) (e.g., may request device diagnostic data) to determine for example whether the system (e.g., an internal system) was affected. The computing system may enable the surgical hub to be requested to perform an internal diagnostic check on a relevant device (e.g., a device associated with the improved performance). The machine learning data associated with the improved performance may include an instruction (e.g., to a user) indicating to provide additional information (e.g., context information) regarding tasks / techniques that may have been performed differently (e.g., that may have caused the improved performance). The computing system may determine (e.g., verify) whether a system associated with the improved performance has actually experienced the improved performance. For example, the computing system may indicate (e.g., to a user) to verify that the result was unexpected. The computing system can verify that an algorithm associated with determining whether there is improved performance accurately evaluates the performance.

[0211] A computing system can generate machine learning data associated with metadata related to used biomarkers, surgical steps, staff interactions, user behavior, and / or other surgical events / interactions (e.g., related to unexpected events). For example, the machine learning data can include documentation of drugs and / or patient biomarkers that can be associated with unexpected events. The machine learning data can include staff information such as the presence of staff in the OR at the time of an unexpected event. The machine learning data can include continuous action variations with respect to surgical steps and / or surgical plans. For example, variations to a surgical plan can include deviations in actions from an initial surgical plan, such as changes in the approach to the surgical site (e.g., internal and / or trocar placement) that may have contributed to an unexpected event. Variations to a surgical plan can include unexpected complications (e.g., excessive adhesions encountered during mobilization). The machine learning data can include, for example, staff biomarkers that can indicate that a biomarker rises and / or falls irregularly from a steady state (e.g., a threshold state). The machine learning data can include a list of available surgical instruments that a user has selected not to use.

[0212] For example, a computing system can document (e.g., perform automatic documentation of) user responses, which can include creating data associated with the user's reaction to the actions performed by the computing system (e.g., autonomous actions such as annotating a dataset and / or generating a data package for a target system using the annotated dataset). For example, the computing system may determine that the user has performed an override action with respect to an autonomous action (e.g., an action associated with selective distribution of data / information to a target system(s)). The computing system may create machine learning data associated with the override action such that, for example, an AI model learns that the executed autonomous action is associated with the override action by the user. The machine learning data may include data associated with events and / or the risk of events leading to user overrides. The AI model can be used to perform subsequent autonomous actions. The computing system may use the AI model to avoid execution of autonomous actions associated with the override action.

[0213] A computing system can determine retention conditions for surgical data. For example, the computing system can determine to store data in storage for a longer / shorter period based on one or more of storage space availability, communication capabilities, system utilization, data level, facility regulations, retention procedures, etc. For example, data retention for data can be determined based on storage space or communication constraints (e.g., low storage space). For example, data can be retained for a shorter amount of time based on storage space and / or communication constraints. Less data can be stored based on storage space and / or communication constraints (e.g., compared to data storage when storage space is not limited and / or communication is not restricted). The computing system can instruct to scale down memory and / or retention, for example, based on successful results and / or lack of events. Memory overload can trigger alternative actions for data storage and / or archiving.

[0214] The hierarchy of data retention can be used. For example, the level of data can be adjusted based on the retention period. Metadata may be released and deleted for different retention periods. The hierarchy of data retention may be determined based on, for example, depth, scale, and / or relationship with the patient. For example, patient biomarker data can be associated with the longest retention. Patient biomarkers can be entered into the electronic medical record (e.g., for long-term retention). Annotated video data may have a longer retention period than secondary instrument data. The source and / or integrity of the data can affect data retention. For example, calculated and / or derived data may be associated with a shorter retention period than directly measured data. Video and / or timeline-based data can be associated with a longer retention period compared to data overlaid on the annotation and / or video. Product inquiry data may be associated with different levels of data retention. For example, product inquiries may have a different retention period than other instrument operation parameters. The product inquiry retention period of the data can be adjusted using user requirements. In an example, patient recovery events can be used to release data from storage.

[0215] A computing system can generate machine learning data associated with the results resulting from automated operations. For example, the machine learning data can include data related to the expected and actual results associated with the planned automated operations. The machine learning data may flag operations involving automation. The machine learning data can, for example, flag the successful performance of an automated operation against the expected result (e.g., the planned response). The machine learning data may flag automated operations that have been modified and / or overridden. The machine learning data can include, for example, characterizations associated with the automated steps to enable user monitoring and improved confidence that the automated operation was successful (e.g., the task was completed successfully).

[0216] For example, automated operations may be associated with pre-surgical CT scans, MRIs, and / or active lap imaging. The scans and / or imaging may be used in automated operations, for example, to identify and / or highlight the margins of a tumor (e.g., based on linking landmarks and link points together). The user can view real-time imaging from the scans and / or images. The user can determine the margins based on the real-time images. The user may decide that the margins (e.g., the margins generated by automated operations and / or the margins determined by the user) need to be adjusted based on the automated operations. Verification that the margin creation step has been accurately performed can be carried out. Identification of additional information determined in real time (e.g., causing an adjustment to the margin) may be performed.

[0217] Storage of surgical data can be automated. The target system (e.g., a computing system that can send data packages) may be a storage system. For example, the storage system may be automatically determined. The location and / or duration of surgical data storage may be determined (e.g., automatically determined). Recall parameters associated with the stored surgical data can be determined. The recall parameters can be associated with how the stored surgical data can be recalled, when the stored surgical data can be recalled, and / or how the surgical data can be recalled. For example, the stored surgical data is recalled based on monitored data (e.g., current surgical data).

[0218] Storage identification and / or pruning can be automated. For example, the identification and / or segmentation of data (e.g., data packages) for transport can be automated. Data packages can be generated (e.g., as described herein) and sent to target system(s) that can use (e.g., may require) the information within the data packages. A computing system can determine a retention period associated with a sent data package, an archival method for the data package, and / or a storage location for the data package. The computing system can determine, for example, for a target system to which a data package is sent, an editing and / or protected configuration of the data within the data package. A data package can include separated data. The data within a data package may be separated based on a surgical job / task, result, constraint, technique, user, surgical step, and / or the like. A data package may be organized based on the separation within the data package. For example, data for linked and / or similar surgical steps may be segmented (e.g., for review and / or export). For example, data associated with a particular surgical instrument (e.g., an ENSEAL device) may be pooled together for review together, which may enable a user, facility, and / or manufacturer to review the actions together. A storage location can be determined for incorrect and / or irregular surgical data. The computing system may use an alternative process, for example, for storage and / or review of data if the data is determined to be incorrect and / or irregular. Data that results in an incident and / or complication can be stored differently.

[0219] Surgery may use a patient-specific surgical plan when planning and performing surgery. Creating a patient-specific surgical plan for an operation may be performed manually by medical professionals and requires a lot of time and effort. Medical professionals may analyze a large amount of pre-surgical data and patient-specific data when planning a patient's surgery. The time spent devising a patient-specific surgical plan may include the time spent not performing other tasks. However, a surgeon cannot simply use a surgical plan template for each operation because each operation is tailored to the patient's needs. There are many variables to consider when determining a patient-specific surgical plan.

[0220] FIG. 11 shows an example of the aggregation of pre-surgical data and the generation of a patient-specific surgical plan. As shown in FIG. 11, a surgical computing system may obtain surgical data (e.g., pre-surgical data). The surgical data may be used to generate a patient-specific surgical plan. The surgical data may be associated with a pre-surgical data source 50200 (e.g., scans, images, electronic medical records, etc.), a surgical plan 50215 (e.g., a surgical plan template), facility information 50220 (e.g., staff allocation, room availability, etc.), and the like. The pre-surgical data source may include patient records 50205 and / or a pre-surgical sensor system 50210. The surgical data may be patient-specific surgical data. The patient-specific surgical data may include data associated with the patient and / or the patient's scheduled surgery. For example, the patient-specific surgical data may include pre-surgical examinations and / or images obtained for the planned surgery.

[0221] Surgical data can be automatically collected. For example, a surgical computing system can automatically obtain surgical data associated with a patient-specific surgery. The surgical computing system can send requests for a patient and / or a patient-specific surgery and can receive information associated with the patient and / or the patient-specific surgery. Scans, images, and / or tests performed (e.g., pre-operative scans, images, and / or tests) can be automatically collected for the surgery. Surgical data can be collected from a patient biomarker system, patient records, and / or other pre-surgical sensor systems.

[0222] As shown at 50225 in FIG. 11, surgical data can be processed. For example, surgical data can be aggregated and / or compiled. Surgical data may be synchronized, for example, during aggregation. Aggregation may include summarizing the surgical data. Aggregation may include aligning the surgical data, for example, aligning pre-operative images. For example, surgical data may include different pre-operative patient images. Aggregation may include aligning and / or overlaying pre-operative patient images to provide context for a patient-specific surgery. The aligned pre-operative patient images can be used to identify areas (e.g., tumors, surgical sites, organs, etc.) and / or blind spots within the image for the surgery.

[0223] Surgical data may be filtered. Filtering can be performed to determine inaccurate and / or missing surgical data. Filtering can be performed, for example, so that the surgical data can be interpreted. Pre-processing may be performed on the surgical data, for example, to adjust the data for analysis.

[0224] For example, surgical data can be processed for aggregation and / or compilation to a baseline surgical planning start point. Surgical data can be used to determine subsurface and / or volume information associated with a patient's anatomical structure. For example, medical imaging (e.g., X-ray, fluoroscopy, MRI, CT, and / or ultrasound scan) can provide information associated with a patient's anatomical structure. Surgical data can be used to plan and / or guide a surgical procedure (e.g., multiple images can be used and / or aggregated). Target surgical sites and / or auxiliary landmarks can be identified, for example, using surgical data. Surgical data can be used in the automation of image processing and / or pattern recognition algorithms to determine positions associated with features (e.g., unique features) that can be used as reference points in a surgical procedure. Image sources (e.g., multiple image sources) can enable an artificial intelligence system to process images (e.g., in real time) and / or make decisions based on landmarks when navigating to a target site and that site. Image sources can enable, for example, spatial recognition for a guidance system. Surgical data (e.g., preoperative images) can be aligned, for example, with a patient on an operating table, and thus, the system may identify a path (e.g., an optimal path) for a surgeon to autonomously follow and / or continue. A computing system can identify insertion points (e.g., alternative insertion points) and / or indicate insertion points to a surgeon, for example, for additional instruments and / or trocar placement. Medical images can be used as feedback (e.g., real-time feedback) for a surgical procedure. Medical images used as feedback can improve surgical accuracy, improve precision, reduce margins, avoid sensitive tissues and nerves, improve treatment consistency, and / or do the like. Reference markers (e.g., additional reference markers) may be implanted (e.g., the reference markers may create landmarks during imaging, for example, for use in processing of an image(s) and / or re-alignment when the patient is on an operating table).

[0225] Pre-operative patient data can be aggregated. The aggregated pre-operative patient data may include the patient's biometrics, medical records, diagnostic imaging, disease state and / or progression, previous treatments, and the like. The aggregated pre-operative patient data can be used to create a baseline surgical plan. The baseline surgical plan may include variables for the surgery, such as access, patient positioning, preferred instrument mix, and / or equivalents. The surgeon may simulate the created baseline surgical plan, adjust the plan, add to the plan, and / or modify the variables within the plan.

[0226] Relationships and / or interactions between data within the pre-operative patient data can be identified. For example, the data within the pre-operative patient data may conflict. The data within the pre-operative patient data can be associated with and / or amplified with each other. For example, an interaction biomarker may be highlighted (e.g., automatically highlighted) based on the determination that the biomarker is relevant.

[0227] Thresholds of combined and / or interrelated effects of a patient's pre-surgical data may be shown. For example, the patient's pre-surgical data may be interrelated above a threshold amount, which may affect how the pre-surgical data should be analyzed. For example, a patient may be taking a blood thinner (e.g., warfarin and / or heparin) to minimize blood clot complications. A blood test may show a low platelet count. The relationship (e.g., interaction) between the blood thinner and the low platelet count may be determined. The cumulative effect of bleeding may be greater than that accounted for by the dosage of the blood thinner. A higher probability of bleeding in surgery may be determined and shown (e.g., in a baseline surgical plan). A surgeon may decide to modify the baseline surgical plan, for example, based on an indicator of increased bleeding risk. In one example, a computing system can modify the baseline surgical plan, for example, based on a determination of increased bleeding risk. For example, alternative surgical steps (e.g., different energy devices, different surgical approaches, different mobilizations to free excised tissue, and / or use of secondary hemostatic aids) may be used to account for comorbidity interactions.

[0228] Biomarkers, treatments, and / or pre-operative steps (e.g., pre-operative targets) may be determined to be conflicting and / or above a threshold within a baseline surgical plan. Conflicting biomarkers, treatments, and / or pre-operative steps may be identified. The conflict may be highlighted, for example, in the baseline surgical plan. For example, heart rate and / or blood pressure biomarkers may be high in pre-operative evaluation or monitoring. The patient may be taking medications for heart rate and / or blood pressure. The conflict can be determined based on the fact that the biomarker should be in a lower range based on the use of the drug. The discrepancy (e.g., irregularity) may be highlighted and / or shown to the surgeon. The surgeon may use the discrepancy to determine a course of action (e.g., a treatment plan and / or a modification to the baseline surgical plan).

[0229] Images (e.g., preoperative images) and / or examinations (e.g., preoperative examinations) may be summarized, aggregated, and / or aligned (e.g., automatically aligned). Summarization, aggregation, and / or alignment may be included in a baseline surgical plan. For example, a proposed fit for a baseline surgical plan may be determined based on preoperative data. For example, a preoperative scan may be imported and / or overlaid on a baseline surgical plan. The overlay can indicate the tumor location. The tumor location may be aligned between scans, which may enable a user (e.g., a surgeon) to visualize the location and / or orientation of the tumor. Different angles from different images may provide a better situation regarding the tumor location. The overlaid image may be used to adjust the baseline margin to align with the tumor scan integration.

[0230] The overlaid image may be used to indicate blind spots and / or lack of visualization. Blind spots can cause problems during surgery. The indication of blind spots and / or lack of visualization may indicate that the data is suspect and / or absent. If the data is suspect and / or absent, alternative surgical tasks and / or methods may be determined. Other data may be substituted, for example, for suspect and / or absent data from different imaging sources (e.g., live imaging sources) and / or different preoperative scans.

[0231] FIG. 12 shows an example of a lung image generated from multiple sources. A computed tomography (CT) scan of the lung can be obtained. In the example, a portion of the image can be occluded from the CT view of the tumor. The tumor can be occluded, for example, in the CT scan. FIG. 13 shows an exemplary image supplied by a laparoscope camera or endobronchial ultrasound bronchoscopy (EBUS) to fill in the missing portion of the complete 3D view. As shown in FIG. 13, a laparoscope camera or EBUS can be used to fill in the visualization missing from other scans. The scans may be aggregated to obtain an overall image of the lung. Each scan may not be able to generate a complete image, but aggregating the images can provide a clearer picture to fill in the missing parts. Different cameras and / or camera angles may be used to provide a complete lung image. The aggregated image can be provided to the user to convey the necessary information. The aggregated image can indicate that information may need to be entered to complete the missing information and / or that the scan may need to be re-run. Information missing from the occluded view from the CT scan can be supplemented, for example, by aggregating different images from different sources to provide a complete image.

[0232] For example, as shown in 50230, the surgical steps for each patient can be determined. The surgery for each patient can include surgical steps for each patient. The surgical steps for each patient can be determined based on surgical data (e.g., processed surgical data). For example, the surgical steps for each patient can be determined based on a pre-surgical data source and a surgical plan template associated with the planned targeted surgery. For example, a thoracic surgery can be planned for the patient. The surgery for each patient can be planned using a thoracic surgery plan template. A pre-surgical data source (e.g., having a thoracic surgery plan template) can be used to determine the surgical steps for each patient.

[0233] Surgical steps specific to a patient may be associated with surgical tasks. A surgical procedure may be performed using one or more alternative surgical tasks. A surgical task may be performed using one or more alternative surgical instruments. For example, a surgical task may include using a surgical stapler. A surgical task may be completed by a surgical stapler using one or more energy levels, such as a first energy level or a second energy level. Using the first energy level with a surgical stapler may result in a first outcome, and using the second energy level with a surgical stapler may result in a second outcome. The first outcome may be improved compared to the second outcome. Surgical tasks and / or surgical steps associated with better outcomes (e.g., compared to alternative tasks and / or steps) may be preferred for a surgical procedure. Performing a surgical task in a surgical step may affect subsequent surgical tasks and / or subsequent surgical steps. For example, performing a surgical task may affect the remainder of the surgical tasks to be performed during the surgical procedure.

[0234] Surgical tasks specific to a patient may be associated with better outcomes compared to, for example, general surgical tasks in a surgical planning template. Accommodating a patient's needs and / or facility resources may enable a more efficient and / or successful outcome for a surgical procedure. Surgical tasks specific to a patient may be determined based on, for example, patient-specific information, facility information, staff allocation information (e.g., HCP availability, HCP role, HCP experience, HCP specialty, etc.).

[0235] Characterization may be determined for surgical steps specific to a patient. Surgical steps specific to a patient may be characterized based on, for example, outcome success, efficiency, risk, effectiveness, etc. For example, surgical steps specific to a patient may be associated with a range associated with outcome success. Surgical steps specific to a patient may be associated with different surgical outcomes. Characterization may be determined based on, for example, patient-specific surgical data and / or past data associated with the surgical procedure.

[0236] In an example, the result success of a surgical task option can be characterized. Based on patient-specific surgical data, the result success for a surgical task option can be determined. For example, a first surgical task option may involve using a surgical stapler with a first parameter set, and a second surgical task option may involve using a surgical stapler with a second parameter set. The first set of parameters and the second set of parameters can result in different outcomes based on the patient's anatomical structure and / or patient-specific surgical data. For example, if a patient is prone to bleeding, a set of parameters that can increase bleeding may be associated with a lower result success. The result success can be characterized, for example, to assist in selecting which surgical task option to use in a surgical procedure.

[0237] In an example, the risk of a surgical task option can be characterized. Based on patient-specific surgical data, a risk level (e.g., high risk, moderate risk, low risk, no risk, etc.) can be determined for a surgical task option. For example, a first surgical task option using a first surgical device may pose more risk to a patient compared to a second surgical task option using a second surgical device. The first surgical task option can result in more risk, for example, because the first surgical device can interact negatively with the patient based on the patient's anatomical structure and / or patient-specific surgical data. The first surgical task option may pose a high risk to the patient, and the second surgical task option may pose a moderate risk. The characterized risk level can be used, for example, to select the surgical task option to use in a surgical procedure.

[0238] In an example, the efficiency of surgical task options can be characterized. Based on patient-specific surgical data, the efficiency associated with performing a surgical task can be determined. A first surgical task option can be completed more efficiently than a second surgical task option, for example, based on the surgical device used. For example, a surgical device with stronger energy generation can expedite a surgical procedure compared to a surgical device with weaker energy generation. The efficiency of a surgical task option may be affected by the patient's anatomical structure and / or patient-specific surgical data. For example, a patient's anatomical structure may be more useful for a particular surgical device than another, which can increase the efficiency of the surgical task option.

[0239] Results can be determined (e.g., predicted) for surgical tasks and / or surgical procedure steps, as shown, for example, in 50235. The results may be associated with surgical outcomes, complications, efficiency, and / or the like. The results may be determined based on, for example, patient risk, patient survivability, surgical time, and / or the like (which may be determined based on the data obtained). For example, a first result may be determined (e.g., predicted) for a first (e.g., primary) surgical task, and a second result may be determined (e.g., predicted) for a second (e.g., alternative) surgical task. The first result may be associated with a higher success rate compared to the second result. The first result may have a higher success rate, for example, based on patient-specific data. For example, if a first surgical task uses a first surgical tool and a second surgical task uses a second surgical tool, the first surgical tool may be more appropriate considering the surgical procedure and / or the patient. Thus, using the first surgical tool can result in a better chance of success.

[0240] Results can be determined for surgical tasks even if, for example, there is incomplete and / or conflicting data (e.g., data used to determine the results). For example, the data may be conflicting and / or incomplete, which may lead to inaccurate result predictions. The result prediction may, for example, take into account conflicting and / or incomplete data when determining the results. For example, an instruction may be sent indicating that the determined result is based on incomplete information and / or conflicting information. The instruction may indicate the result prediction certainty associated with a typical surgery (e.g., without missing information). The instruction may indicate that the determined result can be refined, for example, by verifying and / or providing correct information (e.g., used in result determination). For example, a result can be determined for a first surgical task. The result can be determined based on incomplete information. The result can indicate a result prediction certainty range (e.g., 70 - 90%). The result can indicate to the HCP to provide information to refine the result prediction certainty range. The HCP can provide the missing information. Using the provided missing information, the result prediction certainty range may be refined (e.g., to be 85% - 90%).

[0241] Results can be determined based on previous surgical data (e.g., from previous surgeries) and / or previous surgical results. As shown in 50240, past surgical data and / or results may be obtained (e.g., from cloud storage). Past surgical data and / or results can be used to determine results associated with a current surgery and / or surgical task. The predicted results can be stored in the past surgical data and / or results. The actual results of a surgical task / step / surgery can be stored in the past surgical data and / or results, for example, after the surgical task / step / surgery is collected. The past data can then be used to calculate result predictions in subsequent surgeries.

[0242] Past surgical data and / or results can be used, for example, to provide context to patient-specific surgical steps / tasks. For example, a cloud aggregation of results from past results can be used to provide context associated with patient-specific surgical steps / tasks (e.g., highlighting implications for patient-specific surgical tasks / steps and / or baseline plans). Past surgical data and / or results can be used to adapt patient-specific surgical steps / tasks (e.g., baseline surgical plans) based on, for example, implementation (e.g., best practice), clinical trends, and / or changes in the like. Past surgical data and / or results may be overlaid on patient-specific surgical steps / tasks and / or modified by patient data to identify, for example, surgical decision points (e.g., for a user to consider when planning a surgery).

[0243] For example, as shown in 50245, a patient-specific surgical plan may be input. The patient-specific surgical plan may include surgical steps and / or patient-specific surgical tasks. The surgical tasks may be surgical tasks determined based on patient-specific preoperative data (e.g., resulting from predicted outcomes). The patient-specific surgical plan may include a recommended set of surgical steps / tasks (e.g., for each surgical step / task). The patient-specific surgical plan may include alternative surgical steps for the recommended surgical tasks. The patient-specific surgical plan can provide alternative options for an HCP to select and / or provide feedback. Options that are not relevant to the patient and / or options that can lead to unsuccessful outcomes can be determined and excluded from the patient-specific surgical plan.

[0244] The recommended surgical tasks / steps can be determined for a patient-specific surgical plan. The recommended surgical tasks / steps can be determined, for example, based on determined outcomes (e.g., predicted outcomes) associated with the patient-specific surgical steps / tasks. The recommended surgical tasks / steps may be those associated with a higher predicted outcome success. For example, the input patient-specific surgical plan may include the surgical tasks associated with the highest predicted success outcome. The recommended surgical tasks / steps may be those associated with facility information (e.g., staff allocation information, tool availability, OR availability, etc.). The recommended surgical tasks / steps may be those associated with surgeon preferences.

[0245] A user (e.g., an HCP) can provide an input to a patient-specific surgical plan, for example, as shown at 50250. For example, the HCP can select an alternative option for a surgical task (e.g., instead of a recommended surgical task). The tasks following the selected alternative option may be affected by the selection if, for example, selecting an alternative task changes how the surgery (e.g., the remainder of the surgery) can be performed. For example, if the recommended surgical task uses a first surgical tool and the HCP selects an alternative surgical task that uses a second surgical tool, the subsequent surgical tasks that use the first surgical tool can be changed (e.g., to account for the use of the second tool). The subsequent surgical tasks do not necessarily have to conform to the selection of the alternative surgical task option. As shown at 50255, the patient-specific surgical plan can be adjusted. The patient-specific surgical plan may be adjusted, for example, based on user input and / or in response to previously modified surgical tasks / steps.

[0246] The HCP input can function as an override for the automated input of a patient-specific surgical plan. The HCP can still control how the planned surgical procedure can be performed. For example, the patient-specific surgical plan may be used as a baseline (e.g., so that the HCP can view potential strategies and / or options for the surgical procedure). The automated input of the patient-specific surgical plan can reduce the HCP's workload. The HCP may focus on selecting preferred surgical tasks / steps rather than analyzing pre-operative data and / or facility information to determine the surgical plan.

[0247] A patient-specific surgical plan and / or an adjusted patient-specific surgical plan can be generated, for example, for a surgical procedure. As shown in 50260, the generated patient-specific surgical plan can be displayed. The patient-specific surgical plan can be displayed, for example, as a viewable report (e.g., a single viewable report). The patient-specific surgical plan can be sent to a surgical control system. The surgical control system may, for example, instruct a surgical instrument and / or a surgical system to autonomously perform surgical tasks (e.g., based on the patient-specific surgical plan). For example, the surgical control system can determine parameters of a surgical instrument to be used during a surgical procedure based on the patient-specific surgical plan.

[0248] FIG. 14 shows an example of a patient-specific surgical plan report. The patient-specific surgical plan report 50270 can include surgical steps for a patient-specific surgical procedure. The surgical steps can include, for example, step 1 50275 to step N 50310. The surgical steps can include surgical tasks. The surgical tasks can include recommended surgical tasks and / or one or more alternative surgical tasks.

[0249] For example, step 1 50275 may include a plurality of surgical task options. Step 1 50275 may include option A 50280 and / or option B 50290. Option A 50280 may be associated with a first result success (e.g., predicted result success) that may be in the range of 70 - 75%. Option B 50290 may be associated with a second result success that may be in the range of 90 - 95%. Option A 50280 may use access point A 50282a and / or surgical device A 50282b. Option B 50290 may use access point B 50292a and / or surgical device B 50292b. Option A 50280 may be flagged, for example, if the surgical task is associated with incomplete and / or conflicting information / data. For example, option A may be determined using incomplete and / or conflicting data / information. As shown at 50284, flagging option A 50280 can indicate to the HCP to review the data / information. For example, the HCP can provide missing data / information and / or verify conflicting data / information. The result success associated with option A can be modified based on HCP input.

[0250] The patient-specific surgical plan report can be interacted with (e.g., by the HCP). For example, the HCP can select one or more surgical tasks for a surgical step. The selection can affect subsequent surgical tasks / steps in the surgery. For example, the selection of a surgical task in step 1 50275 can affect the surgical task options in subsequent steps such as step N 50310.

[0251] A patient-specific surgical plan may include a recommended starting point, surgical tasks, and / or alternative surgical tasks for a surgical procedure. A patient-specific surgical plan may include potential complications (e.g., aggregation of complications), identification of ancillary information, and / or the like. A patient-specific surgical plan may include identification of relationships (e.g., interactions) between surgical steps and aggregated patient data. A patient-specific surgical plan may include, for example, initial access port location identification (e.g., for improving access to the surgical site), which may be determined based on, for example, surgical steps, instrument selection, and / or patient data.

[0252] For example, a patient-specific surgical plan may include aggregation of potential complications and / or identification of ancillary information. For example, a patient-specific surgical plan may include highlighting and / or notation indicating identified comorbidities that may interact during, for example, a surgical procedure. Complications may be determined, for example, by identifying that comorbidities may amplify each other's effects and / or may affect the treatment that may be selected. For example, interacting disease states that may increase the probability of complications may be calculated and / or shown.

[0253] A patient-specific surgical plan may include instructions associated with threshold biomarkers. Indicators related to threshold biomarkers may be related to the identified disease state.

[0254] A patient-specific surgical plan may include notifications related to surgical steps that may be affected and / or may be changed. Surgical steps may be determined to be affected and / or may be shown to be changed based on, for example, one or more of the determined disease, the state of disease progression, HCP, facility, and / or staff situation awareness, and / or the like.

[0255] The surgical plan for each patient may include instructions (e.g., highlighting) associated with the identification of metallic objects. For example, metallic objects (e.g., clips, staples, battless, etc.) can be detected. The surgical plan for each patient may include an instruction (e.g., on an image and / or scan) indicating the location (e.g., relative location) of the metallic object. The instruction may indicate an area where a surgical tool / device (e.g., an ultrasonic device and / or a radio frequency bipolar instrument) may interact poorly (e.g., harm the patient and / or not function properly) due to, for example, the metallic object. The surgical plan for each patient may include a proposal for surgical step options to avoid metal. The surgical plan for each patient may include a proposal for alternative instrument(s) to use in high metal areas. The surgical plan for each patient may include the suitability of energy algorithms and / or integrity tests (e.g., to improve detection) to verify integrity before a harmful event occurs and / or after passing through an area.

[0256] The surgical plan for each patient may include an instruction indicating the relationship (e.g., interaction) between the surgical steps and the patient-specific surgical data. For example, the relationship (e.g., interaction) between the surgical steps and the patient-specific surgical data can be identified. The patient may be scheduled for a surgical procedure such as a colorectal sigmoid resection (e.g., for Crohn's disease which may refer to a chronic condition related to inflammation). Crohn's disease can affect and / or be related to (e.g., interact with) the patient's biomarkers and / or physiological aspects. Comorbidities can be affected. Comorbidities of patient data and / or treatments (e.g., blood pressure, blood sugar, blood thinners, painkillers, etc.) may have a physiological impact that can affect (e.g., interact with) device selection, device setup, or surgical steps. The surgical plan can be adapted based on the relationship (e.g., interaction). For example, the surgical plan for each patient can indicate a recommendation to adapt the surgical plan based on the relationship (e.g., interaction).

[0257] The surgical plan for each patient may include an access port location (e.g., an initial access port location). The access port location (e.g., for a surgical procedure) may be identified based on, for example, patient-specific surgical data, instrument selection, surgical steps, and / or the like. Surgical site access may be improved based on, for example, the access port location. The access port location may be associated with a location on the patient and / or an angle of a surgical device being used at the surgical access site.

[0258] The surgical plan for each patient may indicate, for example, the amount of access to one or more preselected access ports (e.g., the amount of access that each preselected access port may provide). The surgical plan for each patient may be able to indicate an overlap of interactions of instruments used in a region, for example, based on trocar position. The surgical plan for each patient may include an overlay of patient-specific data and / or images on the input surgical plan. The surgical plan for each patient may indicate the trocar location(s). The surgical plan for each patient may be able to indicate access capabilities based on, for example, a combination of patient-specific surgical data.

[0259] Figure 15 shows the identification of the initial access port position. For example, the access port can be identified. As shown in Figure 15, the first access port (e.g., port A) and the second access port (e.g., port B) can be identified. The access port may be determined based on standard human anatomical structures. The access port can be determined based on standard human anatomical structures and / or patient-specific surgical data (e.g., biometric data and / or preoperative imaging of the patient's anatomical structure). The patient-specific surgical plan can show the patient, the surgical site, and / or the view of the access port. The patient-specific surgical plan can show occlusion (e.g., partial occlusion) based on, for example, patient-specific surgical data / images. The patient-specific surgical plan can include anatomical images (e.g., 3D anatomical structures), and the anatomical images can be patient-specific from the aggregated images. The patient-specific surgical plan can show the planned access port, for example, through the thorax for instruments and / or cameras. The patient-specific surgical plan can include an alternative access port approach that can provide, for example, additional access and / or visualization (e.g., to view the occluded area and / or better access to the tumor), and can include images with overlays.

[0260] Figure 16 illustrates an exemplary overlay of patient data and imaging on a surgical plan. As shown in Figure 16, the patient-specific surgical plan and / or overlay can show the surgical steps that may be affected. The surgical steps can be affected based on the selected access port. For example, using different access ports can change the surgical plan. The overlay can be used to show the surgical steps affected by an inadequate access port.

[0261] An instrument positioning and / or access envelope can be determined. The instrument positioning and / or access envelope can be determined, for example, based on patient-specific surgical data and / or a surgical template. An instrument movement envelope can be predicted for the inside (e.g., end effector, shaft) and / or outside (e.g., shaft, shroud, robotic arm) of the patient's wall. The instrument positioning and / or access envelope may be displayed. For example, the instrument positioning and / or access envelope may be included in a patient-specific surgical plan.

[0262] Identification of the robotic modality (e.g., outside the patient) may be automated. The interaction of the robotic modalities may be determined (e.g., automatically). For example, the identification may be performed using patient-specific surgical data, surgical steps, surgical baseline / template, and / or surgical device / tool selection. Conflicts (e.g., potential conflicts) and / or collisions may be determined (e.g., as a baseline). Alternative access port positions may be indicated, for example, based on potential conflicts and / or collisions. Alternative access port positions, patient position, instrument mix, and / or the like may be determined (e.g., and emphasized) to minimize determined complications.

[0263] The optimal instrument positioning and / or access envelope can be determined, for example, to minimize inappropriate interactions (e.g., fighting with a sword) of the shaft within the patient's body (e.g., when the HCP interacts with the surgical site). The determined instrument positioning and / or access envelope can be determined, for example, to prevent collisions of the handle, arm, and / or shroud (e.g., based on the interaction of the access port position, surgical steps, and / or patient-specific surgical data).

[0264] The surgical plan for each patient may be determined (e.g., automatically determined) based on, for example, facility information. The facility information may include the availability of surgical tools, the availability of surgical devices, inventory stock, the availability of surgical equipment, staff allocation, the availability of facility rooms, and the like. The facility information can be used to adjust the surgical plan for each patient. For example, the surgical plan for each patient may indicate using a first surgical tool during the surgical operation, but there may be a case where the first surgical tool is not available (e.g., not sterilized, down for repair / maintenance). The surgical plan for each patient may be adjusted to use a second surgical tool (e.g., an alternative surgical tool) instead of the first surgical tool. Based on the adjusted use of the second surgical tool, the surgical plan for each patient can be adjusted (e.g., subsequent tasks / steps in the surgical plan can be adjusted).

[0265] The computing system can perform verification (e.g., automatic verification) in the planned surgical operation. For example, automated verification can be performed for the instrument requirements, the surgical date, the use of the facility, the received shipments associated with the surgical tools / consumables used in the surgical operation, and / or the like. The computing system can determine (e.g., automatically determine) problems related to the planned surgical instruments to be used in the surgical operation.

[0266] The scheduling time of the surgical operation can be determined based on, for example, the surgical plan for each patient. The computing system can determine the scheduling time based on the instruments selected in the surgical plan for each patient, the availability of staff, the availability of the facility, and the like. For example, the scheduled time may be a time when the selected instrument is available (e.g., mostly available, all available).

[0267] The device / consumable may be ordered based on, for example, inventory information and the device / consumable selected to be used in a surgical procedure. For example, a computing system may determine that a surgical consumable should be used in a surgical procedure and may determine that the inventory is low. The computing system may order replacement / additional consumables for the surgical procedure if, for example, replacement / additional consumables are used (e.g., required) to complete the procedure.

[0268] The patient-specific surgical plan may include instructions associated with the facility inventory and the surgical device selected for the surgical procedure. The instructions may indicate inventory options (e.g., inventory that may be used in place of the inventory selected in the surgical plan). The instructions may highlight surgical devices selected for the surgical procedure that may be lost / unavailable (e.g., at the facility).

[0269] A computing system can determine a patient-specific surgical plan based on facility information and / or other surgical procedures that are planned. A healthcare facility may perform multiple surgical procedures, and the surgical procedures may be performed simultaneously and / or overlap in time. Surgical devices may be used in two or more surgical procedures. Surgical devices and / or consumables may already be selected for a surgical procedure and thus may not be used in different surgical procedures performed simultaneously. The computing system can consider other surgical procedures that are planned (e.g., when determining the patient-specific surgical plan). The computing system can interact with other users who plan procedures that may use the same staff / personnel and / or equipment. The computing system can prioritize surgical procedures. For example, a first surgical procedure may be prioritized over a second surgical procedure. The first surgical procedure may have priority in device selection and / or consumable selection. The prioritization may be performed based on requirements associated with the surgical procedure, the risk level associated with the surgical procedure, and / or the like.

[0270] In multiple instances, a surgeon may create a baseline surgical plan / map. The baseline plan may include surgical devices to be used in the surgery. The computing system may determine (e.g., based on facility information and the baseline plan) that a selected surgical instrument (e.g., Echelon 60) is not available in stock. The computing system may indicate that the surgical instrument is not available for the surgery. The computing system may indicate, for example, an alternative surgical instrument (e.g., Echelon 45) that may be available. The computing system may indicate to the surgeon that the selected surgical instrument is not available and recommend an alternative surgical instrument. The surgeon may accept or reject the recommendation. The computing system may update the baseline surgical plan, for example, if the surgeon accepts the recommendation to use an alternative surgical instrument. The baseline surgical plan may be updated to accommodate the use of an alternative surgical instrument (e.g., instead of the selected surgical instrument). For example, the baseline surgical plan may be adjusted (e.g., to account for differences in two surgical devices) such that the surgical steps may include the possibility of additional firings from an alternative stapler. The computing system may indicate that the adjusted baseline surgical plan is adjusted to increase the number of cartridges (e.g., to compensate for differences with the alternative surgical instrument). The computing system may indicate to the surgeon the cartridge color selection for additional firings. The computing system can update the baseline surgical plan (e.g., if the surgeon accepts the modification) and can order additional supplies based on the modified surgical plan. The computing system may verify that an additional approach (e.g., additional firings using an alternative surgical tool) is acceptable from a risk perspective.

[0271] The surgical plan for each patient can be determined, for example, based on the availability of rooms, patients, and / or staff. The surgical plan for each patient can overlay the availability of rooms, patients, and / or staff onto a baseline surgical plan. The surgical plan for each patient can include follow-up observations based on the availability of rooms, patients, and / or staff. For example, automated patient scheduling for surgery and / or follow-up observations may be performed.

[0272] For example, a computing system can perform scheduling correlations (e.g., automated scheduling correlations) for the availability of facility rooms, equipment, personnel, and / or surgeons. The computing system can determine, for example, the time (e.g., optimal time) for a planned surgery based on, for example, a surgical procedure (e.g., an acute critical need for surgery). For example, the computing system can consider whether a planned surgical procedure is an emergency. The computing system can perform escalations associated with surgical timing and / or scheduling based on, for example, patient needs and / or other planned surgeries. The computing system can aggregate a patient's scheduled treatments to determine, for example, the implications of co-morbidities associated with a planned surgical procedure. The computing system may consider, for example, preoperative patient progress towards a predefined goal (e.g., weight loss) as a trigger for changing surgical scheduling. The computing system can update patient biomarkers and / or patient examinations of patient health from monitoring HCPs (e.g., surgeons, attending physicians, pharmacists, physical therapists, and / or equivalents). The computing system may perform comparisons (e.g., automated comparisons) between patients (e.g., patients of other scheduled surgeries), which can affect current scheduling and / or create conflicts with current scheduling. The computing system can propose changes to the scheduling of a surgical procedure, for example, to resolve determined conflicts.

[0273] A computing system may perform surgical scheduling based on the availability and / or scheduling of personnel. The computing system may consider the availability of HCPs including, for example, anesthesiologists, physician assistants, scrub nurses, oncology / radiology specialists, technicians, urologists, etc.

[0274] Systems, methods, and means for surgical image situation recognition and identification of image shapes based on annotation of shapes or pixels are disclosed. A surgical video including video frames may be acquired. Surgical context data for a surgical procedure may be acquired. Elements within the video frames may be identified using image processing based on the surgical context data. Annotation data may be generated for the video frames, for example, based on the surgical context data and the identified element(s).

[0275] Elements within the surgical video may be identified, for example, by a surgical computing system. For example, an image shape may be identified in the surgical video and the image shape may be linked to a surgical element. For example, the image shape may be identified as a structure, organ, feature, and / or other element in the surgical procedure. For example, an image element may be identified as a lung, tumor, blood, artery, and / or other anatomical element. Features associated with the element may be identified. The identified features may include, for example, one or more of the following. State, condition, type, tissue type (e.g., fat, fat and blood vessel, duct, organ, fat and organ, connective tissue, etc.), tissue condition (e.g., inflammation, fragility, calcification, edema, hemorrhage, carbonization, etc.).

[0276] The identification of such elements and / or features can be performed, for example, using image processing on a surgical video (e.g., a surgical video frame) and using situation awareness associated with the surgical image. Situation awareness can be achieved using surgical context data and / or surgical procedure data obtained via a situation awareness system (such as described herein with respect to FIG. 7). Annotation data can be determined based on the identified elements. The annotation data may be inserted into the surgical video.

[0277] A surgical video or surgical video frame can be analyzed into grouped elements and / or groups of pixels. Throughout the surgical video, the elements can indicate movement, behavior, and / or outcomes (e.g., surgical events such as bleeding). The elements can be tracked between surgical video frames (e.g., consecutive frames) within the surgical video. Tracking data (e.g., behavior, movement, and / or outcome data) can be determined using, for example, surgical context data and the identified elements. The tracking data can be included in the annotation data and may be inserted into the surgical video. The tracking data can be used for verification of expected movement, behavior, and / or outcomes.

[0278] A surgical computing system can perform annotation of pixels in an image. The annotation of pixels can be used for machine learning, object identification, object tracking, self-identification, etc. The annotation can include situation awareness and annotation of individual pixels and / or groups of pixels within the surgical video.

[0279] FIG. 17 shows an example of annotating a surgical video based on situation awareness of a surgical image. As shown in FIG. 17, a surgical computing system 50350, which can be a surgical hub 20006 as shown in FIG. 1, can obtain a surgical video 50352 and a surgical context 50354 (e.g., surgical context data).

[0280] The surgical context 50354 may indicate, for example, surgical operations, surgical events, surgical steps, surgical phases, surgical tasks, complications, etc. The surgical context 50354 can be determined. For example, the surgical context 50354 can be determined using situation awareness (e.g., as described herein with respect to FIG. 7).

[0281] As shown, the surgical context 50354 associated with the surgical video and / or surgical operation can be determined, for example, based on surgical data 50366 (e.g., surgical operation data). The surgical data 50366 may be surgical data generated before, during, and / or after the surgical operation. For example, the surgical data 50366 may be surgical operation data 50020 as shown in FIG. 8. The surgical data 50366 may include, for example, surgical data generated from surgical systems such as surgical images 50366a, surgical sensors (s) 50366b (e.g., wearable), surgical instruments (s) 50355c, surgical devices 50366d, monitoring systems (s) 50366e, etc.

[0282] The surgical context 50354 can be determined, for example, based on surgical data 50366 and a surgical plan 50368 (e.g., a surgical operation plan). The surgical plan 50368 may include, for example, information associated with the surgical operation such as surgical steps, surgical tasks, surgical instruments used, surgical devices used, etc. The surgical operation plan 50368 may be a patient-specific surgical operation plan (e.g., as described herein with respect to FIGS. 11, 14, and 16). The surgical context 50354 may be a patient-specific surgical context for the surgical operation. The surgical context 50354 may be determined, for example, based on the patient-specific anatomical structure and / or medical records.

[0283] The surgical context 50354 may represent, for example, surgical steps. The surgical steps may be associated with making an incision in a part of the lung. The surgical context 50354 may include information associated with the surgical instruments used, the surgical equipment used, the organs at the surgical site, the characteristics of the surgical site, etc. The surgical context 50354 may be used to guide a surgical operation.

[0284] As shown in FIG. 17, the surgical computing system 50350 may acquire a surgical video 50352. The surgical video may be composed of video frames (e.g., surgical video frames) such as video frame 50352a and video frame 50352b. The surgical video may be a video of a surgical operation.

[0285] The surgical computing system 50350, which may be a surgical hub 20006 as described in FIG. 1, may perform element recognition and / or element identification as shown in 50356. The surgical video may be associated with endoscopic surgery (e.g., in-body video). For example, the surgical video can capture the video inside the patient's body during a surgical operation. The surgical video can capture videos showing elements inside the patient's body such as organs, structures, features, anatomical structures, surgical instruments (e.g., parts of surgical instruments such as tips, jaws, knives, end effectors, etc.), tissues, tumors, arteries, veins, bronchi, etc. The elements inside the patient's body can be identified.

[0286] Element identification and / or recognition may be performed, for example, using image processing (e.g., as described herein) based on a surgical context 50354 and situation recognition (e.g., as described herein). As described herein, the surgical context 50354 can provide context regarding a surgical procedure and / or elements identified within a surgical video during a surgical procedure. For example, the surgical context 50354 can provide information related to a lung surgery. The surgical context 50354 can be used to identify elements within a surgical video, such as a lung. The surgical context 50354 can be used to distinguish elements within a surgical video from other similar elements. For example, organs may share similar shapes in a surgical video, but the appropriate organ can be identified using the surgical context. For example, during a lung surgery, the surgical context can be used to identify an element as a lung rather than a different organ having a shape similar to the lung. For example, the surgical context 50354 can indicate that a particular surgical device is being used and can be used to identify the surgical device or a part of the surgical device within a surgical video frame(s).

[0287] The surgical computing system 50350 can identify elements within the surgical video 50352. The surgical computing system 50350 can perform element identification and / or recognition on the surgical video 50352, for example, by using video frames. The surgical computing system 50350 can perform element identification and / or recognition on each video frame, such as a first video frame (e.g., video frame 50352a) and a second (e.g., subsequent or previous) video frame (e.g., video frame 50352b). The video frames can be processed to identify elements within each video frame.

[0288] The surgical video 50352 and / or video frames may be composed of pixels. Elements within a video frame may be composed of, for example, groups of pixels. A first group of pixels may be clustered and identified as constituting a first element within the video frame. The surgical computing system 50350 may determine a first group of pixels associated with the first element. The identified element(s) may be composed of respective groups of pixels. The identified element(s) may be separated into sub-elements each including the respective element. The sub-elements may be composed of sub-groups of pixels associated with each sub-element.

[0289] The surgical computing system may use one or more selection methods, for example, to identify elements within a surgical video. The selection methods may be associated with the use of one or more of a bounding box, a polygon, a point (e.g., a keypoint), a cuboid, and semantic segmentation.

[0290] For example, the surgical computing system can identify elements within a surgical video using the bounding box selection method. The bounding box selection method may be associated with using a rectangular structure to match elements. For example, the rectangular structure may be related to x(min) / y(min) and x(max) / y(max). The bounding box may enable the detection and / or recognition of objects of different classifications. The box may be anatomically positioned, for example, corresponding to the expected position of an organ, structure, and / or tissue (e.g., based on the identification of important landmarks such as other organs, structures, and / or tissues). For example, the distinction between elements of similar shapes (e.g., the distinction between the stomach and the liver) may be possible regardless of shape and / or color similarity. Boxing (e.g., automatic boxing) may enable the surgical computing system to define differences (e.g., differences between elements of similar shapes and / or colors) using, for example, limited processing in a gross approach.

[0291] For example, a surgical computing system can identify elements within a surgical video using a polygon selection method. The polygon selection method can be associated with using a series of selected points that, when used together, can create a polygon structure that enables classification differentiation. For example, the points may be around the element. The points may be an outer perimeter based on one or more of color, texture, the morphology of multispectral imaging with visual imaging, etc. The number of points defining the polygon may increase the processing and / or selection timing.

[0292] For example, a surgical computing system can identify elements within a surgical video using a cuboid selection method. The cuboid selection method can be associated with using a box (e.g., a three-dimensional box) selection that enables a multi-dimensional display.

[0293] For example, a surgical computing system can identify elements within a surgical video using a semantic segmentation selection method. The semantic segmentation selection method can be associated with per-pixel tissue semantic annotation of tissue (e.g., all tissue). An overlay of positions where elements (e.g., organs and / or tissues) can be generated can be generated. A method for predicting the elements may be determined. In an example, each pixel and / or group of pixels can be annotated and / or labeled with respect to which element the pixel is associated with.

[0294] A surgical computing system can identify (e.g., automatically identify) that the selection method used for element identification is inappropriate (e.g., inaccurate). For example, the surgical computing system can use a predefined baseline approach. For example, if a threshold amount of irregularities and / or inconsistencies is identified using the selection method, the surgical computing system can adjust (e.g., automatically adjust) the selection method. The surgical computing system can compare the results (e.g., between different selection methods) and select a selection method that has the minimum amount of resources used and / or the minimum amount of false positives.

[0295] A surgical computing system can perform a selection rendering. The selection rendering can be associated with instance segmentation and / or bordering. For example, the surgical computing system can perform an instance segmentation associated with the identified element. The instance segmentation can include taking an element and separating it into a subset of multiple instances. The surgical video 50352 and / or the elements within the video frame can comprise sub-elements. For example, the element can be an organ (e.g., a lung), and the organ can have distinguishable parts.

[0296] Identified element(s) can be separated into sub-elements each containing the respective element. A sub-element can be composed of a sub-group of pixels associated with each sub-element. For example, a surgical computing system can identify that an element is the small intestine or jejunum. The jejunum can be separated into multiple anatomical parts (e.g., 4 anatomical quadrants). The surgical computing system can identify the overall structure of the jejunum. The surgical computing system can perform a bunching of groups of quadrants within the jejunum to define 4 parts of the jejunum. The surgical computing system can use the distinguishable parts of the jejunum to determine information associated with a video frame. For example, the surgical computing system can use the information associated with the distinguishable parts to identify where the surgical system is looking.

[0297] For example, a group of pixels associated with an element can be further divided into sub-groups of pixels. The sub-groups of pixels may constitute a sub-element. For example, a first sub-group of pixels can include pixels that constitute the left lung, and a second sub-group of pixels can include pixels that constitute the right lung. The first sub-group of pixels can include pixels that constitute the first lobe in the left lung, and the second sub-group of pixels can include pixels that constitute the second lobe in the left lung.

[0298] For example, the lung can be composed of the left lung and the right lung. The left lung may be distinguishable from the right lung (e.g., the left lung may be smaller than the right lung, and / or, for example, may be composed of notches to provide space for the heart). The lung can be composed of lobes (e.g., can be divided into lobes). The lung can be composed of 5 lobes. For example, the left lung can include 2 lobes, and the right lung can include 3 lobes. The lobes may be distinguishable from each other. The surgical computing system can identify sub-elements (e.g., lobes, left lung, and / or right lung) within an element based on surgical context data 50354.

[0299] A surgical computing system may use surrounding elements and / or features within a video frame, for example, to identify the lungs and / or sub-elements of the lungs (e.g., lobes). The surgical computing system may use the shape of the lungs, rib cage, and / or other hard landmarks to, for example, orient itself and separate different segments of the lungs. The user can demarcate and / or review the parts of interest and / or pay attention to the impact on other parts.

[0300] Based on surgical context data 50354, a surgical computing system can, for example, perform outlining in a video frame. The surgical computing system can distinguish tissue and / or organs, for example, from the connective tissue background. For example, when a structure (e.g., an element) is identified and / or selected and partitioned into instances, the distinction can be made. The boundary around the sub-elements (e.g., instances) can be defined. An object may be defined (e.g., self-defined) based on the boundary, for example. When a boundary overlaps with the boundary of another segment, the shared boundary can be aligned between organs and / or tissues, for example, to eliminate the overlap.

[0301] A surgical computing system can highlight and / or draw lines around elements within a surgical video and / or video frame(s). When a structure is identified and / or segmented, the surgical computing system can change the contrast and / or brightness of the video frame, for example, to highlight elements and / or features within the frame. An object of interest may be selected to be highlighted. The contrast and / or brightness of the object of interest may be changed, for example, to enable the user to distinguish the object of interest from other elements within the surgical video. Instances and / or sub-elements can be highlighted based on, for example, surgical context data 50354.

[0302] A surgical computing system can determine characteristics and / or features associated with identified elements. For example, the surgical computing system may determine element identification. The surgical computing system can determine characteristics and / or features such as, for example, one or more of the following: state, status, type, tissue type (e.g., fat, fat and blood vessels, ducts, organs, fat and organs, connective tissue, etc.), tissue state (e.g., inflammation, fragility, calcification, edema, hemorrhage, carbonization, etc.).

[0303] As shown in FIG. 17, a surgical computing system 50350 can perform element tracking 50358. The surgical computing system can perform element tracking (e.g., object tracking) on elements in a surgical video, for example. Element tracking can include determining tracking information about an element in a video frame. For example, an element in a first video frame can be identified in a second video frame. Behavior, movement, and / or outcome can be determined for the element using the first and second video frames. For example, the second video frame can show the element at a location different from the location where the element was in the first video frame. Information about the element can be determined across video frames.

[0304] The surgical computing system 50350 can perform verification, as shown at 50360, for example. Verification can include verifying and / or tracking identified elements. Element tracking and verification are described in more detail with reference to FIG. 19.

[0305] As shown in FIG. 17, the surgical computing system 50350 may generate annotation data, for example, based on performed element recognition, element tracking, and / or verification (e.g., as shown in 50362). The surgical computing system may determine annotation data for surgical videos and / or video frames. The computing system may determine annotation data for surgical videos and / or video frames, for example, using identified elements within the video frame and surgical context data. For example, the annotation data may include element identification information and / or element tracking information. The annotation data may include sub-element identification within the element. The annotation data may include general information associated with the video frame, such as, for example, surgical steps, surgical tasks, surgical events, etc. associated with the video frame. The annotation data may include one or more of the following: element identification information, element grouping information, element subgrouping information, element type information, element description information, element condition information, surgical step information, surgical task information, surgical event information, surgical instrument information, surgical device information, and / or the like.

[0306] The annotation data may be generated based on surgical events shown in the surgical context. Event-driven annotation (e.g., automatic annotation) may be performed. For example, the annotation may be automated using the surgical computing system. Data flagging may be performed, for example, by the surgical computing system or a sensor. For example, event-driven annotation may be performed for linked surgical tasks, linked instruments, or instrument setups.

[0307] Event-driven annotation associated with the micro-outcome of a previous surgical step may be performed. For example, the previous surgical step may indicate bleeding. Event-driven annotation may include determining the annotation data to include the amount of bleeding, the timing of the bleeding (e.g., over how many frames the bleeding occurs), the time associated with the release from the primary surgical instrument, the use of secondary observation intervention surgical tools, and / or the like.

[0308] Event - oriented annotations may be associated with the determination of an extracorporeal image. For example, a surgical computing system may be configured to edit (e.g., blank out) an image detected as being extracorporeal.

[0309] Event - oriented annotations may include inserting annotation data that includes local metadata and / or global metadata. Local metadata may include lead - up information, e.g., information associated with adjacent video frames. Local metadata may include continuous variables and / or discrete variables. Global metadata may include information associated with a patient, an instrument, a surgery, a surgeon, and / or the like. The metadata may be inserted into the surgical video.

[0310] Event - oriented annotations may include inserting annotation data associated with a response to an identified event. For example, a video frame may be associated with a surgical event. A response to the surgical event may be determined. An algorithm (e.g., a query algorithm) may be used to detect the resulting event. For example, carbonization may be detected. The annotation data may indicate, for example, focusing on generator data to provide information about the detected event.

[0311] Annotations may be performed during and / or after a live surgery. Annotation data associated with complex improvements to a surgery may be performed after the surgery. Live annotations may include simpler annotations and / or processing tasks. For example, live annotations may be performed to minimize the need for secure storage and / or processing of live surgical data.

[0312] Annotations can be guided by a surgical map. FIG. 18 shows exemplary annotations associated with a surgical map. The surgical map can be included in a surgical plan. The surgical map can show potential surgical steps and / or surgical outcomes associated with a surgery. For example, the surgical map can show potential characteristics associated with elements (e.g., organs, structures, and / or features) in a surgery. For example, the surgical map can show potential tissue types, tissue states, and / or surgical intentions associated with a surgery. The surgical map can guide a surgery. The surgical map can be used, for example, as a dictionary for performing annotation of a surgical video and / or video frames associated with a surgery.

[0313] The surgical map can show a surgical workflow. For example, the surgical map can show potential results associated with a surgical workflow.

[0314] In some examples, annotation data can be inserted into a surgical video. The annotation data can be information associated with the frame itself and / or metadata associated with a surgery and / or event. An annotated surgical video 50364 can be generated and output, for example, to a surgical system, a storage device, etc. The annotation data can be inserted into the surgical video, for example, at the video frame level. The annotation data can be attached to the video frame. The annotation data can be inserted into the surgical video, for example, at the pixel level (e.g., pixel group level). For example, each pixel in a video frame can have respective annotation data inserted into the pixel. The annotation data for each pixel can include information identifying the element with which the pixel is associated.

[0315] In an example, a group of pixels can be associated with a lung. A surgical computing system can identify the lung and determine annotation data associated with the lung. The surgical computing system can insert lung annotation data into the group of pixels associated with the lung. The annotation data inserted into the pixel group can include identification information, lung condition information, and / or the like. The surgical computing system can identify an instance of the lung (e.g., a lobe of the lung). The instance of the lung can be composed of a subgroup of pixels. The surgical computing system can insert sub-element annotation data associated with each instance of the lung into the associated subgroup of pixels. For example, the surgical computing system can insert respective sub-element annotation data into the subgroup of pixels associated with the first lobe of the left lung. The sub-element annotation data inserted into the subgroup of pixels associated with the first lobe of the left lung can include identification information that can indicate the sub-element.

[0316] In some examples, the annotation data may be used, for example, to generate a control signal (e.g., indicating determined control parameters) to a surgical control system. In some examples, the annotation data may be stored and / or transmitted separately from the surgical video (e.g., to a surgical control system, a surgical analysis system, etc.).

[0317] FIG. 19 shows an exemplary flow for determining element tracking information and verifying elements in a surgical video. The surgical video may include video frames. For a selected video frame(s), the element tracking information and verification may be determined using, for example, a previous video frame. The element tracking information may be used for element tracking information and verification determination for subsequent video frame(s). As shown in FIG. 19, the current frame 50380 of the surgical video is after the temporally previous frame 50382 and may be before a subsequent frame 50384. The surgical computing system may perform element identification on the previous frame 50382 (as shown at 50388, for example). The element identification for the previous frame may be performed based on surgical context data 50386 (as described herein, for example).

[0318] The surgical context data 50386 may be refined and / or updated (as shown at 50390 in FIG. 19, for example). For example, the surgical context data may be refined and / or updated based on identified element(s) in a first video frame. The element(s) in a second video frame may be identified using, for example, the refined and / or updated surgical context data (using image processing, for example). Annotation data for the second video frame may be determined using the refined surgical context data and the element(s) identified in the second video frame.

[0319] For example, surgical context data 50386 can be refined and / or updated using identified elements within a previous frame 50382. For example, identified elements within the previous frame 50382 can provide additional information regarding the surgical context of a surgical procedure. The additional information can be used to verify the surgical context data. If there is a discrepancy between the surgical context data and the identified elements, the surgical context data can be updated. For example, the surgical context data can indicate that a first surgical instrument was being used at the time associated with the previous frame 50382. The element identification information may indicate that a second surgical instrument was being used in the previous frame 50382 (e.g., instead of the first surgical instrument). The surgical context data can be refined and / or updated for use in subsequent frames. The element identification information can indicate that the first surgical instrument was being used in the previous frame 50382 (e.g., the same as the surgical context data). The surgical context data can be verified, for example, based on confirmation identification information.

[0320] Refined surgical context data 50930 can be used for element identification, element tracking, and / or verification for the current frame 50380. Element identification may be performed, for example, for the current frame 50380 (as shown at 50392) using refined surgical context data 50390 (such as described herein). Element tracking and / or verification can be performed, for example, for a surgical video using the previous frame 50382 and the current frame 50380. Although not shown in FIG. 19, those skilled in the art will appreciate that element tracking information derived from the current video frame can be used to refine and / or update the element identification of the previous video frame(s).

[0321] As shown in 50394, element tracking and / or verification can be performed using the identified elements from the previous frame 50382 and the identified elements from the current frame 50380. Tracking information associated with the surgical video can be determined using, for example, multiple video frames. The elements identified within the first video frame and the elements identified within the second video frame can be used, for example, to determine tracking data. The tracking data may be associated with element behavior, element movement, and / or result information (e.g., associated with the element). The determined tracking data can be included in the annotation data.

[0322] For example, movement, behavior, and / or result information can be determined for elements spanning the previous frame 50382 and the current frame 50380. The elements in the previous frame 50382 can be identified as being in different positions within the current frame 50380. The tracking data can be determined based on the movement of the elements.

[0323] The difference in elements between the previous frame 50382 and the current frame 50380 can indicate the behavior of the elements. For example, the elements can be determined to be pulsating and / or exhibiting repetitive motion over the video frames. The repetitive movement of the elements can be confirmed, for example, by tracking the elements over the video frames. Element identification can also be performed using the identified repetitive motion of the elements. For example, the elements may behave and / or move in a manner indicative of a particular organ and / or feature. The tracking data of the expected rhythmic movement of the elements can indicate the organ and / or feature. The determined movement can also be used, for example, to improve the surrounding boundary of the element with respect to the background of the video frame (as described herein with respect to the selection method). The tracking data can be used to confirm and / or disprove the type and / or classification of the element.

[0324] The difference in elements between the previous frame 50382 and the current frame 50380 may indicate the result associated with the elements. For example, the elements within the previous frame 50382 and the current frame 50380 may be identified as organs. The state of the organ in the previous frame 50382 may be identified, for example, as not bleeding. The state of the organ in the current frame 50380 may be identified as bleeding. The change from the non-bleeding state to the bleeding state may indicate a surgical event. A bleeding event can be determined and included in the annotation data.

[0325] The element identification of the current frame 50392 can be used to further refine and / or update the surgical context data (as shown, for example, in 50396). The further refined and / or updated surgical context data 50396 may be used for element identification, element tracking, and / or verification for subsequent frames (such as subsequent frame 50384, etc.). For example, as shown in 50398, the element identification of subsequent frames can be performed. Element tracking and / or verification can be performed based on the current frame 50392 (such as elements from the current frame 50380) and subsequent frame 50398 (such as elements from subsequent frame 50384) (as shown, for example, in 50400).

[0326] Tracking data and / or information may be verified, for example, using predicted tracking information. The predicted tracking information may be obtained, for example, based on surgical context data and / or a surgical plan. The annotation data may include an indication that the elements and / or tracking data have been verified. The annotation data may include an indication that the elements and / or tracking data may require further verification.

[0327] Verification can be performed on identified elements within a video frame(s), for example, based on predicted tracking information. The predicted tracking information can be associated with a surgical context (such as indicated by surgical context data). The predicted tracking information can be determined based on surgical context data, for example, by using a surgical plan (such as a patient-specific surgical plan described herein with reference to FIG. 11). For example, the surgical plan can indicate that a surgical instrument is to be used during a surgical step. An element can be verified as a particular surgical instrument, for example, based on the surgical computing system identifying the element as a particular surgical instrument and the surgical context data indicating that the particular surgical instrument data is appropriate for the surgical step. The verification can be included in annotation data. For example, the annotation data can indicate that the identified element has been verified.

[0328] Jobs, results, and / or constraints can be determined, for example, for a surgical video. Jobs, results, and / or constraints can be determined based on surgical context data. Surgical task information associated with a first video frame can be determined using surgical context data and identified elements within the first video frame. Comorbidity information can be determined, for example, in relation to a determined surgical task using surgical context data, surgical task information, and identified elements within a first surgical video frame. Result information associated with a surgical task can be determined, for example, based on surgical context data, surgical task information, and comorbidity information.

[0329] Jobs, results, and / or constraints can be compiled, for example, for a surgical procedure and / or surgical step. For example, annotation data can be used to compile jobs, results, and / or constraints. The compiled jobs, results, and / or constraints can be used, for example, for identifying problems, changing control algorithms and / or parameters, and recommending procedural changes for subsequent steps and / or procedures (such as in conjunction with machine learning).

[0330] Surgical task information can be determined for video frames in a surgical video. For example, the surgical task information can be determined using surgical context data and identified elements within the video frame. For example, the surgical task information can indicate information associated with the surgical task performed in the video frame. The video frame can be associated with removing a part of the lung using a surgical instrument. The surgical task information can include information associated with the surgical instrument used (e.g., the type of surgical instrument, surgical instrument parameters, etc.), the time of removing a part of the lung, and the like.

[0331] Complication information may be determined for video frames in the surgical video. For example, the complication information can be determined based on surgical context data, surgical task information, and identified elements within the video frame. The complication information can include information indicating a complication detected in the video frame. The complication information can include information indicating possible complications that may occur based on the analysis of the video frame. For example, the video frame can be annotated with information indicating the risk of complications in subsequent surgical steps in the surgical procedure. The annotation data can indicate potential complications.

[0332] Outcome information can be determined for video frames in the surgical video. The outcome information can be associated with the surgical task within the video frame. The outcome information can be determined based on surgical context data, surgical task information, and complication information. The outcome information can indicate the likelihood of success of the surgical step. The outcome information can indicate the likely outcome of the surgical task being performed within the video frame. The outcome information can be used to change subsequent surgical tasks and / or steps in the surgical procedure.

[0333] The change and / or control parameters may be determined for a surgical procedure by analyzing the surgical video frame(s). The change and / or control parameters may be determined, for example, based on surgical task information, complication information, and / or outcome information. A determination may be made to change the parameters associated with the surgical procedure. The parameters may include parameters associated with operating a surgical instrument, surgical equipment, and / or the like. Based on the determination to change the parameters associated with the surgical procedure, the change and / or control parameters may be determined. A control signal may be generated. The control signal may include an instruction indicating the determined control parameters. The control signal may be transmitted, for example, to a surgical control system that transmits parameter information to a surgical system (e.g., a surgical instrument, surgical equipment, etc.). The change and / or control parameters may be used to perform the surgical procedure.

[0334] The change parameters for a surgical procedure may be determined based on annotation data. For example, the change parameters may be determined based on surgical task information, complication information, and / or outcome information. For example, the outcome information may be able to indicate potential complications of the surgical procedure, for example, when the current parameters are used for subsequent surgical tasks and / or steps. A surgical computing system may be able to determine that different parameters associated with better surgical outcomes (e.g., more efficient surgery, lower risk surgery, and / or better outcomes) may be used.

[0335] A surgical computing system may determine to change parameters associated with performing a surgical procedure. The surgical computing system may determine change parameters for performing the surgical procedure. The surgical computing system may transmit a control signal (e.g., indicating the determined control parameters) to, for example, a surgical control system. The surgical control system may control surgical instruments, surgical devices, and / or the like associated with performing the surgical procedure. The parameters may be adjusted (e.g., automatically) by, for example, the surgical control system to use the change parameters indicated in the control signal.

[0336] FIG. 20 shows an exemplary flow diagram for generating annotation data associated with a surgical video using surgical context data and situation recognition. As shown at 50370, surgical context data may be obtained. The surgical context data may be obtained using, for example, surgical procedure data, surgical procedure plans, and / or the like. As shown at 50372, a surgical video including surgical video frames may be obtained. As shown at 50374, element(s) may be identified within the surgical video frames. The element(s) may be identified based on, for example, the surgical context data. The element(s) may be identified using, for example, image processing (e.g., as described herein).

[0337] As shown in 50376, annotation data can be generated based on, for example, surgical context data and elements (plural) identified within a surgical video frame. For example, the annotation data can include one or more of element identification information, element grouping information, element subgrouping information, element type information, element description information, element condition information, surgical step information, surgical task information, surgical event information, surgical instrument information, surgical device information, and / or the like. In some examples, the annotation data can be inserted into the surgical video. The annotation data may be inserted into the video frame. For example, the annotation data may be inserted at the pixel level (e.g., each annotation data is inserted into each pixel and / or group of pixels). In some examples, the annotation data may be used, for example, to generate a control signal (e.g., indicating determined control parameters) to a surgical control system. In some examples, the annotation data can be transmitted separately from the surgical video (e.g., to a surgical control system, a surgical analysis system, etc.).

[0338] In a surgical operation, parameters can be used to determine between primary control and verification control for a process and / or operation. For example, parameters can be monitored in a surgical operation. Uncertainty associated with the parameters can be determined. The uncertainty associated with the monitored parameters can be used, for example, to determine between primary control and verification control for a process and / or operation (e.g., as a means).

[0339] The reliability and / or uncertainty of data from surgical operation data (e.g., from surgical sensors (plural)) can be used, for example, to determine how to perform an automated task. For example, the reliability and / or uncertainty of data from surgical sensors (plural) (e.g., wearable) can be used to determine which sensors to use to drive and / or execute an automated task.

[0340] The detected reliability and / or uncertainty of the monitored data and / or the magnitude of the uncertainty of the first sensor or detection array and the second detection array can be used, for example, to determine prioritization. For example, the prioritization can be associated with the prioritization of each array in the hierarchy of system or automated task control.

[0341] For example, an important organ proximity detection system(s) can analyze a body cavity. An automated task can use a device (e.g., a surgical instrument) to move within the body cavity. Sensors can be used, for example, to capture data associated with the analysis of the body cavity. Based on multiple sensors, an autonomous system (e.g., a surgical computing system that can be a surgical hub 20006 as described in FIG. 1) can determine (e.g., using an algorithm) which sensor has the most accurate (e.g., best) proximity measurement (e.g., compared to other sensors) to identify the location of an important organ (e.g., to determine a safe path for the movement of a surgical instrument within the body cavity). A camera sensor can have an obstructed field of view, but the camera sensor may be tracking organs and energy devices and estimating (e.g., inferring) their location (e.g., of an important organ). A device (e.g., a surgical instrument within the body cavity) can have an ultrasonic sensor that can identify the location of nearby objects. The autonomous system can cross-reference both sensors (e.g., the camera sensor and the ultrasonic sensor) and determine the safest movement based on the reliability of the data received from the sensors. The organ risk may be considered depending on the device used to navigate the body cavity. For example, activating an energy device near the heart is more dangerous than activating it near the lungs.

[0342] A surgical job or surgical step may be used, for example, to describe (e.g., provide context for) the choices that a surgical system may have for controlling its operation based on, for example, a plurality (e.g., two) of related but different data streams and means for determining how the reliability and selection of each may affect the automatic control of the process.

[0343] Targets of uncertainty data may include, for example, impedance and / or reflectivity, which may be used to characterize tissue in terms of, for example, compression and / or mechanical properties. The impedance and / or reflectivity data may include unstable measurements or competing data elements (e.g., lower impedance may be associated with ore compression). Then, if the data uncertainty exceeds a threshold level, another metric (e.g., reflectivity and / or ultrasonic imaging) may be used as a complement and / or may replace the use of impedance.

[0344] The reliability and / or risk level of an aspect of a surgical procedure may be used, for example, to determine the control dependence on a particular monitored data stream. Automatic risk analysis may be performed, for example, to determine the dependence of a control mechanism on a particular data stream.

[0345] Automatic event recording and storage may be performed. For example, device control may be able to use aggregated intraoperative information from other devices in the same surgical procedure.

[0346] For example, surgical data may be collected autonomously (e.g., from multiple surgical devices). Surgical data may be aggregated from multiple devices throughout a surgical procedure.

[0347] Surgical data may be aggregated throughout a surgery, for example, to predict high levels of patient and / or tissue factors (e.g., factors that may affect the outcome associated with other devices used in the surgery). For example, if the surgical data indicates that a patient is bleeding more than average (e.g., a risk factor), the risk factor may be presented to other surgical devices for use in the algorithms and / or operations of those other surgical devices. For example, stapling parameters and / or controls (e.g., clamp load, pre-compensation time, firing speed, etc.) may be automatically collected from energy device activations throughout the surgery and adjusted based on the queried tissue and / or patient information.

[0348] For example, the surgical data may indicate that the energy activation throughout the surgery may have taken longer than typical to ensure proper sealing. The results may be associated with patient-specific factors, for example, that the patient is more prone to bleeding (e.g., compared to others) and / or does not clot as rapidly as normal biological structures. The information may be included in and / or used in the stapling algorithm and / or operation. The surgical system may adjust surgical instrument parameters (e.g., clamp load, pre-compression time, firing speed, etc.) to reduce bleeding of the staple line, for example, if the patient has a higher bleeding tendency. The surgical hub may recommend staples that are closer together than those typically used by the surgeon for the surgery.

[0349] Surgical device data may be mapped to the location of the patient from whom the data is generated. For example, video analysis and / or other markers may be used to track the location of device use throughout the surgery, and for example, map device data collected for future device use. In an example, during mobilization in a low anterior resection (LAR), energy cutting near the rectum may provide data indicating lower relative tissue perfusion and / or a lower relative likelihood of staple line bleeding in that region. In an example, in the same surgery, energy cutting above the colon may be more perfused and may have a higher likelihood of leakage. To limit bleeding during surgery, staple control and reload selection may vary at each location. The surgical system may track energy activation by location and may track the location of end cutter firing. If the end cutter is proximate to the location where a given energy activation has been completed, the end cutter control and / or parameters may be adjusted accordingly.

[0350] The surgical system may track device data. For example, visible bleeding from video analysis may be determined. Tissue sealing data (e.g., time to completion of sealing and / or cutting) of the energy device can be obtained and / or determined. A perfusion measure can be determined. Densities can be determined and / or viscoelastic analysis can be performed using tissue load curves from various device clamps. Tissue fluid content may be determined, for example, based on impedance (e.g., from an energy device).

[0351] The surgeon may actively request tissue information. The energy device tissue interrogation mode may be used to detect the thickness and / or condition of tissue for use in stapling. Feedback may be provided to the surgeon to support cartridge selection and / or directly applied to the compression and / or firing algorithms.

[0352] For example, the surgeon may activate the tissue matching mode on the bipolar device before cutting with the stapler. The bipolar device may enter a non-therapeutic energy supply mode. The surgeon can, for example, clamp and release the intended staple line cut while non-therapeutic energy passes between Joe's. Impedance and / or other energy data may be used to estimate tissue thickness. The surgical system may communicate the estimated tissue thickness and recommend staple reload parameters to the surgeon. Data recorded from tissue interrogation may be used, for example, as part of the staple fastening compression and / or firing algorithm if the surgeon uses a stapler. The bipolar device may use impedance as a predictor. The harmonic device can use an ultra-low energy mode. Tissue resistance, density, vibration response, and / or the like may be used as a predictor (e.g., by a harmonic device). The staple fastening algorithm may be adjusted based on energy device feedback to determine tissue type, thickness, density, condition, and / or the like. The energy device feedback may be used to confirm the tumor margin before stapling.

[0353] For example, the / or situational relevance can be determined for a surgical procedure, for example, to control the automatic memory of events and timings. Task relevance, risk, and / or criticality can be used to determine, for example, whether aspects of the situation and / or event are remembered, when, where, and / or how often they are remembered.

[0354] Motion sensing may be used, for example, instead of frame - to - frame comparison, for automatic event recording and / or memory, for example. An event - based camera may be used. Identification and re - identification of objects in combination with memory of previous arrangements of keys and / or common objects may be performed to enable object positions to be remembered. The stored coordinate system may be adjusted, for example, as an object is moved, to free frame - to - frame comparison of pixels and images for a system having, for example, stationary, fixed, and / or predefined positions. The amount of image processing for a part of the system view that is less important (for example, where object position is still important) may be limited.

[0355] The following are numbered embodiments that may or may not be claimed. 1. A computer - implemented method comprising: obtaining surgical data associated with a surgical procedure from a plurality of surgical systems; annotating the surgical data using surgical context data related to the surgical procedure; determining data needs for subsequent target surgical system tasks associated with a target surgical system, based at least in part on the annotated surgical data; generating a data package associated with the data needs, the data package including a subset of the annotated surgical data; and transmitting the data package to the target surgical system.

[0356] Annotating surgical data with surgical context data may mean that the surgical data is paired with the surgical context data so that context is provided to the surgical data. In one example, a computer may be able to process the annotated surgical data to read out the surgical data and an indication of the context to which the surgical data relates.

[0357] In one example, the data needs correspond to the data used for the task.

[0358] In one example, the data needs for a future target surgical system task are the data used for that task (associated with the target surgical system). At its broadest level, a target surgical system is simply a surgical system associated with (e.g., performing or monitoring) each surgical task. Subsequent target surgical system tasks can be future surgical tasks to be performed after the current task of a surgical procedure.

[0359] In one example, generating a data package includes selectively differentiating annotated data, e.g., by reverse compiling the annotated data such that only a subset associated with the data needs remains. In other words, a second subset of the annotated data not associated with the data needs is removed / ignored.

[0360] Advantageously, annotating surgical data to provide context for that data enables the processor to more quickly identify a given context, e.g., a subset of that data associated with a particular surgical step. Advantageously, by pre - determining the data requirements for a surgical task and sending a data package that includes data matching those requirements (rather than all of the acquired data), the target surgical system associated with that task is provided with relevant data upfront and without extra data. This reduces latency in performing the surgical task since it is no longer necessary to analyze all of the data acquired to find the relevant subset. The process of determining the data requirements itself is made more efficient by the previous annotation step, and thus there is a cumulative time - efficiency benefit across all that is claimed.

[0361] 2. Determining surgical context data based on surgical data, the surgical context data including an indication of a current surgical step in a surgical procedure, and further including determining an indication of a subsequent target system task associated with the target surgical system, the method according to Embodiment 1.

[0362] Advantageously, determining surgical context data based on surgical data eliminates the need for a medical expert to enter context, thereby automating the surgery, increasing its efficiency, and reducing delays. This then improves the surgical outcome. In one example, it may be determined that the current surgical step is a major vessel management step in a lobectomy and the subsequent task is to cut a fissure.

[0363] 3. The method according to Embodiment 1 or Embodiment 2, further including determining a subsequent target system task based on annotated surgical data.

[0364] Advantageously, determining subsequent tasks based on annotated data enables data needs to be determined without input from a medical expert, thereby automating the surgery, increasing its efficiency, and reducing delays. This then improves the surgical outcome.

[0365] 4. The method according to any one of Embodiments 1 to 3, wherein determining data needs includes determining data needs based on at least one of prioritization data, system usage data, or hierarchical segmentation information.

[0366] 5. The method according to any one of Embodiments 1 to 4, further including determining a risk level based on the obtained surgical data, and wherein determining data needs includes determining data needs based on the determined risk level.

[0367] 6. The subset of annotated surgical data is the first subset of annotated surgical data, determining a classification associated with a second subset of the annotated surgical data; The method according to any one of Embodiments 1 to 5, further comprising performing an edit on the second subset of the annotated surgical data based at least on the classification associated with the second subset of the annotated surgical data.

[0368] 7. The subset of the annotated surgical data is the first subset of the annotated surgical data, determining a classification associated with a second subset of the annotated surgical data, wherein the first subset of the data at least partially includes the second subset of the data and the classification is associated with private data or outlier data; The method according to any one of Embodiments 1 to 6, further comprising performing an edit on the second subset of the surgical data based at least on the classification associated with the second subset of the annotated surgical data.

[0369] 8. The subset of the annotated surgical data is the first subset of the annotated surgical data, determining whether the second subset of the annotated surgical data is private data; determining whether the target system is outside the patient data privacy protection boundary; The method according to any one of Embodiments 1 to 7, further comprising performing an edit on the second subset of the surgical data based on the condition that the second subset of the annotated surgical data is private data and the target system is outside the patient data privacy protection boundary.

[0370] 9. The method according to Embodiment 8, further comprising transmitting the second subset of the annotated surgical data to local storage, wherein the second subset of the annotated surgical data is stored before the edit is performed.

[0371] 10. A method according to any one of embodiments 1 to 9, wherein a plurality of surgical systems includes one or more of a surgical instrument, a monitoring system, a surgical procedure sensor, or a surgical device.

[0372] 11. A method according to any one of embodiments 1 to 10, wherein the target system is a surgical instrument, a subsequent target system task is associated with using the surgical instrument, and the data needs include data associated with using the surgical instrument.

[0373] Advantageously, automating the delivery of relevant data for subsequent surgical tasks performed by the instrument to the surgical instrument reduces delays in surgery.

[0374] 12. A method according to any one of embodiments 1 to 11, wherein the target system is a facility system, a subsequent target system task is facility maintenance, and the data needs include one or more of a surgical instrument used, a surgical consumable used, a surgical procedure plan, or a surgical schedule.

[0375] 13. A surgical computer system comprising a processor configured to execute the method according to any one of embodiments 1 to 12.

[0376] 14. A computer program comprising instructions that, when executed by a computer, cause the computer to execute the method according to any one of embodiments 1 to 12.

[0377] 15. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to execute the method according to any one of embodiments 1 to 12.

[0378] The following is a numbered list of embodiments that may or may not be claimed. 1. A computer-implemented method, comprising Obtaining surgical context data related to a surgical operation, Obtaining a surgical video of a surgical operation including one or more surgical video frames, Based on the obtained surgical context data and using image processing, identifying one or more elements within a first surgical video frame, wherein each of the one or more elements includes respective groups of pixels, Based on the surgical context data and the one or more elements within the first surgical video frame, generating annotation data for the first surgical video frame, the annotation data including respective element annotation data associated with each of the identified one or more elements,

[0379] Advantageously, using the surgical context data to identify elements within the video frame and generate annotation data associated with those elements flags relevant elements of the surgical video with data related to the surgical context. This helps medical providers more quickly / easily identify important features within the video and understand their context simultaneously. This then enables more rapid decision-making and better surgical outcomes.

[0380] The surgical context data related to a surgical operation may indicate a surgical operation, a surgical event, a surgical step, a surgical phase, a surgical task, a surgical device, a complication, and / or the like.

[0381] 2. The method according to Embodiment 1, further including inserting annotation data for the first surgical video frame, wherein each respective element annotation data is attached to a respective group of pixels.

[0382] Advantageously, inserting annotation data into the video itself, and in particular attaching the annotation data directly to groups of pixels of the elements, enables the healthcare provider to more quickly / easily identify and understand the context of important features within the video, as the relevant elements are flagged / labelled at their positions within the video image.

[0383] 3. Identifying, using image processing, a plurality of sub-elements associated with a first element identified within a first surgical video frame based on surgical context data, each sub-element including a respective sub-group of pixels, and the element annotation data including respective sub-element annotation data associated with the plurality of sub-elements, and further attaching each respective sub-element annotation data to the respective sub-group of pixels, the method according to embodiment 1 or embodiment 2.

[0384] 4. The annotation data includes one or more of element identification information, element grouping information, element sub-grouping information, element type information, element description information, element condition information, surgical step information, surgical task information, surgical event information, surgical instrument information, or surgical device information, the method according to any of embodiments 1 to 3.

[0385] 5. Refining surgical context data based on one or more elements identified within a first surgical video frame, identifying, using image processing, one or more elements within a second surgical video frame based on the refined surgical context data, and further generating annotation data for the second surgical video frame based on the refined surgical context data and the one or more elements within the second surgical video frame, the method according to any of embodiments 1 to 4.

[0386] Advantageously, refining the surgical context data improves the identification of relevant elements as well as the content and relevance of the annotation data.

[0387] 6. Using image processing to identify one or more elements within a second surgical video frame based on the acquired surgical context data, Determining tracking data regarding the second surgical video frame based on one or more elements within a first surgical video frame and one or more elements within the second video frame, wherein the tracking data is associated with one or more of element behavior, element movement, or outcome information, Generating annotation data for the second surgical video frame based on the surgical context data and one or more elements within the second surgica...

Claims

1. A computer-implemented method, comprising: obtaining surgical data associated with a surgical operation from a plurality of surgical systems; annotating the surgical data using surgical context data regarding the surgical operation; determining data needs for subsequent target surgical system tasks associated with a target surgical system, at least partially based on the annotated surgical data; generating a data package associated with the data needs, the data package including a subset of the annotated surgical data; and transmitting the data package to the target surgical system.

2. The method according to claim 1, further comprising: determining the surgical context data based on the surgical data, the surgical context data including an indication of a current surgical step in the surgical operation; and determining an indication of the subsequent target system task associated with the target surgical system.

3. The method according to claim 1, further comprising determining the subsequent target system task based on the annotated surgical data.

4. The method according to claim 1, wherein determining the data needs includes determining the data needs based on at least one of prioritization data, system usage data, or hierarchical segmentation information.

5. The method according to claim 1, further comprising determining a risk level based on the obtained surgical data, and wherein determining the data needs includes determining the data needs based on the determined risk level.

6. The subset of the annotated surgical data is a first subset of the annotated surgical data, and the method further comprises: determining a classification associated with a second subset of the annotated surgical data; and performing an edit on the second subset of the annotated surgical data, at least based on the classification associated with the second subset of the annotated surgical data.

7. The subset of the annotated surgical data is a first subset of the annotated surgical data, and the method determining a classification associated with a second subset of the annotated surgical data, wherein the first subset of the data at least partially includes the second subset of the data, and the classification is associated with private data or outlier data, further comprising performing an edit on the second subset of the surgical data based at least on the classification associated with the second subset of the annotated surgical data. The method according to claim 1.

8. The subset of the annotated surgical data is a first subset of the annotated surgical data, and the method determining whether the second subset of the annotated surgical data is private data, determining whether the target system is outside a patient data privacy protection boundary, performing an edit on the second subset of the surgical data based on the condition that the second subset of the annotated surgical data is private data and the target system is outside the patient data privacy protection boundary. The method according to claim 1.

9. The method according to claim 8, further comprising transmitting the second subset of the annotated surgical data to local storage, wherein the second subset of the annotated surgical data is stored before the edit is performed.

10. The plurality of surgical systems includes one or more of a surgical instrument, a monitoring system, a surgical sensor, or a surgical device. The method according to claim 1.

11. The target system is a surgical instrument, the subsequent target system task is associated with the use of the surgical instrument, and the data needs include data associated with the use of the surgical instrument. The method according to claim 1.

12. The target system is a facility system, the subsequent target system task is facility maintenance, and the data needs include one or more of a surgical instrument used, a surgical consumable used, a surgical plan, or a surgical schedule. The method according to claim 1.

13. A surgical computer system comprising a processor configured to execute the method according to any one of claims 1 to 12.

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to execute the method according to any one of claims 1 to 12.

15. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to execute the method according to any one of claims 1 to 12.

16. A computer-implemented method, comprising obtaining surgical context data related to a surgical operation, obtaining a surgical video of the surgical operation including one or more surgical video frames, identifying, based on the obtained surgical context data and using image processing, one or more elements within a first surgical video frame, each of the one or more elements including a respective group of pixels, generating annotation data for the first surgical video frame based on the surgical context data and the one or more elements within the first surgical video frame, the annotation data including respective element annotation data associated with each of the identified one or more elements.

17. The method according to claim 16, further comprising inserting the annotation data for the first surgical video frame into the surgical video, wherein the respective element annotation data is attached to the respective group of pixels.

18. identifying, using image processing and based on the surgical context data, a plurality of sub-elements associated with a first element identified within the first surgical video frame, each sub-element including a respective sub-group of pixels, and the element annotation data including respective sub-element annotation data associated with the plurality of sub-elements, and attaching the respective sub-element annotation data to the respective sub-group of pixels.

19. The method according to claim 16, wherein the annotation data includes one or more of element identification information, element grouping information, element subgrouping information, element type information, element description information, element condition information, surgical step information, surgical task information, surgical event information, surgical instrument information, or surgical device information.

20. refining the surgical context data based on the one or more identified elements in the first surgical video frame; using image processing to identify one or more elements in a second surgical video frame based on the refined surgical context data; generating annotation data for the second surgical video frame based on the refined surgical context data and the one or more elements in the second surgical video frame, the method according to claim 16, further comprising:

21. using image processing to identify one or more elements in a second surgical video frame based on the obtained surgical context data; determining tracking data regarding the second surgical video frame based on the one or more elements in the first surgical video frame and the one or more elements in the second video frame, wherein the tracking data is associated with one or more of element behavior, element movement, or result information; generating annotation data for the second surgical video frame based on the surgical context data and the one or more elements in the second surgical video frame, the annotation data including the tracking data regarding the second surgical video frame, the method according to claim 16, further comprising:

22. obtaining predicted tracking information associated with a surgical context indicated by the surgical context data; validating the one or more identified elements in the second surgical video frame based on the predicted tracking information, the annotation data for the second surgical video frame including an indication of whether the one or more identified elements in the second surgical video frame have been validated, the method according to claim 16, further comprising:

23. The method according to claim 22, wherein the predicted tracking information is obtained based on a surgical plan.

24. Using the surgical context data and the identified one or more elements within the first surgical video frame, determining surgical task information associated with the first surgical video frame; Generating control parameters for the surgical procedure based on the determined surgical task information; Transmitting a control signal to a surgical control system, the control signal including an instruction indicating the determined control parameters to the surgical control system, the method of claim 16 further comprising.

25. Using the surgical context data and the identified one or more elements within the first surgical video frame, determining surgical task information associated with the first surgical video frame; Using the surgical context data, the surgical task information, and the identified one or more elements within the first surgical video frame, determining complication information associated with the determined surgical task; Generating control parameters for the surgical procedure based on the determined complication information; Transmitting a control signal to a surgical control system, the control signal including an instruction indicating the determined control parameters to the surgical control system, the method of claim 16 further comprising.

26. Using the surgical context data and the identified one or more elements within the first surgical video frame, determining surgical task information associated with the first surgical video frame; Using the surgical context data, the surgical task information, and the identified one or more elements within the first surgical video frame, determining complication information associated with the determined surgical task; Using the surgical context data, the surgical task information, and the complication information, determining outcome information associated with the surgical task; Determining to change a parameter associated with the surgical procedure based on the surgical task information, the complication information, and the outcome information; Generating control parameters for the surgical procedure based on the determination to change the parameter; Transmitting to the surgical control system a control signal including an instruction indicating the determined control parameters, the method of claim 16 further comprising.

27. A computing system comprising a processor configured to execute the method according to any one of claims 16 to 26.

28. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to execute the method according to any one of claims 16 to 26.

29. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to execute the method according to any one of claims 16 to 26.

30. A computer-implemented method for generating a patient-specific surgical plan for a patient-specific surgery, comprising: obtaining patient-specific pre-surgical data; obtaining a surgical template including surgical steps; determining, based at least in part on the patient-specific pre-surgical data, one or more surgical task options for a first surgical step and a second surgical step; determining, for each of the one or more surgical task options, a respective characterization based on the patient-specific pre-surgical data, each respective characterization including at least one of a predicted result of the respective surgical task, a predicted result probability of the respective surgical task, a complication of the respective surgical task, a risk level of the respective surgical task, and an efficiency evaluation of the respective surgical task; generating an input patient-specific surgical plan including the one or more surgical task options for the first surgical step and the second surgical step and the respective characterizations associated with the one or more surgical task options.

31. selecting, based on the respective characterizations associated with the respective surgical task options, a set of surgical tasks from the surgical task options for the first surgical step; generating a pre-selected input surgical plan based on the selected set of surgical tasks for the first surgical step.

32. selecting a set of surgical tasks from the surgical task options for the first surgical step based on the respective characterizations associated with the respective surgical task options; generating a preselected input surgical plan based on the selected set of surgical tasks for the first surgical step, the preselected input surgical plan including one or more alternative surgical task options from the one or more surgical task options for the first surgical step and the respective characterizations associated with the one or more alternative surgical task options; which further comprises, the method of claim 30.

33. The method of claim 30, further comprising obtaining facility information, wherein determining the one or more surgical task options for the first surgical step is further determined based on the facility information.

34. The method of claim 33, wherein the facility information includes availability information associated with one or more of surgical devices, surgical instruments, healthcare providers, or facility rooms.

35. The facility information includes availability information associated with one or more of surgical devices or surgical instruments, the availability information indicating whether the surgical device or surgical instrument is selected for an independent surgical procedure, and when the surgical device or surgical instrument is selected for an independent surgical procedure, the availability information further indicating the scheduling time window associated with the independent surgical procedure, the method comprising: The method of claim 33, further comprising determining whether there is a conflict between the patient-specific surgical procedure and the independent surgical procedure, the determination including determining whether the surgical device or the surgical instrument can be used for both the patient-specific surgical procedure and the independent surgical procedure.

36. The method of claim 30, wherein the one or more surgical task options include a plurality of access points to the patient, the characterization associated with the first access point being a first result success probability, and the characterization associated with the second access point being a second result success probability.

37. The one or more surgical task options include a plurality of instrument positionings, the characterization associated with the first instrument positioning is the first result success probability, and the characterization associated with the second instrument positioning is the second result success probability, the method according to claim 30.

38. The method further includes determining that a subset of data from the patient-specific surgical data is incomplete, and when the characterization of the first surgical task option is determined based at least in part on the subset of data, tagging the input patient-specific surgical plan with an indication that the characterization was determined using incomplete data, the method according to claim 30.

39. The indication further instructs the user to input additional data corresponding to the incomplete data, and the method obtaining the additional data corresponding to the incomplete data; and updating the first surgical task option based on the additional data, the method according to claim 38.

40. The respective characterization of each surgical task option is determined based on past surgical results, the method according to claim 30.

41. The method further includes instructing one or more of a surgical instrument, a surgical system, and a surgical robot to autonomously perform one or more of the surgical tasks of the surgical plan, the method according to claim 30.

42. The method further includes determining one or more operating parameters for one or more of a surgical instrument, a surgical system, and a surgical robot to be used during the surgical procedure based on the input patient-specific surgical plan, the method according to claim 30.

43. A surgical system comprising a processor configured to execute the method according to any one of claims 30 to 42.

44. A computer program comprising instructions that, when executed by a computer, cause the computer to execute the method according to any one of claims 30 to 42.

45. A computer-readable medium that, when executed by a computer, includes instructions that cause the computer to execute the method according to any one of claims 30 to 42.

Images