System and method for assisting in planning and performing a procedure

Abstract

Disclosed is a system for assisting in guiding and performing a procedure on a subject. The subject may be any appropriate subject such as inanimate object and / or an animate object. An imaging system may be used to image the subject. A system may include various manipulable or movable members, such as robotic systems. This system may provide feedback regarding the selected procedure that may be real time and / or for future planning.

Description

A001 1698W001SYSTEM AND METHOD FOR ASSISTING IN PLANNING AND PERFORMING A PROCEDURECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Applications Serial Nos. 63 / 683,901 , 63 / 683,819, 63 / 683,878, 63 / 683,912 and 63 / 683,904, all filed 16 August 2024, the entire contents of which are incorporated herein by reference.FIELD

[0002] The subject disclosure is related generally to a surgical planning and execution system, and particularly to a surgical system including procedure data utilization and planning.BACKGROUND

[0003] This section provides background information related to the present disclosure which is not necessarily prior art.

[0004] An instrument can be navigated relative to a subject for performing various procedures. For example, the subject can include a patient on which a surgical procedure is being performed. During a surgical procedure, an instrument can be tracked in an object or subject space. In various embodiments, the subject space can be a patient space defined by a patient. The location of the instrument that is tracked can be displayed on a display device relative to an image of the patient.

[0005] The position of the patient can be determined with a tracking system. Generally, a patient is registered to the image, via tracking an instrument relative to the patient to generate a translation map between the subject or object space (e.g., patientA001 1698W001 space) and the image space. This may occur by identifying one or more points in the subject space and correlating points (e.g., identical points) in the image space.

[0006] After registration, the position of the instrument can be appropriately displayed on the display device while tracking the instrument. The position of the instrument relative to the subject can be displayed as a graphical representation, sometimes referred to as an icon on the display device.SUMMARY

[0007] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0008] A surgical system or suite may include one or more tools, which may include instruments, robotic systems, guides, etc. The suite may further include other systems that may be used to collect the data during a procedure. Further a user, such as the surgeon, may collect or annotate various portions of a procedure to provide further information or data regarding a procedure. Tools or systems in the surgical suite may include an imaging system, instruments for performing a procedure such as a drill, an inserter, an implant, or the like. Other systems may also include a robotic system which may include an end effector to hold a guide, instrument, for performing a procedure, or other appropriate portion for a procedure. In addition, various navigation or tracking systems may also be used to assist in determining or tracking a pose of an instrument during a procedure in a space or volume, such as in a patient space.

[0009] The surgical suite and associated systems, therefore, may be used to provide or collect data during the procedure. The data collected during the procedure may be stored for various purposes. The data collected during a procedure with the surgicalA001 1698W001 suite system and with various other inputs of data, such as from a surgeon, may be stored for analysis. The analysis may include a determination of an order of the procedure, position of various instruments during the procedure, time of a procedure, force applied with an instrument, or the like. The data may be compared to other data or analysis, such as a follow up.

[0010] A procedure may be performed on a subject. After the procedure, follow up data may be acquired from the subject and / or relative to the subject. For example, a range of motion of a portion of the subject, a recovery rate of the subject, a maintaining or increasing of a force applied by the subject relative to the procedure region, or the like. The follow up data may also be acquired and stored for various purposes, such as further analysis. The follow up data may also be sequenced, analyzed, and parameterized for further analysis.

[0011] The procedure data and the follow up data, either pre- or post-analysis, may be compared. The procedure data and follow up data may be compared to assist in determining an efficacy of a selected procedure, efficiency of the procedure, and the like. Further, a plurality of procedure data and follow up data may be compared to each other and as a group to assist in determining or evaluating clusters of the data, trends, or the like. Therefore, the procedure data and follow up data may be analyzed and used to determine trends, compare procedure actions and data between procedures and relative to follow up data. Trends based upon the follow up data regarding selected or desired outcomes may be compared or related to selected procedure data. The data may also be used to assist or create efficacy in a future procedure such as a selected outcome, procedure duration, motions during a procedure, procedure task order, tool selection,A001 1698W001 procedure outcomes, implant placement, etc. Further, the data may be used to assist in procedure, such as future, procedure such as speed, cutting speed, etc.

[0012] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.DRAWINGS

[0013] The accompanying drawings are incorporated into and form a part of the specification to illustrate several examples of the present disclosure. These drawings, together with the description, explain the principles of the disclosure. The drawings simply illustrate preferred and alternative examples of how the disclosure can be made and used and are not to be construed as limiting the disclosure to only the illustrated and described examples. Further features and advantages will become apparent from the following, more detailed, description of the various aspects, embodiments, and configurations of the disclosure, as illustrated by the drawings referenced below.

[0014] Fig. 1 is diagrammatic view illustrating an overview of a robotic system and a navigation system, according to various embodiments;

[0015] Fig. 2 is a detail schematic view illustrating an overview of a robotic system and a navigation system, according to various embodiments;

[0016] Fig. 3 is a schematic view of a subject with possible placement of instruments and implants relative thereto;

[0017] Fig. 4 is an exemplary display of an image and indications regarding a current procedure or analysis of a procedure, according to various embodiments;A001 1698W001

[0018] Fig. 5 is a flowchart of a process, according to various embodiments;

[0019] Fig. 6 is an exemplary display of an image and indications regarding a current procedure or analysis of a procedure, according to various embodiments;

[0020] Fig. 7A is an exemplary display of an image and indications regarding a current procedure or analysis of a procedure, according to various embodiments;

[0021] Fig. 7B is an exemplary display of an image and indications regarding a current procedure or analysis of a procedure, according to various embodiments;

[0022] Fig. 8 is a flowchart of a process, according to various embodiments;

[0023] Fig. 9A and Fig. 9B illustrate a schematic view of an image acquisition system, according to various embodiments;

[0024] Fig. 10 is an exemplary display of an image and indications regarding a current procedure or analysis of a procedure, according to various embodiments;

[0025] Fig. 11 is a flowchart of a process, according to various embodiments;

[0026] Fig. 12 is a schematic view of an instrument with selected sensors and held or controlled by a user or robotic system, according to various embodiments;

[0027] Fig. 13A illustrates a schematic view of an image acquisition system, according to various embodiments;

[0028] Fig. 13B is an exemplary display of an image and indications regarding a current procedure or analysis of a procedure, according to various embodiments;

[0029] Fig. 14 is a flowchart of a process, according to various embodiments; and

[0030] Fig. 15 is a flowchart of a process, according to various embodiments.

[0031] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.A001 1698W001

[0032] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of examples, aspects, and features illustrated.

[0033] In some instances, the apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the of various implementations, examples, aspects, and features so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.DETAILED DESCRIPTION

[0034] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example or embodiment, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and / or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different embodiments of the present disclosure). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and / or a medical device.A001 1698W001

[0035] Before any embodiments of the disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Further, the present disclosure may use examples to illustrate one or more aspects thereof. Unless explicitly stated otherwise, the use or listing of one or more examples (which may be denoted by “for example,” “by way of example,” “e.g.,” “such as,” or similar language) is not intended to and does not limit the scope of the present disclosure.

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0037] The subject disclosure is directed to an exemplary embodiment of a surgical procedure on a subject, such as a human patient. It is understood, however, that the system and methods described herein are merely exemplary and not intended to limit the scope of the claims included herein. In various embodiments, it is understood, that the systems and methods may be incorporated into and / or used on non-animate objects. The systems may be used to, for example, image and register coordinate systems between two systems for use on manufacturing systems, maintenance systems, and the like. For example, automotive assembly may use one or more robotic systems including individual coordinate systems that may be registered together for coordinated or concerted actions.A001 1698W001Accordingly, the exemplary illustration of a surgical procedure herein is not intended to limit the scope of the appended claims.

[0038] Various members or portions thereof may also be tracked relative to the subject. For example, a tracking system may be incorporated into a navigation system to allow tracking and navigation of one or more instruments (which may be the members) that may be tracked relative to the subject, each other, other tracked objects, or combinations thereof. Thus, the subject may be tracked as may any selected member.

[0039] The navigation system may include one or more tracking systems that track various portions, such as tracking devices, associated with instruments. The tracking system may include a localizer that is configured to determine the position of the tracking device in a navigation system coordinate system. Determination of the navigation system coordinate system may include those described at various references including U.S. Pat. No. 8,737,708; U.S. Pat. No. 9,737,235; U.S. Pat. No. 8,503,745; U.S. Pat. No. 8,175,681 ; and U.S. Pat. No. 11 ,135,025, all incorporated herein by reference. In particular, a localizer may be able to track an object within a volume relative to the subject. The navigation volume, in which a device may be tracked, may include or be referred to as the navigation coordinate system or navigation space. A determination or correlation between two coordinate systems may allow for or also be referred to as a registration between two coordinate systems.

[0040] In various embodiments, the first coordinate system, which may be a robotic coordinate system, may be registered to a second coordinate system, which may be a navigation coordinate system. Accordingly, coordinates in one coordinate system may then be transformed to a different or second coordinate system due to a registration. Registration may allow for the use of two coordinate systems and / or the switchingA001 1698W001 between two coordinate systems. For example, during a procedure, a first coordinate system may be used for a first portion or a selected portion of a procedure and a second coordinate system may be used during a second portion of a procedure. Further, two coordinate systems may be used to perform or track a single portion of a procedure, such as for verification and / or collection of additional information.

[0041] Furthermore, images may be acquired of selected portions of a subject. The images may be displayed for viewing by a user, such as a surgeon. The images may have superimposed on a portion of the image, a graphical representation of a tracked portion or member, such as an instrument. According to various embodiments, the graphical representation may be superimposed as any appropriate graphic (including an image or a rendered model of the member) on the image at an appropriate position due to registration of an image space (also referred to as an image coordinate system) to a subject space. The image may be an appropriate image of the subject, such as a MRI, x- ray image, CT image, ultrasound, or combinations thereof and / or generated models based thereon. A method to register a subject space defined by a subject to an image space may include those disclosed in U.S. Pat. Nos. U.S. Pat. No. 8,737,708; U.S. Pat. No. 9,737,235; U.S. Pat. No. 8,503,745; and U.S. Pat. No. 8,175,681 ; all incorporated herein by reference.

[0042] During a selected procedure, the first coordinate system may be registered to the subject space or subject coordinate system due to a selected procedure, such as imaging of the subject. In various embodiments, the first coordinate system may be registered to the subject by imaging the subject with a fiducial portion that is fixed relative to the first member or system, such as a robotic system or other instrument. The known position of the fiducial relative to any portion, such as the robotic system or the subject,A001 1698W001 may be used to register the subject space relative to any coordinate system in which the fiducial may be determined (e.g., by imaging or detecting (e.g., touching)). A registration of a second coordinate system may allow for tracking of additional elements not fixed to a first portion, such as a robot that has a known coordinate system.

[0043] The tracking of an instrument during a procedure, such as a surgical or operative procedure, allows for navigation of a procedure. The navigation may be used to determine a pose of one or more portions, such as an instrument. The pose may include any number of degrees of freedom, such as a three-dimensional location (e.g., x, y, z) and an orientation (e.g., yaw, pitch, and roll). Therefore, the pose of any system or portion, as discussed herein, may relate to its location and orientation in a selected space or volume.

[0044] When image data is used to define an image space, it can be correlated or registered to a physical space defined by a subject, such as a patient. According to various embodiments, therefore, the patient defines a patient space in which an instrument can be tracked and navigated. The image space defined by the image data can be registered to the patient space defined by the patient. The registration can occur with the use of fiducials that can be identified in the image data and in the patient space.

[0045] Fig. 1 and Fig. 2 are diagrammatic views illustrating an overview of a procedure room or arena. In various embodiments, the procedure room may include a surgical suite or procedure suite 18. The procedure suite 18 may include one or more systems or member, as discussed herein. The procedure arena may be in any appropriate location, such as a hospital, manufacturing facility, etc. Further, the procedure suite may include one or more systems that are simple or complex systems or combinations thereof. Systems may include an imaging system, a unitary implant, etc.A001 1698W001

[0046] The procedure suite may include a robotic system 20 and a navigation system 26 that can be used for various procedures. The robotic system 20 may include a Mazor X™ robotic guidance system, sold by Medtronic, Inc. The navigation system 26 may be or include portions of various navigation systems, such as those noted herein including the StealthStation® 8 surgical navigation system sold by Medtronic Navigation, Inc. having a place of business in Colorado, USA.

[0047] The robotic system 20 may be used to assist in guiding a selected instrument, such as drills, screws, etc. relative to a subject 30. Additionally or alternatively, the robotic system 20 may hold and / or move various instruments, such as an imaging system that may be an ultrasound (US) probe 33. The robotic system 20 may include a mount 34 that fixes a portion, such as a robotic base 38, relative to the subject 30. The robotic system 20 may include one or more arms 40 that are moveable or pivotable relative to the subject 30, such as including an end effector 44. The end effector may be any appropriate portion, such as a tube, guide, or passage member. Affixed to and / or in place of the end effector may be the imaging system that may be the US probe 33.

[0048] According to various embodiments, the US probe 33 may be moved relative to a subject, such as a by a user and / or with a robotic system. The robotic system may include an appropriate robotic system, such as a Mazor X™ Robotic Guidance System, sold by Mazor Robotics Ltd. having a place of business in Israel and / or Medtronic, Inc. having a place of business in Minnesota, USA and / or as disclosed in U.S. Pat. No. 11 ,135,025, incorporated herein by reference. The US probe may be moved to achieve acquisition of selected image data.

[0049] The end effector 44 may be moved relative to the base 38 with one or more motors. The position of the end effector 44 may be known or determined relative to theA001 1698W001 base 38 with one or more encoders at one or more joints, such as a wrist joint 48 and / or an elbow joint 52 of the robotic system 20. A robotic processor module 53 may be used to control (e.g., execute instructions) to move and determine a pose of the end effector, such as relative to the base 34. As discussed above, the robotic system 20 including the various portions may be operated to move each portion relative to the base 34. The pose of the base 34 may be known in a coordinate system, such as the patient space of the patient 30 and / or the image coordinate system due to a registration, as discussed above and exemplary disclosed in U.S. Pat. No. 11 ,135,025, incorporated herein by reference.

[0050] The navigation system 26 can be used to track the pose of one or more tracking devices, tracking devices may include a robot tracking device 54, a subject tracking device 58, an imaging system tracking device 62, a tool tracking device 66, and / or an US probe tracking device 81 . All or one of the tracking devices may be generally referred to as a tracking device herein. US probe 33 may be used to acquire US image data of the subject 30 as may an imaging system 80, as discussed herein.

[0051] A member 68, which may include or be referred to as a tool, instrument, implant, or moveable member. The member 68 may be any appropriate tool such as a drill, forceps, or other tool operated by a user 72. The member 68 may also include an implant, such as a spinal implant or orthopedic implant. It should further be noted that the navigation system 26 may be used to navigate any type of instrument, implant, or delivery system, including: guide wires, arthroscopic systems, orthopedic implants, spinal implants, deep brain stimulation (DBS) probes, etc. Moreover, the instruments may be used to navigate or map any region of the body. The navigation system 26 and the various instruments may be used in any appropriate procedure, such as one that is generally minimally invasive or an open procedure.A001 1698W001

[0052] The imaging system or device 80 may be an imaging system that may be used to acquire pre-, intra-, or post-operative or real-time image data of a subject, such as the subject 30. It will be understood, however, that any appropriate subject can be imaged and any appropriate procedure may be performed relative to the subject. In the example shown, the imaging device 80 comprises an O-arm® imaging device sold by Medtronic Navigation, Inc. having a place of business in Louisville, Colorado, USA. The imaging device 80 may have a generally annular gantry housing 82 in which an image capturing portion is moveably placed. The imaging device 80 can include those disclosed in U.S. Pat. Nos. 7,188,998; 7,108,421 ; 7,106,825; 7,001 ,045; and 6,940,941 ; all of which are incorporated herein by reference, or any appropriate portions thereof. It is appreciated that the imaging device 80 may additionally or alternatively be a fluoroscope such as biplane fluoroscopic systems, ceiling mounted fluoroscopic systems, cath-lab fluoroscopic systems, fixed C-arm fluoroscopic systems, isocentric C-arm fluoroscopic systems, 3D fluoroscopic systems, etc. Other appropriate imaging devices can also include MRI, CT, ultrasound, etc.

[0053] The position of the imaging system 33, 80, and / or portions therein such as the image capturing portion, can be precisely known relative to any other portion of the imaging device 33, 80. The imaging device 33, 80, according to various embodiments, can know and / or recall precise coordinates relative to a fixed or selected coordinate system. For example, the robotic system 20 may know or determine its position and position the US probe 33 at a selected pose. Similarly, the imaging system 80 may also position the imaging portions at a selected pose. This can allow the imaging system 80 to know its position relative to the patient 30 or other references. In addition, as discussed herein, the precise knowledge of the position of the image capturing portion can be usedA001 1698W001 in conjunction with a tracking system to determine the position of the image capturing portion and the image data relative to the tracked subject, such as the patient 30.

[0054] Herein, reference to the imaging system 33 may refer to any appropriate imaging system, unless stated otherwise. Thus, the US probe 33 as the imaging system is merely exemplary regarding the subject disclosure. As one skilled in the art will understanding, generally the US probe 33 may emit a US wave in a plane or any appropriate shape and receive an echo relative to any portions engaged by the wave. An echo from the transmitted signal in the plane may be received at the receiver portion. Via the transmitted signal in the plane an acquisition of image data in a field of view occurs to generate images, also referred to as sonograms when images are generated based on ultrasound data. The pose (e.g., distance from a selected portion of the US probe 33 and / or the tracking device 81) of the US wave and / or echo origin may be determined or predetermined and saved for recall with a calibration process and / or jig, such as that disclosed in U.S. Pat. Nos. 7,831 ,082; 8,320,653; and 9,138,204, all incorporated herein by reference.

[0055] The imaging device 80 can be tracked with a tracking device 62. Also, the tracking device 81 can be associated directly with the US probe 33. The US probe 33 may, therefore, be directly tracked with a navigation system as discussed herein. In addition or alternatively, the US probe 33 may be positioned and tracked with the robotic system 20. Regardless, image data defining an image space acquired of the patient 30 can, according to various embodiments, be registered (e.g., manually, inherently, or automatically) relative to an object space. The object space can be the space defined by a patient 30 in the navigation system 26.A001 1698W001

[0056] The patient 30 can also be tracked as the patient moves with a patient tracking device, DRF, or tracker 58. Alternatively, or in addition thereto, the patient 30 may be fixed within navigation space defined by the navigation system 26 to allow for registration. As discussed further herein, registration of the image space to the patient space or subject space allows for navigation of the instrument 68 with the image data. When navigating the instrument 68, a position of the instrument 68 can be illustrated relative to image data acquired of the patient 30 on a display device 84. An additional and / or alternative display device 84’ may also be present to display an image. Various tracking systems, such as one including an optical localizer 88 or an electromagnetic (EM) localizer 92 can be used to track the instrument 68.

[0057] Thus, the pose of the patient 30, the imaging system 33, 80, and other members may be determined by the navigation system 26. In selected spaces or volumes, the pose of each may be determined. Thus, in the same spaces the relative pose of each may be determined. Further, two or more spaces may be registered together to allow relative pose determination of members in two or more different spaces.

[0058] More than one tracking system can be used to track the instrument 68 in the navigation system 26. According to various embodiments, these tracking systems can include an electromagnetic tracking (EM) system having the EM localizer 94, an optical tracking system having the optical localizer 88 and / or other appropriate tracking systems, not illustrated, such as an ultrasound tracking system, or other appropriate tracking systems. One or more of the tracking systems can be used to track selected tracking devices, as discussed herein, sequentially or simultaneously. It will be understood, unless discussed otherwise, that a tracking device can be a portion trackable with a selectedA001 1698W001 tracking system. A tracking device need not refer to the entire member or structure to which the tracking device is affixed or associated.

[0059] Image data acquired from the imaging system 33, or any appropriate imaging system, can be acquired at and / or forwarded from an image device controller 96, that may include a processor module, to a navigation computer and / or processor system 102 that can be a part of a controller or work station 98 having the display 84 and a user interface 106. The processor system 102 may be a processor module, as discussed herein, including integral memory or a communication system to access external memory for executing instructions and / or operated as a specific integrated circuit (e.g., ASIC) It will also be understood that the image data is not necessarily first retained in the controller 96, but may also be directly transmitted to the work station 98. The work station 98 can provide facilities for displaying the image data as an image 108 on the display 84, saving, digitally manipulating, or printing a hard copy image of the received image data. The user interface 106, which may be a keyboard, mouse, touch pen, touch screen or other suitable device, allows the user 72 to provide inputs to control the imaging device 80, via the image device controller 96, or adjust the display settings of the display 84. The work station 98 may also direct the image device controller 96 to adjust the image capturing portion of the imaging device 80 to obtain various two-dimensional images along different planes in order to generate representative two-dimensional and three-dimensional image data.

[0060] The user input 106 may be used by the user 72 to provide or input data during the procedure. The data provided during the procedure may include the considerations of the user 72, perceptions of the user 72, parameters considered by or other considerations of the user 72, or the like. The user input 106 allows the user 72 to provide input that may be in addition to that recorded by the various systems of theA001 1698W001 surgical suite systems and / or to augment their data inputs. Thus, the user input 106 may allow for additional data collection in addition to measurements and sensors provided in the surgical suite.

[0061] With continuing reference to Fig. 1 , the navigation system 26 can further include any one or more tracking system, such as the tracking system including either or both of the electromagnetic (EM) localizer 94 and / or the optical localizer 88. The tracking systems may include a controller and interface portion 110. The controller 110 can be connected to the processor portion 102, which can include a processor included within a computer. The EM tracking system may include the STEALTHSTATION® AXIEM™ Navigation System, sold by Medtronic Navigation, Inc. having a place of business in Louisville, Colorado; or can be the EM tracking system described in U.S. Patent No. 7,751 ,865, entitled "METHOD AND APPARATUS FOR SURGICAL NAVIGATION", issued Jul. 6, 2010; U.S. Patent No. 5,913,820, entitled “Position Location System,” issued June 22, 1999; and U.S. Patent No. 5,592,939, entitled “Method and System for Navigating a Catheter Probe,” issued January 14, 1997; all of which are herein incorporated by reference. It will be understood that the navigation system 26 may also be or include any appropriate tracking system, including a STEALTHSTATION® TREON® or S7™ tracking or surgical navigation systems having an optical localizer, which may be used as the optical localizer 88, and sold by Medtronic Navigation, Inc. of Louisville, Colorado. Other tracking systems include an acoustic, radiation, radar, etc. The tracking systems can be used according to generally known or described techniques in the above incorporated references. Details will not be included herein except when to clarify selected operation of the subject disclosure.A001 1698W001

[0062] Wired or physical connections can interconnect the tracking systems and / or tracking devices, imaging device 80, etc. For example, various portions, such as the instrument 68 may employ a wireless communications channel, such as that disclosed in U.S. Patent No. 6,474,341 , entitled “Surgical Communication Power System,” issued November 5, 2002, herein incorporated by reference, as opposed to being coupled physically to the controller 110. Also, the tracking devices 62, 66, 54 can generate a field and / or signal that is sensed by the localizer(s) 88, 94. The communication of the surgical suite 18 may, therefore, allow for communication between any of the selected portions and / or other systems. For example, each of the systems of the surgical suite may communicate with a single processor system 102 and / or a cloud or network system 112. The communication may be direct and / or through other communication systems.

[0063] Various portions of the navigation system 26, such as the instrument 68, and others as will be described in detail below, can be equipped with at least one, and generally multiple, of the tracking devices 66. The instrument can also include more than one type or modality of tracking device 66, such as an EM tracking device and / or an optical tracking device. The instrument 68 can include a graspable or manipulable portion at a proximal end and the tracking devices may be fixed near the manipulable portion of the instrument 68.

[0064] Additional representative or alternative localization and tracking system is set forth in U.S. Patent No. 5,983,126, entitled “Catheter Location System and Method,” issued November 9, 1999, which is hereby incorporated by reference. The navigation system 26 may be a hybrid system that includes components from various tracking systems.A001 1698W001

[0065] According to various embodiments, the navigation system 26 can be used to track the instrument 68 relative to the patient 30. The instrument 68 can be tracked with the tracking system, as discussed above. Image data of the patient 30, or an appropriate subject, can be used to assist the user 72 in guiding the instrument 68. The image data, which may include one or more image data or images, may be registered to the patient 30. The image data defines an image space that is registered to the patient space defined by the patient 30. The registration can be performed as discussed herein, automatically, manually, or combinations thereof.

[0066] Generally, registration allows a translation map to be generated of the physical location of the instrument 68 relative to the image space of the image data. The translation map allows the tracked position of the instrument 68 to be displayed on the display device 84 relative to the image 108. A graphical representation 68i, also referred to as an icon, can be used to illustrate the location of the instrument 68 relative to the image data 108.

[0067] A subject registration system or method can use the tracking device 58. The tracking device 58 may include portions or members 120 that may be trackable, but may also act as or be operable as a fiducial assembly. The fiducial assembly 120 can include a clamp or other fixation portion 124 and the imageable fiducial body 120. It is understood, however, that the members 120 may be separate from the tracking device 58. The fixation portion 124 can be provided to fix any appropriate portion or region of interest (ROI), such as a portion of the anatomy. In various embodiments, the fiducial assembly 120 can be interconnected with a portion of a spine 126 such as a spinous process 130. The fixation portion 124 can be interconnected with a spinous process 130 of an individual vertebra 131 , in any appropriate manner. For example, a pin or a screw can be driven into theA001 1698W001 spinous process 130. Fiducial portions may also include one or more portions of the subject that may be imaged, such as boney portions. More than one subject tracker may be connected to the subject 30, however.

[0068] The surgical suite 18 may include the various display devices, such as the display devices 84, 84'. In various embodiments, in addition to or alternatively to the display devices 84, 84’, may be an augmented reality display device 150. The augmented reality display device 150 may include those disclosed in the U.S. Patent No. 11 ,839,433 or U.S. Patent application publication No. 2019 / 0175059, all incorporated herein by reference. Additional or alternative augmented reality displays include the Hololens® 2 sold by Microsoft Corporation having a place of business in Washington, USA. The display device 150 may include one or more opaque or partially opaque display screens 154 (including a left screen 1541 and a right screen 154r) that may be viewed by the user 72. The display device 150 may be held relative to one or more eyes of the user 72 such as with mounting portions or temples 156. The display device 150 may be connected to the processor assembly 102 in an appropriate manner, such as with a wire or wireless connection.

[0069] The user 72 may view a display displayed on one or more of the display portions 154 of the display device 150. The portion displayed may include the image 108 which may be viewed by the user 72 as the user views the subject 30. Therefore, the image may be viewed as superimposed on the subject 30 while also viewing the subject.

[0070] The display device 150 may also include a tracking device 160 that may be tracked by the navigation system 26. Therefore, the display device 150 may be registered to any appropriate space, such as the space of the subject 30 to allow for registration or appropriate display of an image relative to the subject 30. That is, the image displayedA001 1698W001 with the display device may be registered to the subject 30 in real time as the user 72 moves the display device 150.

[0071] Additional display devices may include a projector device or system 170. The projector system 170 may include an image projector, laser projector, or other appropriate projector system. The projector system 170 is configured to project an image in a display region or area 174. The display region 174 may be selected portions, such as the subject 30, and displayed or projected onto the display region 174 is the image or graphical representation (e.g., shape). In various embodiments, for example, display of an image such as the image 108 may be superimposed or projected onto the subject 30. As further discussed herein various portions, such as tracked poses of instruments, planned trajectories or positioning, or the like and may also be included and displayed in any of the appropriate displays, such as in the projection 174, it may also be displayed relative to the image 108. Thus, the instrument 68 can be positioned relative to the subject 30 and an image may be projected onto the subject 30. The projected image may include other representations, such as planned trajectories to which the instrument 68 may be moved.

[0072] Therefore, the surgical suite 18 may include various display portions that allow for a viewing of an image or a graphical representation or icon that is displayed on (e.g., projected onto) the subject 30. The image may be displayed directly such as with the projecting region 174 and / or may be displayed through the semi-opaque display screens 154 of the display device 150. Thus, the user 72 or any appropriate user may view an image or a graphical representation relative to the subject 30 while also viewing the subject 30, such as substantially in real time.A001 1698W001

[0073] Turning to reference to Fig. 3, in various procedures the procedure may include positioning one or more implants in the spine 126 of the subject 30. As discussed above, for example, the various display devices may display an image of the subject 30, including portions thereof, such as one or more vertebrae in the image 108. Further, as also discussed above, either the projector 170 may project the image region 174 onto the subject 30 or the display device 150 may be used to display an augmented reality or virtual reality image while also viewing the subject to 30. The projector 170 may also include a projector tracking device 178. The projector tracking device 178 may allow for determining or tracking a pose of the projector 170 relative to the subject 30. Therefore, the subject 30 may be tracked with the subject tracker 58 in the navigation space and the pose of the projector 170 may be tracked in the subject space. Thus, the projection may properly be displayed onto the subject 30 at the appropriate pose based upon a registration of the projector and the subject 30.

[0074] This further allows, as exemplarily illustrated in Fig. 3, various graphic representations displayed thereon such as a first graphical representation 200 and a second graphical representation 204. The graphical representations 200 and 204 may be appropriate portions, such as a graphical representation of a first screw 200 or a graphical representation of a second screw 204.

[0075] The graphic representations 200, 204 may display a planned positioning of the respective implant in the subject 30. That is a procedure plan may have planned parameters that may include planned poses of the implants after implantation. Therefore, the graphical representations 200, 204 may be displayed as projections onto the spine 126 such at a first vertebra 210 and a second vertebra 214. As discussed above, the patient tracker 58 may also be connected to a selected portion of the spine 126, such asA001 1698W001 the vertebra 131. The user 72 may understand a planned pose or position of the implant relative to the subject. 30.

[0076] In addition to a position or planned position of the implant that may be displayed with the graphical representations 200, 204, various planned portions of the procedure may also be displayed. For example, a trajectory line 218 may be displayed relative to the spine 126, for example relative to the vertebra 214. The trajectory line 218 may be displayed for viewing by the user 72 while positioning the selected implant into the subject 30. The trajectory line may be displayed as a planned trajectory for planning or confirming a procedure and / or for comparing to a real time or substantially real time tracked trajectory of an insertion instrument.

[0077] For example, as illustrated in Fig. 4, the trajectory line 218 may be a planned trajectory line and a second trajectory line 222 may also be displayed, such as on the display device 84. The second trajectory line 222 may be a line that represents a real time tracked pose of the instrument 68. Thus, the user 72 may understand the tracked pose of the instrument along the trajectory 222 relative to the planned trajectory 218. The user may determine that the tracked trajectory is appropriate and / or alter the trajectory to assist in performing the procedure. Nevertheless, the user may be able to view both of the planned trajectory 218 and the tracked trajectory 222 in an appropriate manner.

[0078] Again, the display device 84 may display the image 108 of the subject along with other appropriate graphic representations, such as the trajectory lines 222, 218 or other portions such as planned positions of an implant and / or tracked positions of the implant with the graphical representations 210, 214.

[0079] In various embodiments, the display may be any appropriate displays such as a panel or flat display, as illustrated as the displays 84, 84’. Displays that may generateA001 1698W001 a virtual or augmented reality (e.g., the projector 170 or display device 150), the displays may be registered to the subject 30 by the tracking of the respective display portions with a respective tracking device. As the display devices are tracked, if they move relative to the subject 30, the display may also be registered to the subject 30 to allow for the proper pose of the display of the prospective graphical representation. Further, the image 108 may be registered to the subject space 30 such that the tracked pose of the various planned trajectories, actual trajectories, planned or actual poses of various portions such as the implants may also be displayed at the respective pose relative to the subject 30.

[0080] Additional information or indications may also be displayed with one or more of the displays. For example, the selected display, such as the display 84 we also display a planned resection such as in a resection volume or region 230. The resection planned region 230 may include a boundary 232 that may illustrate a boundary within which a resection is planned to occur. In various embodiments, for example, a burr or selected bone removal tool may be used to remove bone from one or more of the vertebrae of the spine or 126. Thus, the boundary 232 may be illustrated as a plan for a bone removal boundary.

[0081] A resected boundary or volume 234 may also be illustrated relative to the plan boundary 232 as a tracked receptive region 236. The demarcation or indications 236 may include a color change, shading, or other appropriate indication relative to the boundary 234 to illustrate a bone removal area or volume. The bone removal volume 236 relative to the boundary 232 may be displayed in the appropriate displays, as discussed above. Thus, the planned boundary 232 of the bone removal region 230 may be illustrated relative to a substantially real time tracked boundary or volume of removal or region of removal 236. The planned bone removal of resection may then be illustrated andA001 1698W001 displayed relative to the tract bone removal and resection, such as by tracking the instrument 68. The planned boundary 232 may be a plan parameter that is a part of a procedure plan. The plan parameter may be predetermined and saved for recall during a procedure or after a procedure.

[0082] Turning reference to Fig. 5, a flow chart 250 illustrates a process for providing an indication to the user 72 regarding an intra-procedure or real time information regarding various systems in the surgical suite 18. The real time data may be displayed or provided relative to pre-operative or pre-procedure data, such as discussed above including planned trajectories and / or placements of an implant. The process 250 may begin in start block 254. The start block 254 may be the initiation or provision of the output to the user 72 with the process 250. In various embodiments, for example, the process 250 may operate or begin once the surgical suite 18 is initiated or powered. Regardless, one skilled in the art will understand that the process 250 may be initiated in the appropriate time, such as receiving an input from a user in block 258.

[0083] Receiving an input from the user may include various inputs. For example, the user may input a current procedure. The current procedure may be specific to a current subject. Therefore, the user may also input a current subject in block 258. In various embodiments, a current state may also be provided, such as a state of readiness of the subject for the procedure.

[0084] The current procedure may be a selected procedure such as positioning of a screw as an implant into the subject. The current subject may also be input, such as a specific subject having a pre-planned procedure. The state of the subject may be an indication of a current state, such as whether any other implants may have been placed previously during the current procedure or prior to a current state the procedure.A001 1698W001

[0085] As discussed above, the surgical suite 18 may have various portions that may be tracked, have sensors to provide data regarding a state of the particular system, or other features. Therefore, receiving of input or data from an instrument may occur in block 262. The receiving of input or data regarding instruments may allow for the provision of data to a selected system, such as the processor system 102, the cloud or machine learning system 1 12, or other appropriate systems.

[0086] In various embodiments, the receiving of input or data from an instrument 262 may include a sub-process or sub-systems that may be provided. Therefore, the receiving an input or data from an instrument may include a routine of selectively and / or periodically receiving inputs regarding various instruments. It is understood that the various inputs may be provided in any appropriate manner or time, including those understood by one skill in the art.

[0087] The received data may include a tracking pose data in block 266. Tracking pose data may be determined in various manners or systems, as discussed above, including with tracking by one or more of the tracking systems with the navigation system 26, determining a pose of the end effector 44 with the robotic system 20, or other appropriate system. The determined pose of various systems, such as the instrument 68, may be determined with the navigation system 26. The process 250 may, therefore, receive the tracking pose data of one or more of the instrument during a procedure. As also discussed above, the tracking pose may relate to an implant, a pose of an imaging system, etc.

[0088] Further input of data may include an instrument sensor data in block 270. Instrument sensor data may include various sensor data such as a torque measured at the instrument 68 or with the instrument 68, such as during positioning of the implants.A001 1698W001Other sensor data may also include a temperature sensor. A temperature sensor may measure the current temperature of an instrument. The temperature of the instrument may relate to a duration of run time, a work being performed, or resistance being made by a subject of a structure that the instrument is encountering (e.g., resecting). Thus, the temperature may be sensed and may be an input or data for the process 250. Other sensor data may include a speed of an instrument, such as a burr or drilling instrument.

[0089] The sensor data may include the various sensor data of sensors associated with the various members and may be correlated to various portions of a procedure, such as a portion being drilled, a speed or time of a portion of a procedure, or the like. Other received data or input data may include a length of use in block 274. A length of use may include a time period during which an instrument, such as the instrument 68, is being used. For example, a duration of a running of a motor may be measured and may be correlated to various portions of the procedure, such as time for positioning an implant, a depth of placement of an implant, a size of a resection, or the like. The run time of an instrument may be measured and received into the process 250, for various purposes.

[0090] The process 250 may also recall a procedure plan in block 280. The recall procedure plan may be based upon the received input from block 258. For example, the current procedure input by the user may allow for the recalling of a selected plan related to the procedure that is input. As discussed above, the planned procedure may include various data such as a trajectory of positioning an implant, making a resection, or the like. Further, a plan may include a volume or region of resection. Various other planned procedures may also include a positioning of an implant, depth of an implant, size of an implant, or other planned procedure tools or portions.A001 1698W001

[0091] The procedure plan may be recalled from a memory system. The plan may be generated by the user based upon various data, such as image data of the subject acquired prior to the procedure. The plan, therefore, may be based upon the selected diagnostic or pre-procedure data and may be based upon various inputs from the user 72. The plan may then be saved for later recall, such as including the various portions as discussed above, including selected trajectories, types of implants, and the like. The plan may be generated prior to initiating the procedure, including the process 250. Further, the plan may include various portions such as a length of use of an instrument, a bony portion or structure to be resected, a volume to be resected, a trajectory of movement of a tool, or the like. Therefore, a plan may include a predetermined or estimated sensor data such as a torque or speed of a resection that may be based upon a type of bony portion, size of a bony portion, or the like as a plan parameter. Other plan parameters may also be included as selected by the user 72.

[0092] The process 250 may compare a recalled procedure plan to received input data in block 290. The comparison may be a direct comparison, such as comparing the two trajectories including the planned trajectory 218 and the current trajectory 222. Thus, the comparison may include a real time comparison of a trajectory of a selected instrument. The comparison may also include a planned and current position of an implant, a planned and current amount of resection or position of a resection, a planned, current, or predetermined torque required for a resection, or other appropriate comparisons. The comparisons may be based upon the predetermined or planned bony portions, positioning of an implant, or the like in the subject based upon the pre-procedure data which may include image data of the subject. All of the comparisons may be ofA001 1698W001 features or characteristics of the plan and the received procedure data that may be referred to as parameters that are plan parameters or procedure parameters.

[0093] Exemplary comparisons of these features may be user or system selected. The comparisons may be differences between a planned characteristic and an actual or procedure characteristic. These may include differences in the planned and actual or implanted pose (i.e., position and orientation) of implants, differences in volumes of tissue (e.g., bone) removed, differences in geometry of tissue (e.g., bone) removed. Other comparisons may include relative motion of individual vertebral segments post or during a procedure relative to a plan or original image.

[0094] Based upon the comparison, a generated selected output may be made in block 294. The output may be an indication that is selected based upon the comparison including a determination of a warning indication, determination of a caution for advancement indication, or other appropriate indication or output to the user 72. The output may be provided to the user in block 298. The provision of the output may include a visual indication, such as a visual indication on the display 84 of the planned trajectory 218, and the current trajectory 222. As discussed above, various other outputs may be displayed or various other displays may be made which may be the provided output. Therefore, the display may be provided at the output in block 300.

[0095] Various other outputs may include graphical outputs which may include a color coding of various portion of the image 108, color coding of various regions relative to the image, or providing various other graphical representations. For example, if the current trajectory 222 is within a planned trajectory and / or within a pre-determined area relative to the planned trajectory 218 the trajectory line 222 may be indicated in a selected color. Additionally or alternatively, a region around the display trajectories, such as anA001 1698W001 area 310 may be displayed on the display device 84 relative to the image 108 and / or other graphical representations, such as the trajectory lines 218, 222. The region 310 may be a color based upon an indication of the comparison such as a green color to indicate that the current trajectory or pose is appropriate or within plan limits. Other colors such as a yellow color or a red color may be used to provide an indication of caution and / or that the procedure should stop and be evaluated based upon the current received data unblock 262.

[0096] Other indications may include a multiple part indication 312. The multiple part indication 312 may be always displayed with no or only one portion highlighted, such as illuminated. The multiple part indication 312 may include a first portion 313 and a second portion 314. The first portion may be illuminated when a procedure parameter meets a plan parameter, such as the tracked trajectory matching the plan trajectory. The second portion 314 may be illuminated as a warning or caution indication.

[0097] Further, for example, the planned resection 230 and the current resected volume 236 may be provided in a selected color based upon an amount of resection, position of a current resection, or other features. For example, if the resection is near a nerve bundle an indication that caution may be made such as providing a visual indication of caution 314 on the display before may be made. Further, the image 108 or a portion thereof (Fig. 6) may be colored in a selected color and / or other graphical representations may be made based upon the comparison. For example, a portion of the vertebra 315 may be colored or highlighted to indicate a status of the procedure based on the comparison.

[0098] Nevertheless, the process 250 allows for a comparison of a current input or data in block 262 to be compared to a recall data from block 280. The comparison mayA001 1698W001 allow for the generation of an output which may then be displayed to the user in any appropriate display including the display 84, 84', the augmented reality display 150, their projection display 174, or the like.

[0099] The process may then end in block 320. Ending the process of block 320 may be ending the process based upon the completion of the procedure and / or ending the process 254 for a selected indication. Nevertheless, it is understood that the process 250 may be performed numerous times to allow for updating of the output, continuously monitoring the current input or received data or the like.

[0100] In various embodiments, the surgical suite 18 may further include an additional recording or imaging system 181 , as illustrated in Fig. 1. The imaging system 181 may acquire optical or viewable images of the subject 30 and may be mounted on a mounting assembly 183. The mounting assembly 183 may be positioned relative to the subject 30 in an appropriate manner. In various embodiments, for example, the mounting system 183 may extend from a ceiling of a room in which the surgical suite 18 is positioned to allow the imaging system 181 to be moved relative to the subject 30. In various embodiments, the mounting system 183 may be mounted relative to or to the robotic assembly 20 and may also be moved relative to the robotic system 20. The images acquired with the imaging system 181 may be any viewable portions of the subject. For example, the imaging system may be an optical camera that may image a subjects surface (e.g., skin). The imaging system 181 may also or alternatively image internal portions (e.g., bone or muscle) if they are exposed, such as though an incision or opening.

[0101] The imaging assembly 181 may be robotically or autonomously controlled to allow movement of the imaging system 181 during a procedure. The imaging system 181 may be moved automatically such as to maintain a selected field of view of the subjectA001 1698W00130. The imaging system 181 may also be moved substantially manually with various inputs to allow for movement of the imaging system 181 relative to the subject 30. The imaging system 181 may still be manually moved with the robotic system, such as the mounting system 183 includes one or more movement joints to allow movement of the imaging system 181 related to the subject 30.

[0102] The imaging system 181 may include at least one optical imaging assembly 185. The imaging system 181 may optionally include more than one imaging system, such as at least a second imaging module 185’. The imaging modules 185 may allow for capturing of images of the subject 30 at a selected region, such as a region of interest 187 of the subject 30. For example, the region of interest 187 may be a region in which a procedure is occurring, such as one or more vertebrae of the subject 30.

[0103] If one optical imaging assembly 185 is provided a mono view of the subject 30 may be provided. If at least two imaging modules 185, 185’ are provided a stereoscopic view of the subject 30 may be determined or captured. This may allow for ranging of various portions of the subject and 30 and / or a greater field of view of the subject 30 to assist in maintaining a field of view of the subject 30.

[0104] The optical imaging portions 185, 185’ may be any appropriate optical imaging portions such as a charged coupled device (CCD), complementary metal-oxide- semiconductor (CMOS), combinations thereof or any other appropriate optical sensor. The optical sensor may have one or more focusing lenses and / or telephoto, zoom, or the like lenses to focus light on the selected sensor. Further, one or more lenses may be provided to allow for a zoom or enhanced imaging of the region of interest 187. The one or more lenses may be provided to manually focus, zoom, or the like the image acquired with the imaging assembly.A001 1698W001

[0105] Turning reference to Fig. 7A and Fig. 7B, an image 360 may be displayed with one or more display devices, such as the display device 84. The image 360 may be a displayed image that may be a substantially real time image of the subject 30. For example, the image 360 may be a real time image of the subject 30 acquired and displayed for viewing by the user 72 during a procedure. The user 72 may move the instrument 68 relative to the subject 30 and an image of the subject 30 and the instrument 68 may be displayed at the image 360 with the display device 84. The display device 84 may display the image acquired with the imaging assembly 181 . For example, the image 360 may be a zoomed or enhanced image that may display various regions or portions of the region of interest 187 in the image 360. For example, a first selected vertebrae 362, a second selected vertebrae 364, and a third selected vertebrae 366 may be displayed with the display device 84. The user 72 may select which portions to display as the image 360, such as by selecting a zoom factor, field of view, or other appropriate parameters.

[0106] The image 360 may be displayed with the displayed device 84 for viewing by the user. Further, as discussed above, the view in the image 360 may be a substantially real time image. Therefore, the image 360 may be analyzed with a selected system or according to selected instructions, such as those executed by the processor assembly 102. For example, the image 360, including a view of one or more of the vertebrae 362- 366, may be analyzed, such as edge detection, determining a volume thereof, relative position thereof, or the like. In various embodiments, a first edge 370 of the vertebrae 364 and a second edge 372 of the vertebrae 366 may be identified and highlighted on the image 360. The identification of the selected images may be based upon various parameters or techniques, such as edge detection techniques, or other appropriateA001 1698W001 techniques. Further, various portions of the anatomy may be further identified, such as identifying the vertebrae relative to other portions, such as an intervertebral disc 374.

[0107] The image 360 may further have additional identification or regions identified, such as segmenting or determining the various portions such as blood regions 380 and parameters thereof, such as volume of blood. As discussed above, the image 360 may be acquired with an appropriate imaging system, such as an optical imaging system 181. The optical imaging system may acquire the image 360 as a color image. Therefore, a plurality of colors may be identified in the image 360. A region or volume including a red volume 380 may be evaluated or identified as a blood region or volume. The processing module 102 may identify the volume or portion of the red identified region 380, such as relative to the other anatomical portions or portions within the image 360. This may allow a determination or evaluation of a volume of blood that is in the region. A determination of a volume of blood may be made for various purposes and may be provided to the user 72 during a procedure. For example, the volume may be displayed as a number in the image 360 or relative to the image, such as with the display 84. Again, as noted above, the image 360 may be a substantially real time image to allow for a viewing of the image 360 during a procedure, such as during a performing of a procedure.

[0108] The instrument 68 may be imaged and displayed with the display 84 in the image 360. The instrument 68, according to various embodiments, may be moved relative to the subject 30. As illustrated in Fig. 7A, the instrument 68 may be a resection instrument. The instrument 68 may rotate or otherwise move during an operation or selected procedure. In various embodiments, for example, the instrument 68 may rotate around a central axis of at least a portion of the instrument 68. The central axis 384 may define a substantially stable portion or selectively stable portion of the instrument 68.A001 1698W001Accordingly, if the central axis or a portion of the instrument 68 moves relative thereto an identification or evaluation of the instrument may be made. For example, the instrument 68 may include the central axis 384. However, if a distal tip 68t moves relative to the central axis 384, such oscillating greater than about 1 millimeter (mm), an evaluation of the instrument 68 may be made. In various embodiments, for example, a high oscillation or an oscillation greater than about 1 mm from the central axis 384 may be an indication that the instrument 68 is worn or requires service. Therefore, an indication or evaluation of the instrument 68 may be made during the procedure due to the imaging of the region of interest in the image 360.

[0109] The image 360 may be registered to the subject and to other image data, such as preoperative image data. Therefore, the image 360 may be analyzed or compared relative to a preoperative plan, as illustrated in Figs. 3 through 5, including regarding positioning of an implant, trajectory of the instrument 68, or other preoperative plan parameter. The image 360 may be analyzed to determine a position of the instrument 68 relative to a preoperative plan.

[0110] Further, the procedure plan may include an identification of regions or volumes, such as a region 390. The region 390 may be identified prior to a procedure, such as prior to performing any portion of the procedure on the subject 30. The region 390 may be identified as a caution or no-go zone relative to the subject. Thus, the subject 30 may include one or more of the regions 390 that are identified preoperatively. The image 360 may be registered to the preoperative image or plan according to various techniques, such as image-to-image registration, fiducial registration, or the like. Thus, the image 360 may be used to identify a pose of the instrument 68 relative to any predetermined regions, such as the region 390.A001 1698W001

[0111] The image 360 may include a graphic and / or a demarcation relative to the image 360, such as on the display 84, to indicate a pose of the instrument 68 relative to the predefined region. For example, a visual indication 394 may be displayed relative to the image 360 to provide an indication to the user 72 of a pose of the instrument relative to the predetermined region 390, or any other appropriate region. In various embodiments, for example, the indication may be a colored portion or a shaped portion to provide an indication to the user 72 of a pose of the instrument 68 relative to the subject 30, particularly to the region 390.

[0112] The indication may be a safety or error state indication. The indication may include a proximity warning or other indication of a safety indication. The indication 394 may also provide a do go or positive operative indication. Thus, the indication 394 may provide information to the user 72 during a procedure. The indication may also have more than one portion to indicate more than one status.

[0113] In reference Fig. 7B, the image 360 may be displayed with the display device 150. The image 360 may be displayed in the display device 150, such as a left viewing screen 1501 the image 360 may be a magnified image to allow for visual detail of the subject 72, including the spine 126 thereof. In a second display portion, such as a right to display 154r a full field of view of the user 72 may be viewed. In various embodiments, for example, the right screen may be substantially transparent such that the user 72 is able to view the entire field of view available to the user 72. This may allow the user 72 to view the spine 126 with no graphical overlays or representations in the right display 154r. This may allow the user 72 to understand both a magnified view of various portions, such as the three vertebrae 362, 364, 366 and the instrument 68. The user 72 may also view a fullA001 1698W001 field of view through the right display 154r. Viewed through the right display 154r may also be a view of any other portion of the operating theater or suite 18.

[0114] Accordingly, the user is able to view the subject 30 in an appropriate manner. In various embodiments, an image may include the image 360 that is a magnified or a detailed view of a portion of the subject 72. The image may be analyzed in an appropriate manner, such as generally understood by one skilled in the art and is noted above. Thus, the user 72 may have both real time image data of the subject 30 that is of greater magnification or detail than viewable by the user 72 unaided. The image 360 may also include additional detail or information, such as the indications as noted above. Further, the user, in various embodiments, may view the subject 30 or other portions with no further view or analysis, such as with the display device 150.

[0115] Turning to Fig. 8, a process 580 may be followed to illustrate or provide the display of the image 360, as discussed above. The process 580 may include the acquisition of the image 360 with the imaging system, as noted above. The image, once acquired, may be analyzed according to various embodiments, registered to prior acquired images and / or a plan, and displayed to the user 72.

[0116] The process may start in start block 600. Starting the process 580 in the start block 600 may include various processes, such as initializing the procedure, operating the imaging system, or other appropriate procedures. The process 580 may then acquire real time image or image data of the subject in block 604. The acquisition of the real time image data in block 604, may be with the imaging system, as discussed above. Generally, the imaging system may be positioned to acquire image data of the subject 30 for analysis and / or display, as discussed above.A001 1698W001

[0117] After the acquisition of the image data in block 604 a determination of whether the image data is of the region of interest (ROI) may be made in block 608. The determination of whether the image is in the ROI in block 608 may include an evaluation of the initially acquired image data, such as a comparison to a pre-procedure image, identification of one or more fiducials in the acquired image, or the like. In various embodiments, the analysis or determination of the images being in the ROI in block 608 may include a segmentation of the image acquired, such as a determination of one or more anatomical features for comparison to prior acquired image, identification of one or more fiducials, or the like. In various embodiments, the comparison may include a comparison to a prior image, comparison to an identified feature that is input by the user (e.g. a fiducial), or other portion. The analysis in block 608 may allow for a determination of whether the ROI is in the image to determine whether additional image data should be captured or if the imaging system should be moved.

[0118] If a determination is made that the image is not in the ROI, a no path 610 may be followed to move the imager in block 612. Moving the imager may include manually moving the imager, such as by physical contact with the imager. Manual movement may also include a manual input of a new or additional image or position or pose relative to the subject 30 by the user 72. In various embodiments, however, the imager may also move relative to the subject 30 in a predetermined pattern substantially automatically. As noted above, the imaging system may be positioned on a mount that allows movement of the imager relative to the subject 30 so that movement may be substantially automatic. In various embodiments, the imager may also move in a selected manner based upon a determined pose of the imager and a selected or desired pose to image the ROI. For example, if the image includes two fiducials and a pose of third fiducialA001 1698W001 is known the imager may move in a selected manner that may be selected to best achieve an image of the three fiducials. Further, a zoom or other focused feature may also be performed by the imaging system to attempt to image the ROI.

[0119] Following moving the imager in block 612, image data may again be acquired in block 604 and a determination of whether it is an ROI in block 608. Once the image is of the ROI, a yes path 614 may be followed. Following the yes path 614 may allow for further analysis and display of a selected image, such as the image 360 as discussed above.

[0120] With continuing reference to Fig. 8, the process 580 may process the image acquired with the selected imaging system, such as the imager or assembly 181. The image may be segmented, optionally, in block 616. The segmentation of the image may be performed in any appropriate manner, such as with edge detection, color or contrast differentiation, or the like. As discussed above, the edges of various portions such as vertebrae may be identified based upon a contrast difference between the vertebral tissue and soft tissue surrounding the vertebrae. Further, various colors may be identified in the image 260 and those may be segmented. For example, the red region 280 may be identified and segmented.

[0121] In various embodiments, a volumetric or area comparison may be made between the segmented portions. For example, a vertebra may be segmented and the area or volume may be compared to the area or volume of the segmented red region 380. Based upon a comparison and amount of blood in the image may be determined, such as by comparing the volume of the region 380 and a portion of the subject 30 which may also be identified and prior acquired images. Nevertheless, various portions of the image 360 may be segmented.A001 1698W001

[0122] The image acquired in block 604 may be analyzed in block 620. The analysis in block 620 may be performed in any appropriate manner, such as by executing instructions with the processor 102. The analysis may include the comparison of volume of the red region 380 and other portions of the image, the pose of the instrument relative to other portions in the image 360, or the like. Further, as discussed above, the real time acquired image data from block 604 may be registered to prior acquired image data, such as image data acquired with the imaging system 80. It is also understood that other appropriate image data may be acquired of the subject 30 and may be registered to the real time image data acquired with the imaging assembly 181 . The registration may allow for an evaluation of a pose of the instrument 68 in the image relative to various portions that may be determined to known poses in the prior acquired image.

[0123] Based upon the analysis and / or registration in block 620, a procedure output may be made in block 630. The procedure output may be in the appropriate output, such as based upon the analysis and / or registration. For example, an instrument position indication may be output in block 634. The indication may be the indication 394 based upon an image portion of the instrument 68 relative to various portions of the subject 30 in the image 360. The output may be based upon the registration of the subject image data that is prior acquired to the real time image data wherein various structures, such as nerves or blood vessels, may be identified in the prior acquired images. Therefore, the real time image that images the instrument 68 and other portions of the subject 30 may allow for a determination of a pose of the instrument 68 relative to the predetermined portions. In various embodiments, an indication that the instrument is within a proximity of a selected region by a selected amount may be provided to the user 72. As discussedA001 1698W001 above, the indication 394 may be a visual indication that is displayed relative to the image 360 for viewing by the user 72.

[0124] A provided procedure output may include instrument action indication in block 638. The instrument action indication may be an indication of an analysis of the instrument 68. The real time image data may be a movie type or continuous image data collection. Therefore, various actions or motions of the instrument 68 may be analyzed and determined in the image 360, including a plurality of images over a sequence of time. For example, the image 360 that may be acquired with the imaging system 181 may include an indication that the instrument is vibrating or operating in a manner that may be indicative that the instrument is dull or is bent, or other selected characteristic of the instrument 68. The instrument action indication may be provided similar to the indication 394 for viewing by the user to determine or evaluate the indication thereof. Thus, the system may provide an analysis of the image or plurality of images to a system providing the indication to the user. It is understood, however, that, as discussed further herein, one or more images may be output in block 642 and may be displayed in block 646. The user 72, therefore, may view the single image 360 or a plurality of images, such as in a time sequence, to allow for an evaluation of various features, such as movement of the instrument 68.

[0125] Other output provided may be an object notification in block 650. The object notification may be a notification that an object is identified in the image of 360 that may be an unexpected object, such as by a shape or material characteristic. For example, while the instrument 68 may be in a selected position, as illustrated in Fig. 7A, an analysis of the image 360 may identify a shape or color of an object 654. The object or portion 654 may be highlighted in the image 360, such as by blinking, color change, or the like.A001 1698W001Further, an additional indication similar to the indication 394 may be provided for viewing of the user 72. Nevertheless, the object notification in block 650, may be output to the user 72 to allow for evaluation of a portion within the image 260 away from the instrument 68. Again, the image 360 may be substantially magnified relative to the pose of the user 72, as exemplary illustrated in Fig. 7B.

[0126] A further output may include a procedure status 660. The procedure status may include an indication or indicator of a type or portion of the procedure completed, time of a selected portion of the procedure, or the like. For example, an indication may be made regarding an amount or time duration of a resection, such as being performed by the instrument 68. The surgical suite 18 may include communication between all portions thereof, such as operation of the instrument 68. Therefore, the surgical suite 18 may evaluate or calculate the operational time of the instrument 68 and provide an indication or an estimated indication of a status of the procedure.

[0127] As discussed above, the various outputs may be provided and output in selected manners, such as with the selected display 84. Further, optionally, the image may be output and, may be displayed in block 648. Thus, the user 72 may view the image 360 as a single image or plurality of images, such as a plurality of images over time, to evaluate the imaged region. Thus, the user 72 may directly view the images to analyze the subject 30, the instrument 62, and other portions.

[0128] The process 580 may then end in block 670. After ending in block 670, various other portions may occur, and the process 580 may also be repeated, such as that at a second time, to assist in performing the procedure. Thus, the process 580 need not be a single or unitary process but may be performed over a plurality of times.A001 1698W001

[0129] Turning reference to Fig. 9A, the viewing or imaging system 181 may include one or more camera system or lenses, such as the first camera system 185 and the second camera system 185'. The camera viewing system 181 , therefore, may have a field of view 720. The field of view 720 may view a portion of the subject 30, such as a portion or region of interest of the spine 126 of the subject. In various embodiments, for example, a procedure may be performed in a portion of the of the spine 126, such as one or more of the vertebrae including a vertebra 724. The vertebra 724 may include an edge 726 that is adjacent an intervertebral disc 728. The procedure may include resecting at least a portion of the vertebra 724 with the instrument 68. In various embodiments, for example, the disk 728 may be removed and a portion of the vertebra such as at the edge or near the boundary 726 may be resected.

[0130] The field of view 720 may include the portion to be resected or that is being resected with the instrument 68. The vision system 181 may be a stereoscopic or depth of vision system. The two cameras 185, 185’ may each have a different perspective of a discrete portion, such as the edge of 726, relative to a selected point, such as a central point 732. Therefore, the processor assembly 102 or other appropriate processor assembly may execute instructions to determine a distance of a point in an image captured in the field of view 720, such as by differing perspectives, a triangulation, or the like. The processor system 102, therefore, may determine a relative pose or boundary of the vertebra 724 in space and / or relative to an initial state.

[0131] As illustrated in Fig. 9A, for example, the field of view of 720 includes the vertebrae 724 without a resection thereof. As discussed above, the view of the imaging system 181 may generate an image or a plurality of images that may be compared to a prior acquired image, such as a MRI image, a CT image, or the like. The image acquiredA001 1698W001 with imaging system 181 may be registered to the prior acquired image in an appropriate manner, as also discussed above. Therefore, any plan or selected procedure information that may be associated with the prior acquired image may be related to the current image acquired with the imaging system 181.

[0132] During a procedure, a portion of the vertebra 724 may be resected. As illustrated in Fig. 9B, for example, after a period of time the vertebra 724 may be resected such that a new edge or boundary 740 is formed due to resection of the vertebra 724. Again, the imaging system 181 may have the field of view 720 that may include the resected portions or boundaries 740 of the vertebra 724. The imaging system 181 may be used to capture an image of the spine 126 including the vertebra 724 with the resected boundary 740. This later acquired image, such as a later acquired procedure image may be compared to the initial image, as captured with the imaging system 181 prior to any resection or prior to a current resection amount. Therefore, the imaging system 181 may be able to view and generate a current image relative to a prior image that included the boundary 726 of the vertebrae 724. A current and a prior image may be both generated with the imaging system 181 and the imaging system 181 may generate real time images.

[0133] Turning reference to Fig. 10, an image 748 may be displayed with a selected display device such as the display device 150 or the display device 84. The image 748 may be the image that is captured with the imaging system 181 . As discussed above, for example, the image 748 may be the image acquired after a selected resection. Therefore, the resected boundary 740 of the vertebra 724 may be illustrated in the image 748 as the image boundary 740' relative to the image vertebra portion 724'.

[0134] In addition, the image 748 may include an image of the instrument 68 as the image portion 752. The image portion 752 of the instrument 68 may be a direct image ofA001 1698W001 the instrument 68 as the image system 181 captures the image of the region of interest that may include the instrument 68 and the spine 126. Therefore, the image portion 752 may be a direct image of the instrument 68. It is understood, however, that an appropriate representation may be superimposed or displayed with the image 748. The representation may be a graphic representation that is based on a geometric predetermined or pre-known configuration of the instrument 68 such as based on a model thereof (e.g., computer aided drafting model).

[0135] The image 748 may also illustrate a representation of an unresected (e.g., initial) boundary as an initial image boundary 726' and a region that has been removed 756. The region that has been removed 756 may be illustrated in the image of 748 in any appropriate manner, such as with the selected color, image portion, or the like. Generally, the illustration of the unresected edge or boundary 726' may be based upon the image acquired with the imaging system 181 at the initial or prior time period, as illustrated in Fig. 9A compared to a current image of the current status of the vertebrae 724, such as illustrated in Fig. 9B. The two images may be compared to one another and allow for the illustration of the resected region 756. The comparison of the two images may be based upon or include various analysis or processing, such as edge detection, edge matching, or the like. The processor system 102 may execute instructions to identify edges or boundaries of the vertebra 724 in an image acquired with the imaging system such as in the condition illustrated in Fig. 9A and in the condition illustrated in Fig. 9B. The edges or boundaries of the vertebrae 724 may be identified and used to match or overlay the two images. The resected portion 756 may then be illustrated and allow for the illustration of the resected boundary 740’. Therefore, the imaging system 181 may allow for a viewing and determination of a resected portion of the subject 30, such as the vertebrae 724. ThisA001 1698W001 may be stored for analysis of the procedure, displayed for viewing by the user 72, or provided for other appropriate purposes.

[0136] With continuing reference Fig. 10, the image 748 displayed on the display 84 may also include further information displayed relative to the image 748, such as a graphical representation superimposed thereon. For example, the display 84 may include a planning representation 760 of at least a portion of a planned resection of the vertebrae 724 displayed on the display vertebrae 724'. The planning graphic 760 may illustrate a proposed boundary of a resection of the vertebrae 724. The planning graphic 760 may be a plan parameter that is displayed with the image 748. This may allow the current resection boundary 740’ to be illustrated relative to a planned boundary to assist in the user 72 determining an extent of the procedure, a current stage of the procedure relative to a planned procedure, or the like. Further, the planned procedure and boundaries 760 may be displayed at any appropriate time, such as during a procedure, following a portion of the procedure, or after procedure is complete for evaluation of the procedure.

[0137] Returning to reference to Fig. 9A and Fig. 9B, the imaging system 181 may image the subject 30 including the spine 126 to obtain various information, such as a visual indication of a procedure performance or speed, such as an amount of resection. In addition, thereto or alternatively, a 3D surface modeling system may be used. A surface modeling system may include a laser range finder 770. The laser range finder 770 may use the visible or invisible wavelengths (e.g., infrared wavelengths) to emit a beam 774 to the spine 126. The beam may be moved over the spine, such as the vertebra 724, to determine a boundary thereof. The beam 774 may be operated at selected times to determine a boundary of a selected portion, such as the vertebra 724, at various points in time. The range finding system and beam may be based upon various technologies, suchA001 1698W001 as a time of travel or interference to determine a range and determine a current boundary. The current boundary may be compared to a prior boundary, such as at a prior time in the procedure.

[0138] The boundaries may be compared to one another at any selected time, such as during the procedure, after procedure, or other selected times such that the user 72 or users other than the user 72 may evaluate the procedure based upon the boundary which may refer to or indicate in the amount of resection, timing of resection, or the like. The data acquired may be timestamped, correlated to other data, such as by time, and the like. The data, including the determined boundaries, may be indexed for later analysis of related characteristics, such as time.

[0139] In addition to determining a boundary, the imaging system 181 may also be used to image soft tissues relative to the skeleton 126. For example, soft tissues may include musculature, ligaments, tendons, nerve bundles (e.g. the spinal cord), and other soft tissues. Therefore, the imaging system 181 may view a relative position of various portions such as the soft tissues, and hard tissues, such as bone. The imaging system 181 may capture an image that shows a relative position or pose of any of the portions, including soft tissue portions relative to other soft tissue portions, hard tissue portions relative to other hard tissue portions, and soft tissue portions relative to hard tissue portions. Further, the imaging system 181 may be used to acquire images of relative positions or poses of any of the portions of the subject 30 relative to instruments, such as the instrument 68. The type of tissue maybe determine or identified manually, such as by the user, or automatically by analysis the image for various characteristics (e.g., color, placement, etc.).A001 1698W001

[0140] Again, the image acquired with the imaging system 181 may be used for analysis such as during the procedure, after procedure, or at any appropriate time. In other words, the images may be acquired with the imaging system 181 and stored in a selected memory system such as the memory system 103 and / or the cloud storage system 112. The images may be procedure data that may be stored and recalled for later analysis, as discussed further herein.

[0141] Additionally, techniques may be used to determine an amount of resection, including a model duration of an operation of the instrument 68 and amount of tissue that is removed. For example, the use of the instrument 68 may generally be understood to remove 1 cubic millimeter of hard tissue per minute. Therefore, the system, including the surgical suite 18, may determine or track and amount of time of operation in the instrument 68 and determine an amount of tissue removed. The instrument may further include various sensors, such as a torque or a temperature sensor to estimate or determine a type of tissue being resected or otherwise operated on. In various embodiments, the instrument with a selected tip 68t may sense a given torque based on the type of tissue and / or force applied with the instrument 68. Therefore, the system may evaluate an operation of the instrument 68 including a time of operation and various sensor information to determine the type of tissue being resected and extrapolated an amount of tissue removed based upon the type determined type of tissue and the length of operation of the instrument.

[0142] In various embodiments, a material may also be injected into a resected volume to assist in determining an amount of tissue removed. For example, as discussed above, the resection of the vertebra 724 may change the boundary to the boundary 740. The user may inject a volume of material into the vertebra 724 and / or between theA001 1698W001 vertebra 724 and other vertebrae. The amount of material able to be injected may be used to calculate a volume of tissue removed. The volume of tissue removed determined in such a manner may then be recorded, such as by the use 72, in a manual system and / or by determining a mass of material injected and the related volume based upon predetermined or pre-known density of the injected material.

[0143] The various information collected with the surgical suite 18 may then be used to evaluate or determine metrics for a procedure. The metrics may include the sensors associated with the instrument 68, the evaluation of an amount or timing of a tissue removed, the movement or placement of soft tissues, or other selected metrics. These metrics may be evaluated during and / or after the procedure to assist in determining any efficacy of the procedure and / or alternative to a current process and / or planned process. In various embodiments, for example, the actual metrics of a procedure may be compared to a predetermined or planned procedure metrics. The data may also be used to assist or create efficacy in a future procedure such as a selected outcome, procedure duration, motions during a procedure, procedure task order, tool selection, procedure outcomes, implant placement, etc. Further, the data may be used to assist in procedure, such as future, procedure such as speed, cutting speed, etc.

[0144] Turning reference to Fig 11 , a process 810 may be executed using the various portions, discussed above. The process 810 begins at start block 814. Beginning the Process 810 at start block 814 may include initiating a procedure, initiating the surgical suite system 18, or performing other initial steps. Generally, the procedure 810 begins by the user 72 initiating a procedure on the subject 30. The procedure may be any appropriate procedure, and may include positioning the subject 30 relative to the surgicalA001 1698W001 table 30 or other initial steps. Generally the procedure may include a portion, such as obtaining access to an area within the subject 30.

[0145] The process 810 may include various instructions executed by one or more processor systems, such as the processor 102. The various portions may also be automatically performed, as discussed herein. Thus, data may be collected and analyzed by executing instructions with the processor 102 or any appropriate processor. The process 810, therefore, may be an automatic process based on data input and the selected instructions.

[0146] After starting the process 810 in start block 814, a plan for a procedure may be recalled in block 820. Recalling the plan for the procedure in block 820 may include recalling any portion of a plan for performing a procedure on the subject 30. For example, determining a trajectory for an instrument, selecting an implant for positioning within the subject, determining an amount (e.g., volume of removal) of a resection, or any portion of the plan. The plan may further include various selected outcome metrics such as a final spine angle including lordosis or balance, volume of a tissue removed, time of a procedure, time or duration of portions of a procedure, volume of tissue removed, or other metrics. Thus, the plan may include information regarding a proposed procedure, sequence of procedure, timing of a procedure, or the like regarding the subject 30.

[0147] The plan for the procedure recalled in block 820 may be based upon various data acquired regarding the subject 30 prior to a procedure. For example, image data may be acquired of the subject 30, such as with the imaging system 80. The pre-procedure or prior acquired image data may be used to assist in determining a plan for a procedure. The image data may include image data of any appropriate type selected by the user 72 to evaluate the various portions of the subject 30. Based upon the evaluation and relatedA001 1698W001 other steps, such as a diagnosis of the subject 30, the planning portions of the procedure may be made. Thus, the prior acquired data may be used to assist in determining a trajectory for moving an instrument, a selected implant, or other planned procedure portions.

[0148] The plan may be recalled in the procedure 810 at any appropriate time. Thus, the recall of a procedure plan in block 820 is merely exemplary. Nevertheless, the plan for a procedure may be used to set baseline metrics in block 824. Baseline metrics may be determined in block 824 based upon the plan and include the various procedure portions as noted above. The baseline metrics may include an amount of tissue to be removed, a duration of the procedure or portions of the procedure, instruments to be used during the procedure, an order of specific instruments to be used, implants to be used during the procedure, and other various metrics. The metrics may also include parameters (e.g., temperature or torque) that may be sensed relative to one or more of the instruments used relative to the subject 30 including the instrument 68 or any various implants. The plan for the procedure may be used to set baseline metrics in block 824. As discussed herein, the baseline metrics may be compared to intra-procedure metrics to allow for a comparison between the recalled plan based metrics and the procedure measured metrics.

[0149] The process 810 may include recalling a procedure plan at block 820 and setting baseline metrics in block 824 based on the recalled procedure plan. The process 810 may then receive and collect procedure data in block 840. In receiving and collecting procedure data in block 840 various information or data may be collected from the suite system 18. The various portions of the suite system 18 may include sensors for instruments, tracking data regarding poses of one or more portions of the surgical suite,A001 1698W001 and other features. The various data collected may also be analyzed or processed by one or more processing systems. Further, the data may be collected and stored in a selected memory system, such as the memory system 103 or the cloud system 112.

[0150] The received data may include images (e.g., visible light pictures) of a region of interest in block 844. The images may include those images acquired with the imaging system 181. The region of interest may include one or more portions of the subject 30, including the region of interest 187 of the spine 126 of the subject 30. As discussed above, the various images may be used to evaluate the procedure on the subject 30. The evaluation of the images may be performed by executing instructions with one or more processing systems, such as the processing system 102 and / or processing performed by a processing system after a procedure, such as being recalled from the cloud memory 1 12. This allows the data to be collected during a procedure and analyzed at a selected time, such as during a procedure, after a procedure, or after several procedures. In various embodiments, more than one procedure data may be collected and analyzed for evaluation, as discussed further herein.

[0151] The image data acquired in block 844 may also be used for evaluating features or parameters of the procedure on the subject 30. As discussed above, a volume or amount of blood may be calculated based upon the images. Further the image data collected in block 844 may be used to evaluate an amount of tissue removed, a movement distance of selected soft tissues or hard tissues, or other features. In various embodiments, the images collected in a block 844 may be used to evaluate a portion of a procedure as the images may be acquired at selected times which may be timestamped.

[0152] The received or collected procedure data in block 840 may also include the instrument data from block 850. The instrument data may include any appropriate dataA001 1698W001 from the instrument, such as the instrument 68. Various instrument data may include a runtime of the instrument 68. The runtime of the instrument 68 may be evaluated to determine or estimate an amount of tissue removed based upon the instrument 68. For example, a selected burr may be connected to the instrument and a runtime of the burr may be used to estimate a volume or mass of material removed with the instrument 68. Various other data may also be evaluated, such as a speed of the instrument. For example, a greater speed of the instrument may relate to a greater amount of material removed. The speed of the instrument may be collected during the procedure, such as discrete times. The discrete speed measurements may be timestamped to relate to other parameters collected during the procedure. Also, the speed may be correlated to run time. Instrument data may also include a temperature data. Temperature may be used to determine a speed, force, resistance, or the like associated with use of the instruments 68. Therefore, temperature may be used to evaluate an amount of tissue removed. Further, a torque measurement of the instrument may also be collected. The torque measurement of the instrument may be used to evaluate an amount of force applied to the tissue of the subject 30, a type of tissue being encountered by the instrument 68, or other factors. In various embodiments, the torque measured of the instrument may measured with a torque sensor.

[0153] Other data collected may also include surface or volume measurements in block 858. The surface or volume measurements may include those similar to those discussed above. For example, the imaging system 181 may be used to determine a volume of tissue removed or a change from a first time or period to a second time or period. Thus, a tissue volume change (e.g., between time periods) may be calculated based upon the images collected with the imaging system 181. Other surfaceA001 1698W001 measurement systems may also be used, such as a laser range finding system to evaluate a change in a volume or an area, such as a region of interest. Thus, the procedure data may include a surface volume or other measurement change. One skilled in the art will understand that the various measurements may be collected at discrete times or over periods of time to allow for a timestamp or determination of a change over time period.

[0154] The procedure data may also include a measurement or determination based upon a volume or mass injected into the subject in blocking 866. The volume or mass injected may include an injection of a selected amount of material that may be measured based upon a mass or a volume of the material injected. The material injected may be injected into an area to allow for an estimation of a mass or volume of a resection region. The injected material may be measured, such as during or after an injection thereof to determine a volume of a tissue removed during a procedure. The volume or mass may then be recorded and stored, such as with the memory system 103 or other appropriate system, such as the cloud system 112.

[0155] The process 810 may include the procedure data collected in block 840. The procedure data may be used for various purposes and / or may be stored for a later evaluation. Regardless, the procedure data collected in block 840 may be collected with one or more system of the surgical suite 18.

[0156] The procedure 810 in block 880 may then determine intra-procedure metrics that are analogous to baseline metrics from block 824. As discussed above, the planned or baseline procedure metrics may include a volume of tissue to be removed, various geometry of an anatomy after a procedure, time of various portions of a procedure, trajectory of an instrument, or the like. The baseline metrics determined in block 824 may be recalled for the comparison or determination in block 880. The intra-procedure metricsA001 1698W001 may be based upon the data collected in blocking 840. The procedure data collected in block 840 may be used to determine metrics in block 880. The metrics determined in block 880 may be analogous to the baseline metrics, but using the intra-procedure data collected in block 840. For example, the metrics may be determined by evaluating historic data collected from previous procedure data that are classified in a selected manner (e.g., good outcome). Metrics may also be determined by simulation by analysis of the data from the procedure. It is understood by one skilled in the art that the metrics may also be a combination of any of these processes.

[0157] After determining the intra-procedure metrics in block 880, a comparison of the intra-procedure metrics to the baseline metrics may be made in block 890. The comparison may be a direct comparison or a qualitative comparison. For example, a volume of tissue to be removed may be determined in the baseline metric from block 824. One or more of the procedure data collected in block 840 may be used to determine an intra-procedure metric related to a volume of tissue actually removed during a procedure. Thus, the intra-procedure metric may be an estimation or determination of a volume of tissue removed that may be based upon a speed of the instrument, duration of use of the instrument, type of instrument used, image data collected, surface volume determination, or the like. The comparison of the baseline metrics to the intra-procedure metrics may allow for a comparison of a planned procedure metric to an actual procedure metric. The actual metric may be based upon the actual procedure and the data collected regarding the procedure. The data collected regarding the procedure may be based upon one or more sensors or systems, as discussed above. Thus, the process 810 may allow for a substantially automatic comparison of pre procedure metrics to intraprocedural metrics.A001 1698W001

[0158] The comparison may then be output in block 894. The output in block 894 may be any appropriate output such as a visual output, an output for storage for later analysis or recall, or the like. The output of the comparison in block 894 may allow for the user 72 to evaluate the plan relative to the actual measured data from the procedure. As discussed further herein, the comparison may be used to evaluate the procedure relative to the plan and / or other procedures. Thus, the current procedure or actual procedure may be compared to other procedures or actual procedures different than the current procedure.

[0159] The process 810 may then end in block 900. Ending the procedure 810 in block 900 may include the end of collection of data for a selected procedure, ceasing operation of one or more systems of the surgical suite 18, or the like. It is understood that the procedure 810 may be performed sequentially or iteratively for a selected procedure. According to various embodiments, for example, a procedure may include one or more steps or portions such as resection of hard tissue, implantation of an implant, moving or replacement of soft tissue, or the like. The process 810 may be used for one or more procedures of the subject and it may allow for the selected metrics and comparison and output thereof, as discussed above.

[0160] Turning reference to Fig. 12, the instrument 68 may be a drilling or resection instrument including the instrument tip 68t. The instrument tip 68t may be a burr which may remove material from a volume when moved relative to the volume. For example, the tip 68t may have an abrasive exterior that may be used to remove bone tissue. The bone tissue may be a portion of a vertebrae, such as vertebrae of the spine 126.

[0161] The instrument 68 may be tracked in a navigation space with a tracking device 66. The tracking device 66 may be any appropriate tracking device such as anA001 1698W001 optical tracking device, electromagnetic tracking device, or other appropriate tracking device. The tracking device 66 may be exterior to the instrument 68, as illustrated in Fig. 12. In various embodiments, however, the tracking device may be substantially flush or interior to an outer housing of the instrument 68.

[0162] The instrument 68 may include various controls such as with a control lever 930. The control lever 930 may be operated by the user 72. The user 72 may control or contact the control 930 with a digit. Thus, the user 72 may control a speed, a length of operation, or other parameters of operation of the instrument 68 with the control 930. In various embodiments, for example, the user 72 may position the tip 68t in a region or near a region to be resected. The user 72 may then operate the instrument 68 by pressing the control 930. The tip 68t may then rotate and the user may move the instrument 68 to resect tissue in a selected region.

[0163] The instrument 68 may also be connected to and / or incorporated into the robotic system 20. The robotic system 20 may hold the instrument 68 and move the instrument 68 relative to the subject 30. Thus, a pose of the instrument 68 may be determined and tracking the pose of the robotic system 20 including using various sensors therein, such as encoders as discussed above. The robotic system 20 may include portions to contact the control 930. The instrument 68 may be included or connected to the robotic system with the end effector 44, as discussed above. In various embodiments, however, the robotic system 20 may incorporate or include the instrument 68. Therefore, operation of the instrument 68 may be separate from the control mechanism 930 and the robotic system 20 may include various control assemblies, such as being operated via signals through the robotic system 20.A001 1698W001

[0164] Regardless, the instrument 68 may be operated during a procedure. The operation of the instrument 68 may be sensed with various sensors in the instrument 68 and / or relative to the instrument 68. For example, as discussed above, the instrument 68 may be tracked with the tracking device 66. Alternatively or additionally, the pose of the instrument 68 may be determined with the robotic system 20. Therefore, the instrument 68 may have its pose determined with various systems in the surgical suite 18.

[0165] Additional sensors in the instrument 68 may include an accelerometer 940. The accelerometer 940 may allow for a determination of movements of the instrument 68 in addition to or separate from the tracking device 66. The accelerometer 940 may be used to determine minor or small changes substantially instantaneously. For example, the accelerometer 940 may be used to determine a movement in any direction in substantially real time (e.g., less than one second) and may be used to determine a magnitude thereof.

[0166] Other sensors may include a gyroscope 944. The gyroscope 944 may be included with the instrument 68 and be used to determine a pose or change in position of the instrument 68. The gyroscope 944 may be used to determine a substantially instantaneous or real time position change and magnitude of change of pose of the instrument 68. Again, the gyroscope 944 may be provided in addition to or alternatively to the tracking device 66 and / or the accelerometer 944.

[0167] The robotic system 20 may also be included or have one or more of the sensors. For example, the robotic system 20 may include an accelerometer 940’, as well as or alternatively the gyroscope 944’. The sensors may be used to substantially instantaneously determine a change and / or magnitude of change of a position of the robotic system. Further, the sensors may be used to determine a motion or resistance toA001 1698W001 motion by the robotic system in a volume relative to the subject 30. Therefore, the sensors, such as the accelerometer 940’ and the gyroscope 944’ may be used to evaluate a change in pose, change in pose of the robotic system related to the instrument 68 and the subject 30, or combinations thereof.

[0168] Turning reference to Fig. 13A and Fig. 13B, the surgical suite 18 may include various other instruments, such as a microscope 980. The microscope 980 may be used to view a selected region, such as a portion of the subject 30. The subject 30 may be placed on the support. The microscope system 980 may have an objective lens 984 positioned to view a portion of the subject 30, including a region of interest such as the region of interest 187. The view of the microscope 980 may be viewed through one or more eyepieces 988. Additionally or alternatively, the view of the microscope 988 may be displayed on the display device 84. Alternative or additional display devices may include a microscope display device 992.

[0169] The user 72 may move the microscope to view selected portions of the subject 30. The user 72 may view the image through the eyepieces 988, the display on the display device 992, or combinations thereof. The user 72 may move or position the instrument 68 relative to the subject 30. The display on the display device 992 may include a direct or real time display of the instrument and / or a graphical representation thereof 68i. The display with the display device 992 may generally be the view through the microscope 980. The view may be magnified, however, but may have portions obstructed by various structures of the subject 30.

[0170] The images acquired with the microscope of 980 may be analyzed with one or more processing systems, such as the processing system 102. Further, the images may be analyzed, saved, or combinations thereof with the cloud system 112. The cloudA001 1698W001 system 112 may include one or more processors for evaluating the images acquired with the microscope 980.

[0171] Analysis of the images, in various embodiments, may include analysis of Euler angles to attempt to understand or evaluate structures below an exterior surface of the subject 30 viewed with the microscope 980. According to various techniques, the image of the microscope 980 may be analyzed to display movement of tissues below the exterior surface. The movement of tissues may be due to various features, such as a vasculature of the subject. For example, as illustrated in Fig. 13B a vessel 996 may be displayed with the display device 992. The vessel 996 may be determined due to an evaluation of the image acquired with the microscope 980. Thus, the user 72 may understand that the position of the vessel 996 without direct viewing of the vessel 996. The vessel 996 may be displayed on the display device 992 in any appropriate manner such as with a graphical representation of the vessel, direct illustration of the vessel, or the like.

[0172] Thus, the user 72 may view or evaluate various features relative to the subject 30 during a procedure. The information provided regarding movement or types of movement of the instrument 68 may be collected during a procedure and evaluated during a procedure. Further, the movement information may be evaluated following a procedure to assist in evaluating the procedure relative to a predetermined baseline characteristic of parameter as discussed above and further herein. The display may also be used to evaluate or understand a feature of the subject 30, such as a vessel due to analyzing movement of portions of the subject 30. The image evaluation may assist in performing a procedure, such as in determining or confirming a position of selected portions of theA001 1698W001 procedure, information regarding confirming the procedure, or the like. The image analysis may also be used following a procedure to reconcile a plan to the actual procedure.

[0173] Various additional sensors may further include temperature sensors within the instrument 68. The temperature sensor may be used to measure a temperature of one or more portions of the instrument. The temperature measured or sensed may be used to evaluate an efficiency of cutting or resection, quality of the resection or instrument that is resecting, etc. This information may be also used to evaluate the procedure relative to a plan for the procedure.

[0174] Turning reference to Fig. 14, a process 1100 is illustrated. The process 1100 may be a process for comparing procedure metrics determined from collected procedure data to a recalled baseline metric. The process 1100 may begin in start block 1104. The starting of the process in 1104 may include various portions, such as initiating a procedure, ensuring a selected plan is determined, or the like. The process 1100 may then proceed to recall baseline metrics in block 1 108. Recalling the baseline metrics in block 1108 may include recalling predefined baseline metrics. The predefined baseline metrics may include specific metrics, such as a time duration for a procedure, sequencing and / or timing of sequencing of the procedure, a specific implant to be positioned, a volume of tissue or material to be removed, or the like. Further, baseline metrics may be determined based upon various inputs, such as a geometry of an anatomy following a procedure, timing of a procedure, or the like. The baseline metrics may be recalled at any appropriate time but may be based upon pre-procedure data. Pre-procedure data may include pre-procedure image data, pre-procedure planning, pre-procedure measurements, and the like. The baseline metrics may be recalled from the plan and mayA001 1698W001 be predetermined and included in the plan. The baseline metrics may be any appropriate metric and may be based on past procedure data or analysis, as discussed above.

[0175] During a procedure, various data or information may be collected. The information may be collected in any appropriate manner, such as based upon collecting data from portions of the surgical suite 18, input from the user 72, or other data. Further, data may include post procedure evaluations, such as efficacy or prognosis following a procedure, or the like.

[0176] Procedure data may include collecting movement information regarding the robotic system 20 or the instrument or instrument 68. The movement may be based upon accelerometer measurements with an accelerometer sensor associated with one or more of the robotic system portions or instruments. Additional movement information may be determined or collected from a gyroscopic sensor. The gyroscopic sensor may also be associated with the robotic systems or the instruments. The accelerometer and gyroscopic sensors may collect information regarding movements of the various systems. The movement data sensed and collected by the accelerometers of the gyroscopes may include movement regarding absolute movement data other robotic system or the instruments. The movement data may be related directly to movement of the various portions and not within a selected space, such as a navigation space.

[0177] Controller information may be collected in block 11 16. Controller information may include force data, movement data, distance of movement data, or the like. The controller information may include information collected from the controller of the robotic system 20. The controller data may further include a timing of movement, a duration of movement of the robotic system, force applied to the robotic system, force applied by the robotic system, or other related controller data. The controller data may be timestampedA001 1698W001 or otherwise sequenced within a procedure to allow for a determination of various forces, timing, and the like of movement of the robotic system. In various embodiments, for example, the time of a movement of a robotic system or portion thereof may be calculated based upon the controller information. The movement of the robotic system may be evaluated, such as after a procedure, to determine an efficiency of movement. For example, an order of movements to reduce the total movement time of the robotic system.

[0178] Localization information may be collected in block 1120. Localization information may include information from one or more tracking devices associated with the various portions in the surgical suite 18. As discussed above, the tracking device 66 may be associated with the instrument 68. The tracking device 66 may, in various embodiments, require a direct line of sight to a localizer, such as the optical localizer 88. A measurement of a quick change or sequential change in pose of the tracking device 66 may be detected by the navigation system. The user, such as the user 72, may not be aware of the pose change associated or collected with the navigation system. Thus, the navigation system may provide a notification to the user 72 that multiple movement changes have been detected. The multiple changes, particularly in a selected or short time span, may be indicative of an obstructed or partially obstructed tracking device and such as indication may be made to the user 72.

[0179] Information regarding the cutting instruments may be collected in block 1124. Data regarding the cutting instrument may include a runtime of the cutting instrument, a temperature of the cutting instrument, a log of the run time of the instrument, or the like. The various data regarding the cutting instruments may be used to determine an efficacy or efficiency of the cutting instruments, a lifetime of life usage of the cutting instrument, or the like. In various embodiments, the cutting instrument may be a cuttingA001 1698W001 burr and an increased temperature or extended runtime may be used to evaluate a possible end of life of the cutting burr. Further, various other cutting instruments or technologies may be used, such as electrosurgical cutting instruments, laser cutting instruments, or the like. Again, temperature may be used to determine an efficacy or efficiency of the cutting instrument.

[0180] Image data may be collected in block 1128. Image data may include various image data such as microscopic imager data collected by the microscope 980. Other image data may include real time ultrasound image data, or the like. The image data may be analyzed either during a procedure or after procedure for various purposes. For example, as discussed above, visual image data may be analyzed to evaluate Euler angles determined or collected in the image data to determine or evaluate subsurface structures. The subsurface structures may be evaluated to assist in determining a procedure or further the continuation thereof.

[0181] Procedure metrics may then be determined in block 1140. Procedure metrics may be metrics that are analogous to the baseline metrics recalled in block 1108. Procedure metrics may be any appropriate metrics that may be determined based upon the collective procedure data. Accordingly, for example, the efficacy of the various instruments may be determined based upon the collected cutting instrument information in block 1 124. The metric may include a run time or efficiency of movement of the cutting instrument. Further, the image data collected in block 1128 may be used to determine a volume of material removed during a procedure. Therefore, a metric determined in block 1140 may also be a combination of the cutting efficacy based upon the cutting instrument information in block 1124 related to a volume removed as determined by the image data in block 1128. The metrics, however, may be determined in block 1140 based upon theA001 1698W001 procedure data. In various embodiments, the determined procedure metrics may be only metrics that are analogous to baseline metrics that are determined based upon the collective procedure data. The metrics may also be referred to as parameters, in various embodiments, and may relate to one or more parameters or metrics that may be used to evaluate the plan or the procedure.

[0182] In various embodiments, the parameters or metrics may be determined relative to prior procedures, analysis of various users, etc. For example, comparison of time length for a procedure to previous similar procedures. Comparison of the number of steps required to reach or complete a procedural step to previously completed and similar procedures. Estimation of tissue removed over time spent removing tissue and comparing, possibly, to previous procedures or other established targets. Number of clinician interventions or changes made to a surgical plan during the course of a procedure. Deviations of regions / anatomy of interest from imaging data relative to a surgical plan or established baseline (e.g., intended correction, measures from previously completed procedures).

[0183] After determining the procedure metrics in block 1140, the metrics may be compared in block 1150. The comparison of the metrics in block 1150 may be a comparison of the baseline metrics to the procedure metrics. The comparison may be a direct comparison, such as a determination of whether the procedure achieved the baseline metrics. For example, the metrics compared or the baseline metrics evaluated may include a volume of tissue removed, a timing of a procedure or a portion of the procedure, or the like may be used or compared in block 1150.

[0184] After the comparison in block 1150, an output of the comparison may be made in block 1 154. The output of the comparison in block 1 154 may be outputting theA001 1698W001 comparison for storage and later analysis, outputting such as in a visual display, or the like. The comparison may allow for a later analysis or determination.

[0185] For example, a determination and output of a match of the baseline metrics to the procedure metrics may be made in block 1160. The determination and output of a match may include an output of the match based upon the comparison in block 1150. The output may further include a notation or analysis of the comparison of the metrics. For example, a comparison of the cutting instrument information and the related metrics to the baseline metrics may be used to evaluate or output that the cutting instrument did not achieve a selected cutting efficacy or efficiency. Therefore, the output in block 1160 may include an evaluation or determination of a change for the cutting instrument.

[0186] The procedure 1100 may then end in block 1164. In ending the procedure of 1164, the user 72 may evaluate the output of the comparison. Therefore, the user may understand the comparison made in the procedure 1100. Further, the procedure 1100 may allow for an update or a saving of the comparison to assist in a future planning. A current procedure may allow for the collection of the current procedure data to compare it to baseline metrics to determine whether the baseline metrics were achieved or a deviation therefrom.

[0187] If baseline metrics are not achieved, the process 1100 may output the comparison including an output of possible alterations to the various systems for which information was collected to better achieve the baseline metrics in a future procedure. For example, as noted above, a change in the cutting instrument, a change in an order of a procedure, or other changes related to the collected information may be suggested.

[0188] The process 1100 may be understood to be a substantially automatic process that is performed by the collection of the data during a procedure and theA001 1698W001 execution of instruction with a processor system, such as the processor system 102 and / or processor system associated with the cloud system 112. The process 1 100, therefore, may allow for an automatic evaluation of a current procedure relative to the plan and possible suggestions for a future procedure. This may be based on a comparison of the metrics or parameters of the procedure and the plan.

[0189] The surgical suite 18 may include various system as discussed above. These various systems may allow for the collection of data during a procedure. The data collected during a procedure may be compared to preoperative planning for a procedure and collected postoperative data. The operation or procedure may be a procedure in any appropriate subject, and a human subject is merely exemplary. As noted above, the subject may be a non-human or non-living subject. Nevertheless, a plan for a procedure or operation may be made, data regarding the procedure as it is performed may be collected as intraoperative data, and postoperative data may also be collected after the procedure is complete.

[0190] In addition to the surgical suite including the various instruments, the procedure may include positioning of various items or implants relative to the subject. Information or data regarding the implants may be obtained preoperatively, intraoperatively, and postoperatively. Therefore, the data may also be compared at each of the preoperatively, intraoperatively, and postoperatively or between each of the preoperatively, intraoperatively, and postoperatively to the outcomes of the procedure of the subject 30. Further, the subject 30 may have outcomes that may be measured and determined for comparison to the preoperative plan and the interoperative data. The comparison or correlation of the various data may allow for a planning of a future procedure, as discussed herein.A001 1698W001

[0191] Turning to reference Fig. 15, a procedure 1300 may include a determination of a correlation of a preoperative plan, intraoperative data, and postoperative outcomes. The correlation may be performed by executing instructions with the selected processor, such as the processor 102, the processor associated with the cloud system 112, or other appropriate processor. Regardless, the instructions may allow for a determination, if any, of a correlation between outcomes in various preoperative plans and / or intraoperative data. In various embodiments, the correlation may include a linear correlation that may be determined between one or more of the preoperative plan, interoperative data, and the subject outcomes. A nonlinear fit may also be determined. In various embodiments, a machine learning system such as a convolutional neural network, a transformer, or other appropriate machine learning system may also be used or alternatively used to determine a correlation or prediction between the various data and the subject outcomes. Further, it is understood the correlation may be made between the preoperative plan and the intraoperative data to assist in determining the correlation of a plan relative to interoperative actions.

[0192] The process 1300 may begin in start block 1310. The starting of the process in the start block 1310 includes the initialization or beginning of a process, such as a diagnosis of a subject 30 by the user 72. The process 1300, however, may start in any appropriate manner.

[0193] The process 1300 may then include the recall of a generated preoperative plan in block 1320. The preoperative plan may include various information or data regarding the subject. The preoperative plan may include those features, also referred to as parameters, listed in block 1320 and may include: Location of Incision, Number of Incisions, Size of Incisions, Internal Region of Interest (ROI) Access, Trajectory of ROI,A001 1698W001Implant Type (e.g., Type of Screw), Location of Incision, Implant Placement (e.g., Screw Placement, Cage Placement, Tissue Volume and Location Removal, Location and Identification of No-Go Regions, Rod Shape and Positions, Desired Spine Outcomes (Lordosis, Cobb angle, etc.) or any combinations of these. It is understood, however, that other data or less data may be included in the plan. As discussed further herein, the fitting of the preoperative planning data to the outcomes and / or the intraoperative data may provide a determination that a preoperative plan feature or parameter is not relevant to an intraoperative data collection or post operative outcome. Therefore, a weight may be assigned to the preoperative plan parameter that is zero or substantially zero and / or it may not be considered in the future.

[0194] In various embodiments, the preoperative plan data may include a location of an incision or entry point for a procedure. The data may also include a number of incisions and / or size of incisions. Therefore, the access point, whether an incision in soft tissue of a subject, opening or positioning of an instrument in a non-living subject, or the like may be collected or determined. In various embodiments the planning or location of a number of size or the like of incisions may be based upon preoperative image data.

[0195] Further an internal region of interest (ROI) and how it is to be accessed with a trajectory to reach it may be determined. For example, trajectory from one or more of incisions to the ROI may be planned. The trajectory to access the region of interest may also include movement of tissues or portions within the subject, resection of portions of the subject, or the like.

[0196] Preoperative plan data may include a type of implantation member, such as the type of screw (e.g., fixed or movable head), resorbable implant, or the like. Implant placement may also be predetermined such as the positioning of one or more screws,A001 1698W001 placement of a cage for bone in growth or disc replacement, or the like. Also, implants may include multiple implant portions such as a rod to interconnect a plurality of screws. Thus, a rod shape, material, size, and the like may be preplanned.

[0197] A tissue removal or cutting may also be determined in the preoperative plan. The type or volume of tissue to be removed, the location of the removal, or the like may be determined. Further, an instrument for performing the tissue resection may be predetermined. Similarly or alternatively, various locations and identifications of no-go or caution regions may be determined. For example, no-go regions may include nerves or vasculature that are not to be resected, moved, or the like. Therefore, the identification of no-go regions may be made prior to a procedure. This may be based upon image data.

[0198] The preoperative plan may also include desired outcomes. For example, a spine angle or feature, such as lordosis, may be preplanned. Therefore, the planned lordosis may be based upon various features, such as implant type or location, bone resection, or the like.

[0199] Block 1320 may allow for the determination or the recall of various preoperative plan and data. The preoperative plan data may be saved and / or accessed by a processing system, as discussed further herein. Also, the accessing of the preoperative plan may be substantially automatic based upon the completion of a procedure and the identification of the user and / or subject.

[0200] The preoperative plan 1320 may be initially created by the user 72. In various embodiments, however, the process 1300 may be used to generate and / or augment the preoperative plan, as discussed for the herein. The preoperative plan may generally include preoperative data such as preoperative image data, diagnosis of the subject, prior conditions of the subject, and the like.A001 1698W001

[0201] During a procedure intra-procedure or intraoperative data may be collected in block 1340. The data collected may include at least: Final Pose of Implants, Tracked Movement, Tissue Removed, Registration of Images and Space, Tracked Subject (e.g. Vertebrae), Length of Procedure, Instruments Used or any combinations of these. This intraoperative data may include various data that is collected with one or more of the systems associated with the surgical suite 18 including the navigation system, the data related to the instrument 68, and the like. Various tracking devices, motion sensors, force or torque sensors, or the like may be associated with one or more portions of the surgical suite 18, such as the instrument 68, the robotic system 20, implants to be positioned in the subject 30, and the like.

[0202] The intraoperative data may be automatically collected and time stamped to allow for correlation to time during a procedure, a flow of a procedure, and the like. The data may also be manually input or selected, such as by the user 72, another assistant in the operating theatre, and the like. Various intraoperative data may also be used to assist in determining a run time of an instrument, such as a cutting instrument. This may be used to estimate a cutting time (e.g., length of time to achieve a selected cutting) or other cutting parameter (e.g., volume of tissue removal). The intraprocedural data, therefore, may include a time logger, a thermal sensor, a cycle counter, a controller (e.g., integrated power console (IPC)) . These various portions may generate signals that are received by a processor to evaluate and determine the various parameters, such as a run time, usage time, cutting volume, etc.

[0203] The intraoperative data may further be data that is not human understandable data. For example, a signal from a thermal sensor or an encoder may be a digital or analogue signal that is not, at least immediately, understandable by a human.A001 1698W001The processor may, therefore, evaluate the signal and generate a human understandable parameter. The parameter may be an immediately relevant parameter, such as a temperature output, or may be plan relevant parameter such as a range or temperatures or whether a limit is surpassed.

[0204] Nevertheless, the intraoperative data may generally be data that is collected regarding a specific procedure performed on a specific subject. Therefore, the preoperative plan may relate to the specific procedure and subject. However, the process 1300 may be used to assist in evaluating the plan for a specific patient or subject 30, and the like but may also be used to evaluate a plurality of data from a plurality of different subjects for assisting in generating a preoperative plan that may be recalled. In other words, a plurality of intraoperative data and outcomes may be evaluated relative to a plurality of preoperative plans to assist in generating a predictive model regarding a preoperative plan and its parameters and selected outcomes.

[0205] The intraoperative data in block 1340 may include a final pose of an implant positioned in the subject 30. The implant may be any appropriate member that is positioned in the subject 30, such as an anatomical implant or replacement, a screw or other portion positioned in the subject, or the like. The final pose of the implant may be determined based upon a tracking of an instrument, positioning the implant, a direct tracking of a tracking device associated with the implant, a measurement of the implant within the subject 30, or the like. In various embodiments, for example, a positioning of the screw, a spinal cage, a rod, or the like may be tracked or measured with a selected system. For example, the imaging system 80 may acquire an intraoperative image to assist in determining a final pose of an implant.A001 1698W001

[0206] Movements of an instrument or implant may also be tracked. The movement that is tracked may allow for a determination of a trajectory over a period of time of one or more portions, such as the instrument 68, and implant related to the instrument, or the like. Thus, an intraoperative trajectory may be determined by tracking one or more portions over time.

[0207] Intraoperative data may include tissues being removed, such as by tracking a volume of tissue being removed, calculating a volume of tissue being removed, a position of where tissue is removed, or the like. As discussed above, tissue volume may be determined in any appropriate manner.

[0208] The subject may also be tracked, such as individual portions thereof. For example, a tracking device may be associated with two or more vertebrae of the patient 30. The vertebrae may be tracked individually to determine a motion thereof in the navigation space and / or relative to one another. Additional image data may be registered to the images in navigation space. The registration of the images may allow for evaluation of movement of the portions of the subject relative to a preoperative plan. As discussed above, the preoperative plan may include preoperative images and these may be registered to the subject with appropriate registration techniques, as discussed above.

[0209] Additional intraoperative data may include a length of a procedure or portions of a procedure. For example, the length of time to form an incision, length of time to move an implant to a final position, length of time for positioning a second implant relative to a first implant, or the like. Further, a categorization or identification of use of one or more instruments may also be used. The intraoperative data may, therefore, include an evaluation or collection of data regarding actions taken regarding the subject 30 and / or portions used to perform the actions on the subject 30.A001 1698W001

[0210] The interoperative data may be collected during the procedure. The data may be stored for various purposes, such as with the cloud system 112 or another appropriate memory system, such as the memory 103. The preoperative plan may then be correlated to the intraoperative data in block 1350. The correlation of the preoperative plan to the intraoperative data may be an evaluation of the preoperative plan relative to the intraoperative data and to determine any correlations or predictions there between. Various models may be used to predict intraoperative data based upon the preoperative plan. Further intraoperative data may be used to correlate or predict a success of achieving the preoperative plan. Therefore, a correlation of the preoperative plan and the intraoperative data may occur in block 1350 separate from any other correlation. Again the correlation from block 1350 may be used to generate a model, as discussed further herein.

[0211] Post-operative data may also be collected in block 1360. The post-operative data may also be used to generate a selected model, as also discussed. The post operative data may include User Assessment, Comparison to Other Subjects, Subject Assessment, Measurements of the Subject (e.g., Post-Op Variables, Range of Motion etc.), and Secondary Assessment or any combinations of these. The post-operative data may relate to the evaluation of the subject 30 after the procedure is complete, such as a procedure to position an implant in the subject 30. The post-operative data may include an assessment by the user, an assessment by the subject, or a secondary assessment.

[0212] User assessments may include the user 72 assessing the success or outcomes of the procedure. The assessment may include a subjective assessment of the subject 30 following the procedure. The subject assessment may be an assessment of the subject by the subject. The subject assessment may include a level of pain, level ofA001 1698W001 mobility or activity achievement, or the like. The secondary assessment may be an assessment by an individual or member other than the user 72 or the subject 30. For example, secondary assessment may include a family or teammate assessments.

[0213] Assessments may include measurements of the subject. The measurements of the subjects may include range of motion, pain, anatomical features, and the like. The measurements may be made with instrumentation positioned relative to the subject such as a measuring device to measure an amount of movement of a limb. Further, various systems such as monitoring systems may be worn by the subject to acquire data at selected periods, such as substantially continuously for a period of time, once the procedure is complete. The post-operative data may be collected in block 1360 for a selected period of time.

[0214] The post-operative data may include an achievement of predetermined or preplanned parameters. Further, the post-operative data may be a comparison to other subjects having undergone a similar or identical procedure. The post-operative data and the correlation of the preoperative plan and the intraoperative data may then be used to determine a correlation to the post-operative data in block 1370. The correlation in block 1370 may be a calculation between each of the post-operative data and one or more of the preoperative plan and / or interoperative data. Based upon the correlation a model may be generated in block 1380.

[0215] The model generated in block 1380 may be any appropriate model, as discussed above. The model generated may be automatically generated. For example, the model may be generated by executing instructions with a processor based on the various inputs and data, such as the recalled data from block 1320 (including a pre-op plan or a parameter related thereto), intra-op data form block 1340, post-op data fromA001 1698W001 block 1360, other appropriate data, or combinations thereof. The instructions may be appropriate instructions, as discussed herein, operable to determine a correlation and / or determine differences for use in generated a model that reduces various parameters. Reduced parameters may be differences between the pre-op plan and intra-op data, or inefficiencies (e.g., movement of an instrument or run time), etc.

[0216] The model generated 1380 may be a linear predictive model, a curve predictive model, or the like. Various non-curve predictive models may include machine learning models. Machine learning models may be trained with the data collected in the various blocks, such as 1320, 1340, and 1360. Thus, new or additional data may be used to reinforce or alter the training. The model may be used to generate a plan for a procedure, such as a future plan for a future procedure. The future plan may be evaluated by a user prior to implementation.

[0217] In machine learning models for example, CNNs or transformers, weights and other measures may be used to determine a correlation between various inputs and evaluations thereof to generate a predictive output model. The generated model in block 1380, therefore, may use the input data, such as the preoperative plan and the intraoperative data and the correlation to the post-operative data to generate a predictive model between the recalled preoperative plan of block 1320 and the intraoperative data in block 1340 to predict a post operative outcome.

[0218] The model may be used to evaluate or predict intraoperative data or preoperative plans that would likely achieve selected post-operative data outcomes, such as a selected range of motion, subject assessment, or the like. In various embodiments, for example, the generated model in block 1380 may be used to determine a speed ofA001 1698W001 tissue removed, positioning of one or more implants, or the like to predict a range of motion post operative data outcome that is in a selected range.

[0219] The generated model may then be output in block 1384. The output model in block 1384 may be output for various purposes, such as stored for later access. The output mode, therefore, may be output into a memory system to assist in evaluating various data, such as a preoperative plan.

[0220] In various embodiments, the output model from block 1384 may be accessed in block 1390 to evaluate a preoperative plan and / or set intraoperative parameters. For example, the output model from 1384 may be used to evaluate a preoperative plan to predict possible post operative data outcomes. Thus, the model may be used to predict or determine likely outcomes based upon the preoperative plan. Further, the correlation of the preoperative plan to intraoperative data and post operative data may be used to assist in setting intraoperative parameters and / or preoperative plans to achieve selected outcomes or have a selected probability of choosing a postoperative outcome.

[0221] In various embodiments, the process 1300 may also be used to update the model or determine whether the model is to be updated in block 1394. The model may be chosen to be updated and, thus, a yes path 1396 may be followed to recall a preoperative data or plan in block 1320. The process 1300 may, therefore, be iterated to generate a model or update a model based upon additional data such as additional preoperative data, additional intraoperative data, or additional post-operative data. This may also reinforce and / or alter the model based on the additional data. The additional data may be from one or more subjects in addition to a current subject. Further, a current subject may be evaluated based upon a prior generated model, such as accessed in block 1390.A001 1698W001

[0222] If it is determined that the model is not to be updated, a no path 1400 may followed to end the process 1300 in end block 1404. Ending the process 1300 may include storing the model or preparing the model for later access and / or deploying a model for use. Nevertheless, ending the process 1300 may again be understood to not require a complete cessation of operation of the process 1300 but may allow for further application and collection of data, as discussed above. Thus, the process 1300 may be continued for an appropriate period of time and / or restarted.

[0223] As discussed above, data may be collected during a procedure, after a procedure, etc. All of the data may be used for various purposes, such as efficacy in a future procedure or future procedure plan. The future procedure plan may be used to a future procedure, which may then be a current procedure. The future procedure plan and procedure may have efficacy in light of the previous collected data. The data may also be used to assist or create efficacy in a future procedure such as a selected outcome, procedure duration, motions during a procedure, procedure task order, tool selection, procedure outcomes, implant placement, etc. Further, the data may be used to assist in procedure, such as future, procedure such as speed, cutting speed, etc.

[0224] Examples. The following examples provide various embodiments disclosed herein.

[0225] Example 1 - An image acquisition and display system comprising: an image capture system having a field of view of a region of interest (ROI) of a subject and acquire image data of the ROI of the subject during a procedure; a display system configured to display a selected image generated at least in part by the image data from the image capture system; a communication system to communicate the image data in real time as real time image data from the image capture system of the ROI of the subject to aA001 1698W001 processor system; wherein the processor system is configured to execute instructions to, evaluate the real time image data for a selected parameter, generate the selected image for display based on the real time image data and the evaluation of the real time image data, and output the evaluation of the selected parameter with at least one of (1 ) the generated selected image, (2) a graphical representation relative to the generated selected image, or combinations thereof.

[0226] Example 2 - The system of Example 1 , wherein the selected parameter is input by a user during the procedure.

[0227] Example 3 - The system of Example 1 , wherein the selected parameter is predetermined and recalled during the procedure.

[0228] Example 4 - The system of Example 1 , wherein the selected parameter includes a motion of the instrument; wherein the motion of the instrument is evaluated for at least a wearable instrument.

[0229] Example 5 - The system of Example 1 , wherein the selected parameter includes an interaction of the instrument with at least a portion of the subject; wherein the interaction of the instrument is evaluated for at speed, clean cut, or combinations thereof.

[0230] Example 6 - The system of Example 1 , wherein the selected parameter includes a position of the instrument.

[0231] Example 7 - The system of Example 1 , wherein the selected parameter includes a selected color in the real time image; wherein the amount of the selected color in the real time image is operable to evaluate a volume of blood in the ROI.

[0232] Example 8 - The system of Example 1 , further comprising: a robotic mount system having an end effector to move the image capture system automatically to maintain the ROI in the field of view of the image capture system; wherein the parameterA001 1698W001 includes determining whether the view is clear of an obstruction and the processor system is configured to execute instructions to move the robotic system to move the image capture system to maintain the field of view of the ROI.

[0233] Example 9 - The system of Example 1 , wherein the generated selected image for display is a magnified view of the ROI.

[0234] Example 10 - The system of Example 1 , wherein the display system includes a user wearable display system having a first view portion and a second view portion; wherein the first view portion displays a magnified generated selected image and the second view portion at least one of is a direct view of the subject or a non-magnified image.

[0235] Example 11 - The system of Example 1 , wherein the image capture system includes at least a first image capture portion and a second image capture portion; wherein the first image capture portion and the second image capture portion are operable to generate a three dimensional image of the ROI.

[0236] Example 12 - A method of acquiring an image and display a generated image with a display system, the method comprising: acquiring image data of a region of interest (ROI) of a subject during a procedure with an image capture system having a field of view of a region of interest (ROI) of the subject; displaying a selected image generated at least in part by the image data from the image capture system; communicating the image data in real time as real time image data from the image capture system of the ROI of the subject to a processor system; executing instructions with the processor system to, evaluate the real time image data for a selected parameter, generate the selected image for display based on the real time image data and the evaluation of the real time image data, and output the evaluation of the selected parameter with at least one of (1 ) theA001 1698W001 generated selected image, (2) a graphical representation relative to the generated selected image, or combinations thereof.

[0237] Example 13 - The method of Example 12, further comprising: recalling during the procedure the selected parameter that is predetermined.

[0238] Example 14 - The method of Example 12, further comprising: evaluating the selected parameter that includes a motion of the instrument for at least wearable instrument.

[0239] Example 15 - The method of Example 12, further comprising: evaluating the selected parameter that includes an interaction of the instrument with at least a portion of the subject; wherein the interaction of the instrument is evaluated for at speed, clean cut, or combinations thereof.

[0240] Example 16 - The method of Example 12, further comprising: evaluating the selected parameter that includes a selected color in the real time image; determining a volume of blood in the ROI based on the amount of the selected color in the real time image.

[0241] Example 17 - The method of Example 12, further comprising: automatically moving the image capture system with a robotic mount system having an end effector to maintain the ROI in the field of view of the image capture system; and evaluating the selected parameter that includes determining whether the view is clear of an obstruction and the processor system is configured to execute instructions to move the robotic system to move the image capture system to maintain the field of view of the ROI.

[0242] Example 18 - The method of Example 12, further comprising: generating a magnified view of the generated selected image for display.A001 1698W001

[0243] Example 19 - The method of Example 12, further comprising: providing the display system as a user wearable display system having a first view portion and a second view portion; wherein the first view portion displays a magnified generated selected image and the second view portion at least one of is a direct view of the subject or a nonmagnified image.

[0244] Example 20 - The method of Example 12, further comprising: providing the image capture system to includes at least a first image capture portion and a second image capture portion; and generating a three dimensional image of the ROI with the first image capture portion and the second image capture portion.

[0245] Example 1 E - An image acquisition and display system comprising: an image capture system having a field of view of a region of interest (ROI) of a subject and acquire image data of the ROI of the subject during a procedure; a display system configured to display a selected image generated at least in part by the image data from the image capture system; a communication system to communicate the image data in real time as real time image data from the image capture system of the ROI of the subject to a processor system; wherein the processor system is configured to execute instructions to, evaluate the real time image data for a selected parameter, generate the selected image for display based on the real time image data and the evaluation of the real time image data, and output the evaluation of the selected parameter with at least one of (1) the generated selected image, (2) a graphical representation relative to the generated selected image, or combinations thereof.

[0246] Example 2E - The system of Example 1 E, wherein the selected parameter is input by a user during the procedure.A001 1698W001

[0247] Example 3E - The system of any one of the preceding examples, wherein the selected parameter is predetermined and recalled during the procedure.

[0248] Example 4E - The system of any one of the preceding examples, wherein the selected parameter includes a motion of the instrument; wherein the motion of the instrument is evaluated for at least wear of the instrument.

[0249] Example 5E - The system of any one of the preceding examples, wherein the selected parameter includes an interaction of the instrument with at least a portion of the subject; wherein the interaction of the instrument is evaluated for at speed, clean cut, or combinations thereof.

[0250] Example 6E - The system of any one of the preceding examples, wherein the selected parameter includes a position of the instrument.

[0251] Example 7E - The system of any one of the preceding examples, wherein the selected parameter includes a selected color in the real time image; wherein the amount of the selected color in the real time image is operable to evaluate a volume of blood in the ROI.

[0252] Example 8E - The system of any one of the preceding examples, further comprising: a robotic mount system having an end effector to move the image capture system automatically to maintain the ROI in the field of view of the image capture system; wherein the parameter includes determining whether the view is clear of an obstruction and the processor system is configured to execute instructions to move the robotic system to move the image capture system to maintain the field of view of the ROI.

[0253] Example 9E - The system of any one of the preceding examples, wherein the generated selected image for display is a magnified view of the ROI.A001 1698W001

[0254] Example 10E - The system of any one of the preceding examples, wherein the display system includes a user wearable display system having a first view portion and a second view portion; wherein the first view portion displays a magnified generated selected image and the second view portion at least one of is a direct view of the subject or a non-magnified image.

[0255] Example 1 1 E - A method of acquiring an image and display a generated image with a display system, the method comprising: acquiring image data of a region of interest (ROI) of a subject during a procedure with an image capture system having a field of view of a region of interest (ROI) of the subject; displaying a selected image generated at least in part by the image data from the image capture system; communicating the image data in real time as real time image data from the image capture system of the ROI of the subject to a processor system; executing instructions with the processor system to, evaluate the real time image data for a selected parameter, generate the selected image for display based on the real time image data and the evaluation of the real time image data, and output the evaluation of the selected parameter with at least one of (1 ) the generated selected image, (2) a graphical representation relative to the generated selected image, or combinations thereof.

[0256] Example 12E - The method of Example 11 E, further comprising: evaluating the selected parameter that includes an interaction of the instrument with at least a portion of the subject; wherein the interaction of the instrument is evaluated for at speed, clean cut, or combinations thereof.

[0257] Example 13E - The method of any one of the preceding examples 1 1 E-12E, further comprising: evaluating the selected parameter that includes a selected color in theA001 1698W001 real time image; determining a volume of blood in the ROI based on the amount of the selected color in the real time image.

[0258] Example 14E - The method of any one of the preceding examples 1 1 E-13E, further comprising: automatically moving the image capture system with a robotic mount system having an end effector to maintain the ROI in the field of view of the image capture system; and evaluating the selected parameter that includes determining whether the view is clear of an obstruction and the processor system is configured to execute instructions to move the robotic system to move the image capture system to maintain the field of view of the ROI.

[0259] Example 15E - The method of any one of the preceding examples 1 1 E-14E, further comprising: generating a magnified view of the generated selected image for display.

[0260] Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

[0261] Instructions may be executed by a processor and may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit encompasses a singleA001 1698W001 processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more modules.

[0262] The apparatuses and methods described in this application may be partially or fully implemented by a processor (also referred to as a processor module) that may include a special purpose computer (i.e., created by configuring a processor) and / or a general purpose computer to execute one or more particular functions embodied in computer programs. The computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may include a basic input / output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services and applications, etc.

[0263] The computer programs may include: (i) assembly code; (ii) object code generated from source code by a compiler; (iii) source code for execution by an interpreter; (iv) source code for compilation and execution by a just-in-time compiler, (v)A001 1698W001 descriptive text for parsing, such as HTML (hypertext markup language) or XML (extensible markup language), etc. As examples only, source code may be written in C, C++, C#, Objective-C, Haskell, Go, SQL, Lisp, Java®, ASP, Perl, Javascript®, HTML5, Ada, ASP (active server pages), Perl, Scala, Erlang, Ruby, Flash®, Visual Basic®, Lua, or Python®.

[0264] Communications may include wireless communications described in the present disclosure can be conducted in full or partial compliance with one or more IEEE standard 802, including IEEE standard 802.11-2012, IEEE standard 802.16-2009, and / or IEEE standard 802.20-2008. In various implementations, IEEE 802.11 -2012 may be supplemented by draft IEEE standard 802.11ac, draft IEEE standard 802.11 ad, and / or draft IEEE standard 802.11 ah.

[0265] A processor, processor module, module or ‘controller’ may be used interchangeably herein (unless specifically noted otherwise) and each may be replaced with the term ‘circuit.’ Any of these terms may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system- on-chip.

[0266] Instructions may be executed by one or more processors or processor modules, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logicA001 1698W001 arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” or “processor module” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0267] The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the invention, and all such modifications are intended to be included within the scope of the invention.

Claims

A001 1698W001CLAIMSWhat is claimed is:1 . An image acquisition and display system comprising: an image capture system having a field of view of a region of interest (ROI) of a subject and acquire image data of the ROI of the subject during a procedure; a display system configured to display a selected image generated at least in part by the image data from the image capture system; a communication system to communicate the image data in real time as real time image data from the image capture system of the ROI of the subject to a processor system; wherein the processor system is configured to execute instructions to, evaluate the real time image data for a selected parameter, generate the selected image for display based on the real time image data and the evaluation of the real time image data, and output the evaluation of the selected parameter with at least one of (1 ) the generated selected image, (2) a graphical representation relative to the generated selected image, or combinations thereof.

2. The system of Claim 1 , wherein the selected parameter is input by a user during the procedure.

3. The system of any one of the preceding claims, wherein the selected parameter is predetermined and recalled during the procedure.

4. The system of any one of the preceding claims, wherein the selected parameter includes a motion of the instrument;A001 1698W001 wherein the motion of the instrument is evaluated for at least wear of the instrument.

5. The system of any one of the preceding claims, wherein the selected parameter includes an interaction of the instrument with at least a portion of the subject; wherein the interaction of the instrument is evaluated for at speed, clean cut, or combinations thereof.

6. The system of any one of the preceding claims, wherein the selected parameter includes a position of the instrument.

7. The system of any one of the preceding claims, wherein the selected parameter includes a selected color in the real time image; wherein the amount of the selected color in the real time image is operable to evaluate a volume of blood in the ROI.

8. The system of any one of the preceding claims, further comprising: a robotic mount system having an end effector to move the image capture system automatically to maintain the ROI in the field of view of the image capture system; wherein the parameter includes determining whether the view is clear of an obstruction and the processor system is configured to execute instructions to move the robotic system to move the image capture system to maintain the field of view of the ROI.

9. The system of any one of the preceding claims, wherein the generated selected image for display is a magnified view of the ROI.

10. The system of any one of the preceding claims, wherein the display system includes a user wearable display system having a first view portion and a second view portion;A001 1698W001 wherein the first view portion displays a magnified generated selected image and the second view portion at least one of is a direct view of the subject or a non-magnified image.

11. A method of acquiring an image and display a generated image with a display system, the method comprising: acquiring image data of a region of interest (ROI) of a subject during a procedure with an image capture system having a field of view of a region of interest (ROI) of the subject; displaying a selected image generated at least in part by the image data from the image capture system; communicating the image data in real time as real time image data from the image capture system of the ROI of the subject to a processor system; executing instructions with the processor system to, evaluate the real time image data for a selected parameter, generate the selected image for display based on the real time image data and the evaluation of the real time image data, and output the evaluation of the selected parameter with at least one of (1 ) the generated selected image, (2) a graphical representation relative to the generated selected image, or combinations thereof.

12. The method of Claim 1 1 , further comprising: evaluating the selected parameter that includes an interaction of the instrument with at least a portion of the subject; wherein the interaction of the instrument is evaluated for at speed, clean cut, or combinations thereof.A001 1698W00113. The method of claims 11 or 12, further comprising: evaluating the selected parameter that includes a selected color in the real time image; determining a volume of blood in the ROI based on the amount of the selected color in the real time image.

14. The method of any one of claims 1 1-13, further comprising: automatically moving the image capture system with a robotic mount system having an end effector to maintain the ROI in the field of view of the image capture system; and evaluating the selected parameter that includes determining whether the view is clear of an obstruction and the processor system is configured to execute instructions to move the robotic system to move the image capture system to maintain the field of view of the ROI.

15. The method of any one of claims 1 1-14, further comprising: generating a magnified view of the generated selected image for display.

Images