Generating and modeling protocols for material removal procedures

Abstract

In some implementations, a device may obtain input information that indicates one or more outcome parameters for a procedure, and wherein the procedure is associated with one or more anatomical elements. The device may determine, based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements. The device may generate, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements. The device may provide the three-dimensional model.

Description

GENERATING AND MODELING PROTOCOLS FOR MATERIAL REMOVAL PROCEDURESCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority from U.S. Provisional Patent Application Serial No. 63 / 662,205, filed 20 June 2024, the entire content of which is incorporated herein by reference.BACKGROUND

[0002] In some surgical ecosystems, the convergence of connected technology and intelligent implants has enabled a shift toward more predictable procedures and improved patient outcomes in medical interventions. Through interconnected systems, surgical devices and instruments may communicate seamlessly, offering real-time data exchange and precise control over surgical procedures. The convergence of, and connectivity between, multiple components in a connected surgical ecosystem enables precise navigation and imaging, providing surgeons with improved insight into anatomical structures and facilitating improved planning and execution of surgical steps. This connected technology may enhance surgical precision and enable a more proactive approach to patient care, allowing for early detection of complications and personalized medical interventions. The integrated surgical ecosystem enables safer, more efficient procedures while providing surgeons with improved insights into patient physiology.

[0003] In some examples, the connected surgical ecosystem may include one or more surgical robots that facilitate robotically-assisted procedures, such as spinal procedures or other procedures. Surgical robots may assist a surgeon or other medical provider in carrying out a robotically-assisted procedure, and / or may complete one or more surgical procedures autonomously. Providing controllable linked articulating members allows a surgical robot to reach areas of a patient anatomy during various medical procedures.SUMMARY

[0004] Some implementations described herein relate to a device. The device may include one or more memories and one or more processors communicatively coupled to the one or more memories. The one or more processors may be configured to obtain input information that indicates one or more outcome parameters for a robotically-assisted procedure that is associated with a robot, wherein the robotically-assisted procedure is associated with one or more anatomical elements. The one or more processors may be configured to determine, based onthe one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements, and wherein at least one of the one or more surgical actions is configured to be performed via the robot. The one or more processors may be configured to generate, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements. The one or more processors may be configured to provide, for display via a user interface, the three-dimensional model. The one or more processors may be configured to receive, via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions. The one or more processors may be configured to modify the three-dimensional model to a modified three-dimensional model that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs. The one or more processors may be configured to provide, for display via the user interface, the modified three-dimensional model.

[0005] Some implementations described herein relate to a method. The method may include obtaining, by a device, input information that indicates one or more outcome parameters for a procedure, wherein the procedure is associated with one or more anatomical elements. The method may include determining, by the device and based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements. The method may include generating, by the device and based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements. The method may include providing, by the device for display via a user interface, the three-dimensional model.

[0006] Some implementations described herein relate to a non-transitory computer-readable medium that stores a set of instructions. The set of instructions, when executed by one or more processors of a device, may cause the device to obtain input information that indicates one or more outcome parameters for a procedure, wherein the procedure is associated with one or more anatomical elements. The set of instructions, when executed by one or more processors of the device, may cause the device to determine, based on the one or more outcome parameters andanatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements. The set of instructions, when executed by one or more processors of the device, may cause the device to generate, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements. The set of instructions, when executed by one or more processors of the device, may cause the device to provide the three-dimensional model.BRIEF DESCRIPTION OF THE DRAWINGS

[0001] For a better understanding of embodiments of the invention and to show how the same may be carried into effect, reference will now be made, purely by way of example, to the accompanying drawings in which like numerals designate corresponding elements or sections throughout. In the accompanying drawings:

[0007] Figs. 1A-1D are diagrams of an example implementation associated with generating and modeling protocols for robotically-assisted procedures.

[0008] Fig. 2 is a diagram of an example implementation associated with a user interface for modeling protocols for material removal procedures.

[0009] Figs. 3 A and 3B are diagrams of an example implementation associated with a user interface for modeling protocols for material removal procedures.

[0010] Fig. 4 is a diagram of an example implementation associated with a user interface for modeling protocols for material removal procedures.

[0011] Fig. 5 is a diagram of an example environment in which systems and / or methods described herein may be implemented.

[0012] Fig. 6 is a diagram of example components of a device associated with generating and modeling protocols for material removal procedures.

[0013] Fig. 7 is a flowchart of an example process associated with generating and modeling protocols for material removal procedures.

[0014] Fig. 8 is a flowchart of an example process associated with generating and modeling protocols for material removal procedures.

[0015] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, whereconsidered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.DETAILED DESCRIPTION

[0016] The following detailed description of example aspects refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.

[0017] Some procedures (e.g., surgical procedures or clinical procedures) may include material removal, such as bone removal or tissue removal, among other examples. As an example, a spinal decompression procedure may include removal of bone or tissue that is compressing one or more spinal nerves. As another example, a spinal fusion procedure may include the removal of one or more damaged or degenerated discs and the fusion of adjacent vertebrae (e.g., using bone grafts or implants). Other types of procedures may similarly include bone removal or tissue removal, among other examples. Preoperative planning for bone removal or tissue removal in clinical procedures may present many challenges.

[0018] For example, high-quality medical imaging data may be needed to enable accurate assessment of anatomical structures and pathology. However, interpreting the medical imaging data may be difficult because of image noise, artifacts, and / or variations in patient anatomy, among other examples. Additionally, the medical imaging data may include one or more two- dimensional images. Planning surgical actions (e.g., cuts, incisions, drilling actions, burring actions, material removal actions, or other surgical actions) for a bone removal or tissue removal procedure using two-dimensional images may result in inaccurate and / or unreliable planned surgical actions. For example, the lack of depth perception in two-dimensional images increases the difficulty to accurately assess the spatial relationships between anatomical elements. A surgeon may rely on experience and knowledge to mentally reconstruct the three- dimensional anatomical elements based on two-dimensional images, which can lead to inaccuracies and misinterpretations. Further, two-dimensional images may fail to fully capture complex anatomical elements, such as in regions in which overlapping tissues or structures obscure visibility. This limitation may increase the difficulty associated with delineating, during the preoperative planning, the boundaries of tissues or bone that is to be removed or preserved during the procedure, increasing the risk of suboptimal outcomes or complications resulting from the procedure.

[0019] In some examples, the procedure may be a robotically-assisted procedure. In robotically-assisted procedures, a robotic system (e.g., that includes one or more robots, where each robot includes one or more robotic arms) may be used to enhance surgical precision and dexterity. However, the use of robots for robotically-assisted procedures increases the difficulty and / or complexity and reduces the accuracy and reliability of preoperative planning and intraoperative decision making for the robotically-assisted procedures. For example, the robots used for the robotically-assisted procedures may have complex robotic kinematics. A robotic arm may have many degrees of freedom and intricate joint articulation. The complex robotic kinematics increases the difficulty and / or complexity of precisely predicting the maneuverability of a robotic arm within a unique anatomical context of a patient, thereby reducing the accuracy and / or reliability of preoperative planning. Further, the robots used for the robotically-assisted procedures may be associated with ultra-high precision robotic instruments which may increase the risk of inadvertent proximity or contact with sensitive structures of a patient, such as nerves or blood vessels (e.g., because the ultra-high precision robotic instruments may have increased sensitivity to even slight movements or changes and / or reduced margins for error resulting in minor deviations or miscalculations in position bringing the robotic instruments near or in contact with sensitive structures of the patient). The reduced accuracy and / or reliability of preoperative planning and intraoperative decision making may increase the risk of potential intraoperative complications caused by inadvertent proximity or contact with sensitive structures of the patient.

[0020] As another example, the utilization of coordinate systems for robotic control (e.g., that may not align with conventional human comprehension) in combination with the complex robotic kinematics may impede seamless understanding and / or collaboration between surgeons and robotic systems. The incompatibility between human-centric anatomical references and robot-based coordinate frames increases the complexity and / or difficulty of the interpretation of spatial relationships between a position or pose of a robot and / or a robotic arm and anatomical elements in the unique anatomical context of a patient. As a result, an accuracy and / or reliability of preoperative planning may be reduced because a surgeon may be unable to interpret the coordinate frames and / or navigation mechanism of the robot and / or robotic arm. Further, in some examples, a surgeon may have limited visibility and restricted access to a surgical site (e.g., in minimally invasive procedures), impeding an ability of the surgeon to directly observe and / or promptly react to movements or malfunctions of the robot and / or robotic arm. This increases the risk of potential errors or adverse events because the surgeon may not have real-time insight into actions or feedback of the robot and / or robotic arm.

[0021] Additionally, the challenges described above may be increased because of the inherent variability of patient anatomies. For example, the effectiveness of planned surgical actions for a material removal (e.g., a bone removal or tissue removal) procedure may be based on individualized geometric configurations and properties of an anatomy of a given patient. This variability introduces complexity in preoperative planning and control for material removal procedures because generic (or template-based) surgical actions, robotic trajectories, and / or manipulations may not universally accommodate diverse anatomical structures or pathologies of different patients. For example, planning the precise location, trajectory, and / or size of cuts, incisions, burrs, bores, and / or other surgical actions may result in unintended or undesired outcomes and / or increased risk of damage for the patient because of the challenges described herein and the variability of patient anatomies. Additionally, the variability of patient anatomies may increase the risk of damaging sensitive structures (e.g., nerves, blood vessels, organs, tissue, and / or sensory receptors) of a patient during a material removal procedure. For example, a surgeon may rely on inadequate two-dimensional images and their knowledge and experience to predict the location and proximity of sensitive structures. However, variability of patient anatomy and only factors, such as bone density, hardness, and / or other biomechanical properties may introduce additional complexities, increasing the risk of inaccurate selection or planning associated with surgical actions, surgical tools, and / or techniques, among other examples.

[0022] As a result, a surgeon may initiate a material removal procedure that is associated with preoperative planning that is inaccurate and / or unreliable. This may consume resources (e.g., computing resources, processing resources, and / or energy resources) associated with the operation of one or more robotic systems, navigation systems, and / or imaging devices used to perform the material removal procedure that has reduced effectiveness and / or efficacy for a given patient. Additionally, this increases the risk of undesired or negative outcomes for the patient. Further, due to the challenges described above, intraoperative decision making by the surgeon may be degraded, further reducing the effectiveness and / or efficacy of the material removal procedure and / or increasing the risk of undesired or negative outcomes for the patient of the material removal procedure.

[0023] Some aspects described herein enable a planning device to generate and / or model protocols (e.g., surgical plans or one or more planned surgical actions) for material removal procedures, such as bone removal procedures or tissue removal procedures. In some aspects, the planning device may use a data driven approach to generate and / or model patient specific protocols for a material removal procedure based on one or more desired outcomes of thematerial removal procedure, historical procedure information, surgeon profile information, anatomical information of the patient, and / or configuration information of a robotic system, among other examples. For example, the protocol may include one or more surgical actions to be performed to the one or more anatomical elements to accomplish the material removal procedure for the patient.

[0024] In some aspects, the planning device may obtain input information that indicates one or more outcome parameters for a material removal procedure. The one or more outcome parameters may indicate a goal or clinical outcome to be achieved via the material removal procedure. The input information may indicate one or more anatomical elements associated with the material removal procedure (e.g., one or more anatomical element from which material, such as bone or tissue, is to be removed). In some aspects, the material removal procedure may be a robotically-assisted procedure. The planning device may determine, based on the one or more outcome parameters and anatomical information of a patient, a protocol (e.g., a surgical plan) to be performed to the one or more anatomical elements to accomplish the material removal procedure in accordance with the one or more outcome parameters. In some aspects, at least one of the one or more surgical actions may be configured to be performed via a robot (e.g., with the assistance of, or by, the robot). For example, the protocol may indicate one or more surgical actions (e.g., cuts, incisions, burrs, bores, and / or other surgical actions) and location information of respective surgical actions in context of patient-specific anatomy. For example, the location information may indicate a location, size, trajectory, depth, and / or path of respective surgical actions of the one or more surgical actions. The planning device may determine the location information based on the one or more outcome parameters, the anatomical information, and / or the input information, among other examples.

[0025] In some aspects, the planning device may generate a three-dimensional model of the one or more anatomical elements that shows the one or more surgical actions made to the one or more anatomical elements. For example, the planning device may generate the three- dimension model based on, or using, the anatomical information of the patient (e.g., to model the one or more anatomical elements in accordance with the unique anatomical information of the patient). The planning device may generate the three-dimensional model with one or more features that depict or show the one or more surgical actions (e.g., indicated via the generated protocol for the material removal procedure). For example, the planning device may determine and / or generate the one or more features based on, or using, the location information of respective surgical actions of the one or more surgical actions.

[0026] The planning device may provide, for display via a user interface, the three- dimensional model. In some aspects, the planning device may receive, via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions. For example, a surgeon or other surgical team member may provide a user input indicating the one or more modifications (e.g., modifying a surgical action and / or location information associated with a given surgical action). As an example, a user input may include a surface selection (e.g., a superior and inferior surface selection, a medial and lateral surface selection, and / or another surface selection), an indication of an area or surgical action to be modified, and / or an indication of modified location information for a given surgical action, among other examples. In some aspects, the one or more user inputs may be provided via different views of the three-dimensional model. The planning device may determine one or more modified surgical actions based on the one or more user inputs, the one or more outcome parameters, and / or the anatomical information of the patient, among other examples. The planning device may modify the three-dimensional model to a modified three-dimensional model based on the one or more modified surgical actions. For example, the modified three- dimensional model may include one or more modified features that depict respective modified surgical actions. The planning device may provide, for display via the user interface, the modified three-dimensional model.

[0027] As a result, preoperative planning for material removal procedures (e.g., bone removal procedures and / or tissue removal procedures) may be improved. For example, by enabling the planning device to determine the protocol using the anatomical information of the patient, the one or more outcome parameters, and / or the configuration information of the robot, the planning device may model or simulate one or more planned surgical actions using information that is specific to a given patient and / or a current configuration of the robot. This enables the planning device to model or simulate planned surgical actions in relation to anatomical information of the patient, thereby improving the accuracy and / or reliability of preoperative planning for material removal procedures. In some aspects, by using the one or more outcome parameters to generate the protocol, the planning device may improve a surgical plan (e.g., may optimize the one or more planned surgical actions) in accordance with a goal or clinical outcome desired for the material removal procedure. For example, the planning device may determine the protocol to improve values (e.g., maximize or minimize, depending on the context) for respective outcome parameters in connection with the anatomical information of a specific patient and / or the configuration information of a robot to be used to facilitate or perform the material removal procedure. This enables more accurate and / or reliable surgicalplans for material removal procedures, thereby reducing the likelihood that a material removal procedure is initiated in which the material removal procedure cannot be successfully performed (e.g., because of patient anatomy or other factors) and / or that will result in unintended or negative outcomes for the patient. This may conserve resources (e.g., processing resources, computing resources, memory resources, and / or power resources) that would have otherwise been associated with performing the material removal procedure, such as by a robotic system, a navigation system, one or more imaging devices, and / or other devices used as part of the material removal procedure. Further, this reduces the likelihood of unintended or negative outcomes for the patient of the material removal procedure that may have otherwise occurred with less accurate and / or less reliable preoperative planning.

[0028] In some aspects, by modeling the one or more anatomical elements with the one or more planned surgical actions shown in a three-dimensional model, the planning device may provide more insightful and / or useful information for a surgeon during preoperative planning for material removal procedures. For example, the three-dimensional model may enable a surgeon to view one or more surgical actions for a material removal procedure in context of a patient-specific three-dimensional model of one or more anatomical elements. This enables the surgeon to quickly and easily identify any modifications to be made to the protocol (e.g., to the one or more surgical actions) and / or to approve the protocol for the material removal procedure. This conserves resources (e.g., processing resources, computing resources, memory resources, and / or power resources) and / or time that would have otherwise been associated with the surgeon using one or more devices or systems to analyze multiple images (e.g., two- dimensional images), review historical information of previously performed procedures, review other anatomical information of the patient, determine the protocol for the material removal procedure, and / or estimate or predict whether the protocol is achievable or appropriate for a given patient, among other examples. Additionally, by providing the three-dimensional model of one or more anatomical elements showing the one or more planned surgical actions, the planning device may enable a surgeon or other surgical team member to make more accurate and / or reliable determinations as to whether a protocol (e.g., a surgical plan) for a material removal procedure is achievable or appropriate for a given patient. This reduces the likelihood of unintended or negative outcomes for the patient of the material removal procedure that may have otherwise occurred with less accurate and / or reliable preoperative planning.

[0029] Additionally, by enabling a user (e.g., a surgeon or other surgical team member) to interact with the three-dimensional model via the user interface (e.g., to make one or more modifications to the protocol via the user interface), the planning device may reduce thecomplexity and / or conserve resources (e.g., processing resources, computing resources, memory resources, and / or power resources) that would have otherwise been associated with modifying a protocol (e.g., a surgical plan) for a material removal procedure. For example, by enabling the planning device to determine modification to the surgical plan from user inputs to the user interface displaying the three-dimensional model, the planning device may determine one or more modified surgical actions based on the user input(s), the anatomical information of the patient, the one or more outcome parameters, and / or the configuration information of the patient. This enables improved and / or more accurate modification to planned surgical actions for a material removal procedure, thereby enabling the material removal procedure to have a higher likelihood of success and / or a higher likelihood of achieving one or more goals or desired clinical outcomes.

[0030] Figs. 1A-1D are diagrams of an example implementation 100 associated with generating and modeling protocols for robotically-assisted procedures. As shown in Figs. 1A- 1D, example implementation 100 includes a planning device, a client device, an image database, a robot (e.g., that includes one or more robotic arms), a display device, and a navigation system (e.g., of the robot). These devices are described in more detail below in connection with Fig. 5 and Fig. 6.

[0031] As shown in Fig. 1A, and by reference number 105, the planning device may obtain input information for a procedure (e.g., a surgical procedure or clinical procedure), such as a material removal procedure. In some aspects, the planning device may obtain the input information via the client device and / or via an input to the display device, among other examples. A material removal procedure may be a procedure in which bone, tissue, or other biological material is removed from one or more anatomical elements of a patient. In some aspects, the procedure may be a robotically-assisted procedure (e.g., that includes one or more operations or steps performed via the robot). The robotically-assisted procedure may be associated with the robot. For example, the robotically-assisted procedure may be associated with the robot in that the robot (and / or the one or more robotic arms) is to perform one or more operations during the robotically-assisted procedure. In other aspects, the procedure may be a surgical procedure that does not include the use of the robot. The procedure may include anatomical material removal. For example, the procedure may include bone removal, tissue removal, or another type of anatomical material removal. In some examples, the procedure may be a spinal procedure (e.g., a spinal alignment, installing implants, osteotomy, fusion, and / or any other spinal procedure).

[0032] In some aspects, the input information may include a surgical plan for the procedure. The surgical plan may include one or more steps for performing the procedure and / or one or more expected thresholds for monitoring one or more parameters during the procedure, among other examples. For example, the surgical plan may indicate one or more planned operations for the procedure, such as one or more actions or clinical outcomes to be achieved for a patient via the procedure. In some aspects, the surgical plan may indicate one or more planned operations, such as a movement, a path, a trajectory, and / or a surgical action (e.g., a cut, an incision, a tissue or bone removal, among other examples), among other examples, to be performed by the robot (e.g., by a robotic arm and / or by an implement or surgical tool configured for a robotic arm) and / or to be performed by a surgeon.

[0033] The planning device may obtain the surgical plan via a user input. For example, a user (e.g., a surgeon or another user) may input the surgical plan to the planning device for protocol or planned surgical actions (e.g., planned incisions or cuts for the procedure), as described in more detail elsewhere herein. For example, the user may provide one or more user inputs (e.g., to a user interface, such as the user interface described in more detail elsewhere herein) indicating the surgical plan. The one or more user inputs may indicate one or more outcome parameters (e.g., as described in more detail elsewhere herein), a type or category of the material removal procedure, one or more anatomical elements associated with the material removal procedure, and / or one or more surgical actions to be performed for the material removal procedure, among other examples. In some aspects, a user input may include a touch input (e.g., to the user interface), a gesture input, a voice input, and / or another input. For example, a user input may be provided via an input component of the planning device or the display device, such as a mouse, a keyboard, a touchscreen, or another input component.

[0034] In some aspects, the input information may indicate a type or category of procedure (e.g., a type or category of the procedure) to be performed. For example, the input information may indicate that the material removal procedure is a spinal alignment procedure, an implant installation procedure, osteotomy procedure, a fusion procedure, and / or another type of procedure. Additionally, or alternatively, the input information may indicate one or more anatomical elements associated with the material removal procedure. For example, the input information may indicate which anatomical elements are to be operated on, interacted with, or are otherwise involved with the procedure. For example, for a spinal procedure, the surgical plan may indicate which lumbar vertebrae (e.g., the LI vertebrae, the L2 vertebrae, the L3 vertebrae, and / or another vertebrae) are to be operated on as part of the procedure.

[0035] In some aspects, the planning device may determine the one or more planned operations (e.g., one or more surgical actions) for the material removal procedure based on, or otherwise associated with, the type or category of the procedure and / or the one or more anatomical elements associated with the procedure, among other examples. For example, the planning device may determine or obtain a template or generic plan that is configured or designed to accomplish the type or category of procedure and for the one or more anatomical elements.

[0036] In some aspects, the template or generic plan may be associated with a surgeon or clinician who will be performing the procedure. For example, the template or generic plan may be specific to the surgeon or the clinician. In other words, the surgeon or the clinician may design or configure the surgical plan (e.g., the template or generic plan) in accordance with one or more preferences or techniques desired by the surgeon or the clinician. In such examples, the planning device may obtain the template or generic plan from a profile of the surgeon or the clinician (e.g., a clinician profile). For example, an input to the planning device may include an identifier of the surgeon or the clinician (e.g., a name, account identifier, or other identifier). The planning device may obtain the profile for the surgeon or the clinician using the identifier. The planning device may obtain a surgical plan from, or based on, the profile for the surgeon or the clinician.

[0037] In some aspects, the input information may include or indicate one or more outcome parameters. An outcome parameter may indicate a goal or clinical outcome for the procedure. In some example, an outcome parameter may be a parameter to be optimized and / or considered by the planning device when generating a protocol (e.g., a surgical plan) for the procedure, as described in more detail elsewhere herein. For example, an outcome parameter may include a quantity of surgical actions (e.g., a quantity of cuts, incisions, bores, burrs, or other surgical actions), an amount of time to complete the material removal procedure, a procedure- specific parameter (e.g., a parameter that is specific to the condition being treated by the material removal procedure), a percentage of decompression (e.g., for a decompression procedure), a neurological improvement, a pain relief improvement, a mobility and / or functionality improvement, a stability improvement (e.g., to stabilize an anatomical element), and / or a level of risk, among other examples.

[0038] For example, if an outcome parameter includes the percentage of decompression, then the planning device may generate a protocol for the material removal procedure that improves (e.g., optimizes or maximizes) the percentage of decompression achieved via the protocol for a given patient. As another example, if an outcome parameter is a neurological improvement,then the planning device may generate a protocol for the material removal procedure that improves (e.g., optimizes or maximizes) the neurological function of one or more anatomical elements of the given patient. The planning device may use the one or more outcome parameters as a guide for generating surgical actions to achieve or accomplish the material removal procedure and / or to determine location information (e.g., where to place a surgical action on an anatomical element of the patient, how large to make a surgical action, and / or a path or trajectory of a surgical action). This enables the planning device to generate protocols that are customized or tailored to the goal and / or clinical outcome for a material removal procedure that is desired by a surgeon. This may enable the planning device to generate more accurate and / or reliable protocols (e.g., surgical plans) for a material removal procedure (e.g., in context of what is expected or intended by a surgeon), thereby conserving resources (e.g., processing resources, computing resources, memory resources, and / or power resources) that would have otherwise been associated with a surgeon generating the protocol and / or making multiple modifications to the protocol, among other examples.

[0039] In some aspects, as shown by reference number 110, the planning device may obtain configuration information associated with the robot. For example, the planning device may obtain the configuration information associated with the robot based on, or otherwise associated with, the material removal procedure being a robotically-assisted procedure. For example, if the procedure is not a robotically-assisted procedure, then the planning device may not obtain and / or otherwise use the configuration information. The planning device may obtain the configuration information from the robot and / or from another source, such as a database, a server device associated with the robot, and / or the navigation system, among other examples.

[0040] The configuration information may indicate one or more configurations, capabilities, and / or other information for the robot and / or for the one or more robotic arms. For example, the configuration information may indicate one or more parameters that define one or more capabilities of the robot, one or more operational characteristics of the robot, and / or one or more spatial constrains of robot, among other examples. For example, the configuration information may indicate a range of motion (e.g., of one or more robotic arms), a precision of movement supported by the robot, and / or a compatibility with one or more surgical tools or implements, among other examples.

[0041] In some aspects, the configuration information may include configurations for respective robotic arms of the one or more robotic arms. For example, the configuration information may include a quantity of the one or more robotic arms, a kinematic structure of the one or more robotic arms, and / or one or more degrees of freedom of the one or more roboticarms, among other examples. In some aspects, the configuration information may indicate a spatial boundary in which the robot (and / or the one or more robotic arms) can safely operate. The spatial boundary may be referred to herein as a safety volume (e.g., indicating a volume of space in which the robot can safely operate). The safety volume may indicate an area in which the one or more robotic arms can safely move (e.g., an area that is reachable by the one or more robotic arms and that does not cause the robot and / or the one or more robotic arms to contact a human or another device). In some aspects, the spatial boundary may define or indicate a reachable workspace of the one or more robotic arms and / or any constraints that may limit a movement and / or availability of the one or more robotic arms, such as for given anatomical regions or elements.

[0042] In some aspects, the configuration information may include location information of the robot. The location information may indicate a position of the robot within the surgical field. For example, the location information may indicate a position of the robot and / or the one or more robotic arms relative to other devices, components, and / or objects in the surgical field. For example, the location information may indicate a position of the robot and / or the one or more robotic arms relative to a bed or platform where the patient is expected to be located (e.g., a bed of the patient), one or more imaging devices, the navigation system, and / or expected positions of a surgical team (e.g., one or more surgeons, doctors, nurses, clinicians, or other members of the surgical team) or position of the patient (e.g., a pose of the patient), among other exmaples.

[0043] The location information may be indicated via one or more coordinates and / or reference points. For example, the location information may be coordinates and / or orientations relative to a fixed reference point in the surgical field. This may enable the positioning device to determine a position and / or alignment of the robot, the robotic arm(s), and / or other objects within the surgical field.

[0044] As shown by reference number 115, the planning device may obtain medical imaging data of a patient. The patient may be associated with the material removal procedure. In some aspects, the planning device may obtain the medical imaging data via the image database (e.g., as shown in Fig. 1 A). Additionally, or alternatively, the planning device may obtain the medical imaging data via one or more imaging devices. The medical imaging data may include X-ray image data, X-ray -based image data (e.g., obtained via a fluoroscope, a computed tomography (CT) scanner, or other X-ray machine), magnetic resonance imaging (MRI) scanner-based image data, optical coherence tomography (OCT) scanner-based image data, positron emission tomography (PET) scanner-based data, ultrasound image data, endoscope image data,microscope image data, optical camera image data, thermographic camera (e.g., an infrared camera) image data, radar system (which may include a transmitter, a receiver, a processor, and one or more antennas) image data, and / or other image data. In some aspects, the medical imaging data may include one or more images, one or more videos, a feed or stream of images or video, and / or other image data. In some aspects, the medical imaging data may include two- dimensional (2D) image data, three-dimensional (3D) image data, and / or four-dimensional (4D) (e.g., 3D + time) image data.

[0045] For example, the input information (e.g., received by the planning device as described in connection with reference number 105) may identify the patient. For example, an input to the planning device may include an identifier (e.g., a name, a patient identifier, or another identifier) of the patient. The planning device may use the identifier to obtain the medical imaging data of the patient. For example, the planning device may perform a lookup operation via the image database using the identifier. In some other aspects, the medical imaging data may be provided to the planning device as an input (e.g., with, or in connection with, the surgical plan). The medical imaging data may enable the planning device to provide a patientspecific protocol and / or modeling of the protocol for the material removal procedure, as described in more detail elsewhere herein.

[0046] In some aspects, the planning device may analyze the medical imaging data to obtain anatomical information of the patient. For example, the planning device may determine, using medical imaging data of a patient of the material removal procedure, anatomical information of the patient. The anatomical information may include information of one or more anatomical elements associated with the material removal procedure. In some aspects, the planning device may generate the anatomical information using the medical imaging data. In some aspects, the planning device may extract the anatomical information from the medical imaging data.

[0047] In some aspects, the planning device may use a machine learning-based technique to analyze the medical imaging data. For example, the planning device may determine, generate, and / or obtain the anatomical information using a computer vision model or another model trained or configured to identify, determine, segment, or otherwise output anatomical information using medical imaging data. For example, the planning device may perform segmentation to partition medical imaging data into distinct regions corresponding to different anatomical elements (e.g., anatomical structures or tissue types). The segmentation may be based on intensity, texture, or shape characteristics, allowing for the planning device to identify organs, bones, blood vessels, and / or other relevant anatomical elements from the medical imaging data.

[0048] In some aspects, the planning device may perform anatomical localization to identify or determine locations of one or more anatomical elements in the anatomy of the patient. For example, the planning device may align the medical imaging data with preoperative or intraoperative reference frames. The planning device may detect and / or match corresponding anatomical features or markers between different image modalities or across sequential imaging time points, enabling accurate spatial alignment and combination of multiple imaging datasets. Additionally, the planning device may extract quantitative anatomical measurements and geometric parameters from the medical imaging data. The anatomical measurements and geometric parameters may provide insights into anatomical morphology, dimensions, and / or spatial relationships of one or more anatomical elements of the patient. The planning device may perform segmentation and / or identification of landmarks included in the one or more anatomical elements using one or more image processing operations (e.g., thresholding, active contouring, manual tracing, and / or geometric heuristics), and / or machine learning or deep learning models, among other examples.

[0049] For example, the anatomical information may include a size or volume of area occupied by the patient within the surgical field. The anatomical information may include spatial orientation and / or anatomical coverage of the patient. In some aspects, the anatomical information may include spatial orientation and / or anatomical coverage of the patient relative to a surgical bed or table or another reference point. In some aspects, the anatomical information may be based on the surgical plan in that the orientation of the patient (e.g., a supine position, a prone position, a lateral position, a lithotomy position, or another position) may be based on the surgical plan. In some aspects, the spatial orientation and / or anatomical coverage of the patient may be based on, or in relation to, a region of interest for the material removal procedure. For example, the spatial orientation and / or anatomical coverage of the patient may be based on, or in relation to, a one or more vertebral levels in the lumbar spine (e.g., the T10 to L5 vertebral levels).

[0050] The anatomical information may include information for one or more anatomical elements. For example, the anatomical information may include location information of respective anatomical elements within the anatomy of the patient. The location information may indicate a position, size, depth, and / or orientation, among other examples, of the anatomical elements. The one or more anatomical elements may include bone, tissue, vertebrae, ligaments, muscles, tendons, cartilage, joints, organs, blood vessels, and / or other anatomical elements. In some aspects, the anatomical information may include a bone density,a body mass index, a muscle mass, a body composition, an organ size and / or shape, vascular anatomy information, and / or neural anatomy information.

[0051] In some aspects, the anatomical information may include information for one or more sensitive structures of the patient. The one or more sensitive structures may be structures or elements in the anatomy of the patient that are vulnerable to damage or disruption and / or which may result in functional impairment or negative outcomes for the patient if damaged. For example, sensitive structures may have a delicate structure, a high degree of innervation, and / or critical physiological functions, among other examples. The one or more sensitive structures may include nerves, blood vessels, organs, tissue, and / or sensory receptors, among other examples. The anatomical information may include location information of the one or more sensitive structures. In some aspects, the location information of the one or more sensitive structures relative to one or more anatomical elements that are to be interacted with or removed during the material removal procedure.

[0052] In some aspects, the planning device may determine or obtain at least a portion of the anatomical information from one or more other sources (e.g., other than the medical imaging data). For example, the planning device may obtain an input indicating at least a portion of the anatomical information, such as a body mass index, a height, a gender, and / or a weight, among other examples, of the patient. The planning device may determine at least a portion of the anatomical information based on, using, or otherwise associated with information of the patient that is input to the planning device.

[0053] As shown in Fig. IB, and by reference number 120, the planning device may determine a protocol for the procedure (e.g., for the material removal procedure). For example, the planning device may determine one or more surgical actions to be performed during the material removal procedure (e.g., to accomplish the material removal procedure). As described elsewhere herein, a surgical action may include a cut, an incision, a bore, a burr, a material removal, and / or another type of surgical action. For example, the protocol may be an ordered series of steps, operations, and / or actions configured to accomplish and / or complete the material removal procedure.

[0054] The planning device may determine the protocol based on the anatomical information, the input information (e.g., the one or more outcome parameters), and / or the configuration information, among other examples. For example, the input information may indicate a surgical plan or a goal for the material removal procedure. The planning device may analyze the anatomical information of the patient and the one or more outcome parameters to determine the one or more surgical actions that accomplish the material removal procedure for the specificanatomical information of the patient and in accordance with the one or more outcome parameters.

[0055] The planning device may determine the protocol using a heuristics-based model (e.g., an algorithmic approach that utilizes heuristics) that uses one or more defined or configured rules to model, simulate, and / or otherwise obtain information associated with surgical actions performed via one or more anatomical elements of the patient. Additionally, or alternatively, the model may include a physics-based model that is configured to simulate the behavior of objects in accordance with the laws of physics. Additionally, or alternatively, the model may include an agent-based model that is configured to model or simulate autonomous agents that follow a set of rules to interact with an environment (e.g., the one or more or anatomical elements) in accordance with one or more defined behaviors (e.g., where one or more agents may include a surgeon, a clinician, and / or a robotic arm). Additionally, or alternatively, the model may include a probabilistic model that is configured to use probabilistic techniques, such as stochastic techniques, to represent uncertainty and variability.

[0056] Additionally, or alternatively, the model may include one or more machine learning models or artificial intelligence models that are trained to simulate and / or determine one or more surgical actions for the material removal procedure. For example, the one or more machine learning models or artificial intelligence models may be trained or configured to perform surgical action planning or determination using anatomical information, one or more outcome parameters, and / or other information described herein. The one or more machine learning models or artificial intelligence models may include one or more deep learning models configured to represent complex spatial relationships from information, such as the anatomical information, the configuration information (e.g., of the robot), and / or the medical imaging data, among other examples. For example, the planning device may provide the anatomical information, the medical imaging data, the input information, the one or more outcome parameters, and / or the configuration information, among other examples, as input to the model. The model may output the protocol that includes the one or more surgical actions (e.g., that are based on, or optimized in accordance with, the one or more outcome parameters).

[0057] In some aspects, the planning device may determine a template protocol based on the type of the material removal procedure (e.g., indicated by the input information). The template protocol may indicate one or more template surgical procedures to accomplish the material removal procedure. In some aspects, the template protocol may be associated with a surgeon or clinician who will be performing the procedure. For example, the template protocol may be specific to the surgeon or the clinician, as described elsewhere herein. In some aspects, thetemplate protocol may be indicated via the input information. For example, a user (e.g., a surgeon or other surgical team member) may input the template protocol to the planning device. The planning device may determine the protocol based on the template protocol. For example, the protocol may include one or more modifications to the template protocol. In some aspects, the one or more modifications may be based on at least one of the anatomical information or the one or more outcome parameters.

[0058] For example, the planning device may determine that a template surgical action is not appropriate for the patient based on the anatomical information. For example, the planning device may determine a location and / or size of an anatomical element and / or a sensitive structure of the patient. The planning device may determine that the template surgical action will cause an implement or surgical tool to contact and / or come too close to an anatomical element and / or sensitive structure that is not a target of the material removal procedure. For example, the planning device may determine that a distance between the implement or surgical tool and the anatomical element and / or sensitive structure during the template surgical action satisfies (e.g., is less than or equal to) a proximity threshold. The planning device may determine that the template surgical action is not appropriate for the patient based on the distance satisfying the proximity threshold. In such examples, the planning device may determine the protocol by modifying the template surgical action such that the distance between the implement or surgical tool and the anatomical element and / or sensitive structure during the modified surgical action does not satisfy (e.g., is greater than or equal to) the proximity threshold. The modified surgical action may be included in the protocol (e.g., rather than the template surgical action).

[0059] In some aspects, the planning device may determine the protocol based on a profile of a clinician or surgeon associated with the material removal procedure. For example, the profile may indicate historical information for one or more previously performed procedures. The planning device may determine one or more techniques, approaches, and / or surgical plans, among other examples, that are preferred by the clinician or surgeon. The planning device may determine the protocol such that the protocol includes the one or more techniques, approaches, and / or surgical plans where applicable.

[0060] In some aspects, the planning device may determine the one or more surgical actions based on the input information. For example, the input information may include information associated with the one or more surgical actions (e.g., indicating the one or more surgical actions or identifying a type of surgical action). Additionally, or alternatively, the planning device may determine the one or more surgical actions based on the anatomical informationand the one or more outcome parameters. For example, the planning device may automatically calculate a patient- specific plan for the material removal procedure based on the unique anatomy of the patient that is based on the medical imaging data, a type of pathology (e.g., a spine pathology), one or more patient characteristics (e.g., age, body mass index (BMI), or other characteristics), procedure type or approach (e.g., open or minimally invasive), anticipated implant sizing, and / or desired outcomes (e.g., fusion or other desired outcomes of the material removal procedure), among other examples.

[0061] The planning device may determine the protocol based on one or more constraints indicated by the anatomical information, such as disease information that may cause certain techniques and / or approaches to not be suitable for the patient. For example, the one or more constraints may cause a therapy type (e.g., fusion, osteotomy, discectomy, laminectomy, disc replacement, or other therapy types), a delivery type or approach (e.g., open or minimally invasive), a targeted stenosis reduction goal, and / or an anatomy to be treated, among other examples, to be constrained and / or not available for the material removal procedure. The planning device may determine the protocol in accordance with the one or more constraints of the anatomical information specific to the patient.

[0062] In some aspects, the planning device may identify, based on the input information, an anatomical element (or a portion of an anatomical element) to be removed as part of the material removal procedure. The planning device may determine, based on the anatomical information and the one or more outcome parameters, one or more surgical actions that will enable a surgeon and / or the robot to remove the anatomical element (or the portion of the anatomical element). For example, the one or more surgical actions may be determined in context of the anatomy of the patient and the outcome or goals of the surgeon or clinician. For example, the planning device may determine the one or more surgical actions to optimize (e.g., improve (e.g., increase or decrease), maximize, or minimize, depending on the parameter and / or context) the one or more outcome parameters while still accomplishing or completing the material removal procedure (e.g., while still enabling the surgeon and / or robot to remove the anatomical element (or the portion of the anatomical element) by performing the one or more surgical actions).

[0063] In some aspects, the planning device may determine an order of the one or more surgical actions for the protocol. For example, the protocol (e.g., the surgical plan) for the material removal procedure may include multiple steps (e.g., where each step includes one or more surgical actions). The planning device may determine an order of the multiple steps. For example, the order of steps may be based on an amount of effort involved in the one or more surgical actions for a surgeon or other clinician performing the material removal procedure.For example, the planning device may determine, based on the anatomical information and historical information of previously performed material removal procedures, effort scores for respective surgical actions of the one or more surgical actions. The effort scores may indicate a level of effort for performing the respective surgical actions. For example, an effort score may indicate a level of effort for a surgeon to perform a surgical action. An effort score may be based on an estimated amount of time for the surgeon to perform one or more operations or steps for the surgical action. Additionally, or alternatively, the effort score may be based on a quantity of operations or steps for the surgical action to be performed by the surgeon. Additionally, or alternatively, the effort score may be based a proximity (e.g., a distance) to one or more sensitive structures for the operations or steps for the surgical action to be performed by the surgeon (e.g., because operations or steps closer to a sensitive structure may require more care or attention from the surgeon). An effort score may be a measure of an amount of work to be performed by the surgeon. For example, the effort score may indicate an amount of work to be performed by the surgeon to finish one or more surgical actions (e.g., one or more bone cuts). For example, the effort score may indicate an amount of effort and / or an amount of time that the surgeon is estimated to use (e.g., using one or more surgical tools or implements, such as a Kerrison rongeur) to perform or finish one or more surgical actions for a given step.

[0064] The planning device may determine the order of the steps to improve an execution efficiency of the surgical plan. For example, the planning device may determine an order of the one or more planned operations based on the effort scores. For example, if two or more steps could be performed in any order relative to each other, then the planning device may order the two or more planned operations in order of decreasing effort scores (e.g., recommending that the planned operation taking more effort from the surgeon be performed first). For example, the planning device may determine that a step that is estimated to take more effort from the surgeon is to be performed first because the surgeon may have a lower level of fatigue earlier in the material removal procedures. In some aspects, one or more steps may be prerequisite steps to be performed before other steps. In such example, the prerequisite step(s) may be ordered before the other steps (e.g., even if the other steps have higher effort scores). The protocol may include the suggested order of steps for the material removal procedure.

[0065] In some aspects, the planning device may determine the protocol based on historical information associated with the surgical plan. For example, the planning device may obtain the historical information (e.g., from one or more databases or other devices). In some aspects, the historical information may be included in a profile (e.g., the clinician profile) of the surgeon or clinician who will be performing the material removal procedure. The historical informationmay indicate anatomical results of one or more historical procedures. An anatomical result may indicate whether a historical procedure was successful, whether any complications occurred, a functional outcome (e.g., a range of motion, pain score, or other metric), and / or a cosmetic outcome, among other examples. The historical information may be specific to a surgeon or clinician. For example, the planning device may obtain the historical information from a profile of the surgeon who is going to perform the material removal procedure.

[0066] The planning device may determine the protocol based on the historical information, such as by setting values of one or more thresholds or values used to evaluate one or more rules based on the historical information. For example, if the historical information indicates that the surgical plan has a historically low success rate and / or a high rate of complications or issues, then the planning device may set values of one or more thresholds or values used to evaluate one or more rules to a more conservative level (e.g., by setting the value(s) higher or lower depending on the context) to ensure that the determination of the protocol takes into account the relatively higher risk associated with the material removal procedure. As an example, a more conservative value in the context of a distance threshold for evaluating a proximity to a sensitive structure may be a higher value (e.g., to cause the threshold to be satisfied by larger distances), thereby causing the planning device to determine a location of a surgical action further from the sensitive structure.

[0067] As another example, if the historical information indicates that the surgeon has little or no experience performing the material removal procedure (e.g., if the quantity of previously performed procedures by the surgeon for the material removal procedure is less than or equal to an experience threshold), then the planning device may set values of one or more thresholds or values used to evaluate one or more rules to a more conservative level (e.g., by setting the value(s) higher or lower depending on the context) to ensure that the determination of the protocol takes into account the relative inexperience of the surgeon for this material removal procedure. As another example, if the historical information indicates that the surgeon has successful experience performing the material removal procedure (e.g., if the quantity of previously performed procedures that were successful as performed by the surgeon for the material removal procedure is greater than or equal to an experience threshold), then the planning device may set values of one or more thresholds or values used to evaluate one or more rules to a more aggressive level (e.g., by setting the value(s) higher or lower depending on the context) to ensure that the determination of the protocol takes into account the experience of the surgeon for this surgical plan. As an example, a more aggressive value in the context of a distance threshold for evaluating a proximity to a sensitive structure may be a lower value(e.g., to cause the threshold to be satisfied by smaller distances), such that the planning device may determine a location of a surgical action to be closer to a sensitive structure to improve another outcome parameter for the material removal procedure.

[0068] In some aspects, the planning device may determine the protocol based on configuration information of the robot, such as when the material removal procedure is a robotically-assisted procedure. For example, as described elsewhere herein, the configuration information may indicate a safety volume in which the robot (and / or the one or more robotic arms) can safely operate or move. The planning device may modify the safety volume based on the anatomical information of the patient. For example, the area or volume in which the robot (and / or the one or more robotic arms) can safely operate or move may consider the patient-specific anatomy. For example, one or anatomical elements of the patient (e.g., that are not to be contacted, cut, or otherwise interacted with during the robotically-assisted procedure) may extend into the safety volume indicated by the configuration information. The planning device may modify the safety volume to a modified safety volume that does not include the one or anatomical elements of the patient (e.g., that are not to be contacted, cut, or otherwise interacted with during the robotically-assisted procedure). Additionally, or alternatively, the planning device may modify the safety volume (e.g., to the modified safety volume) based on, or using, location information of other objects in the surgical field, such as a surgical table or platform, a ceiling boom, a head drape, one or more members of the surgical team, and / or other objects or devices. The planning device may define a boundary constraint for the determining the protocol using the modified safety volume. For example, the planning device may determine the protocol such that the robot and / or the robotic arms are expected to remain within the modified safety volume during the performance of one or more surgical actions.

[0069] In some aspects, the planning device may determine location information for respective surgical actions. The location information may indicate a location, size, depth, path, trajectory, orientation, and / or other location information relative to one or more anatomical elements of the patient (e.g., that are identified based on the anatomical information). For example, the planning device may determine the location information based on the input information, the anatomical information, and one or more rules or thresholds. For example, the planning device may determine the location information to optimize the one or more outcome parameters while also satisfying one or more thresholds. As an example, a threshold may include a distance threshold. The distance threshold may be a value to incorporate a safety margin for the operation(s) of a surgeon and / or the robot. For example, the planning device may determine, based on the anatomical information, a given surgical action such that a distancebetween a surgical tool configured to perform the given surgical action and an edge or boundary of an anatomical element of the patient (e.g., a sensitive structure or another anatomical element that is not intended to be contacted or interacted with during the material removal procedure) is estimated to satisfy the distance threshold during a performance of the given surgical action for the patient.

[0070] As another example, a rule or threshold may be based on a proximity of a surgical tool configured to perform the given surgical action to one or more sensitive structures of the patient. The anatomical information may indicate sensitive structure information (e.g., indicating one or more sensitive structures of the patient and locations of respective sensitive structure). The planning device may determine the protocol based on the sensitive structure information. For example, a sensitive structure rule may indicate that the surgical tool is not to contact or come within a proximity of a sensitive structure of the patient. For example, the sensitive structure rule may be associated with a distance threshold (e.g., to incorporate a safety margin between the surgical tool and any sensitive structures of the patient). The distance threshold may be the same as the distance threshold described above or may be a different distance threshold. The rules and / or thresholds described herein are provided as examples. Other rules and / or thresholds may be configured and / or used by the planning device in a similar manner to determine protocol.

[0071] In some aspects, the anatomical information may indicate structure condition information of a structure of the one or more anatomical elements (e.g., that is to be cut, removed, and / or otherwise interacted with during the material removal procedure). The structure condition information may include a soft tissue location, a soft tissue condition (e.g., density, mass, consistency, or other conditions), a bone depth, and / or a bone density, among other examples. In some aspects, a rule used to determine the protocol may be that a safety barrier (e.g., a bone barrier) is to be maintained during all surgical actions. The safety barrier may be width or volume of bone or other material that is to remain in an anatomical element after performing a given surgical action. For example, the safety barrier may enable a barrier to remain between an anatomical element being cut or removed and other anatomical elements of the patient, such as one or more sensitive structures. The planning device may determine a depth of a surgical action (e.g., an incision, cut, and / or bore) of the one or more surgical actions based on the structure condition information and a safety barrier parameter indicated by the input information. For example, the safety barrier parameter may indicate a value of the safety barrier. The planning device may determine the one or more surgical actions such that an estimated or expected remaining depth or volume of an anatomical element (e.g., the structure)during or after the performance of the one or more surgical actions is greater than or equal to the value of the safety barrier.

[0072] In some aspects, the planning device may determine multiple protocols in a similar manner as described herein. For example, the planning device may determine multiple protocols where the multiple protocols have different surgical actions and / or different variations or locations of one or more surgical actions. For example, the planning device may determine the multiple protocols to optimize respective outcome parameters (e.g., where a first protocol is configured to optimize a first outcome parameter, a second protocol is configured to optimize a second outcome parameter, and so on). The planning device may determine the multiple protocols such that each protocol is based on the anatomical information, the input information (e.g., the one or more outcome parameters), and the one or more rules or thresholds described herein. For example, the multiple protocols may be different manners and / or variations of the material removal procedure that are each configured to accomplish a goal or clinical outcome of the material removal procedure.

[0073] In some aspects, a protocol may include multiple steps, as described elsewhere herein. The planning device may determine multiple variations of a step for one or more of the multiple steps. For example, variations of a step may include different surgical actions, different locations of a surgical action, and / or different techniques or approaches of a surgical action, among other examples. For example, each variation of the step may satisfy or meet the one or more rules or thresholds used by the planning device to determine the protocol. In other words, each variation of a step may enable a surgical action for the material removal procedure to be completed in accordance with the goal or clinical outcome of the material removal procedure. The variations of a given step and / or the multiple protocols may enable a surgeon or clinician to quickly and easily view different options to perform the material removal procedure, thereby reducing the likelihood of the surgeon or clinician requesting modifications to the protocol (e.g., because the presentation of multiple options to perform the material removal procedure improves the likelihood of presenting an acceptable option to the surgeon or clinician). Reducing the likelihood of the surgeon or clinician requesting modifications to the protocol may conserve resources (e.g., processing resources, computing resources, memory resources, and / or power resources) that would have otherwise been associated with receiving a modification request, modifying the protocol, and / or presenting the modified protocol for display, among other examples.

[0074] As shown by reference number 125, the planning device may generate a 3D model of one or more anatomical elements showing or modeling the protocol. The one or moreanatomical elements may be anatomical elements to be cut, burred, drilled, and / or otherwise interacted with during the protocol. The planning device may generate the 3D model based on, or using, the anatomical information. For example, the planning device may create a surface mesh (e.g., one or more interconnected vertices, edges, and / or faces) that reflect the shape and / or topology of the one or more anatomical elements based on the anatomical information of the patient. The planning device may use one or more 3D modeling techniques to generate a 3D model of the one or more anatomical elements using the surface mesh.

[0075] Additionally, the planning device may generate the 3D model to show, depict, and / or simulate the one or more surgical actions. For example, the planning device may insert or model a surgical action using location information of the surgical action. The planning device may generate a feature in the 3D model showing the surgical action (e.g., showing a planned trajectory, size, shape, and / or orientation of the surgical action). Additionally, or alternatively, the planning device may generate a simulation of a result of performing one or more surgical actions. For example, the planning device may generate a simulated 3D model of the one or more anatomical elements simulating material removal that will occur as a result of performing the protocol. In some aspects, the planning device may generate one or more 3D models and / or one or more features of the 3D model for each protocol and / or for each variation of a step of the protocol.

[0076] In some aspects, the planning device may modify medical imaging data of the patient to be provided for display in connection with the 3D model. For example, the planning device may modify one or more medical images (e.g., 2D images) to show or depict the one or more surgical actions and / or the protocol. As an example, the planning device may insert one or more features or visual elements into the one or more medical images to depict the one or more surgical actions and / or the protocol.

[0077] The planning device may generate display information to enable the display device to view and / or interact with one or more 3D models and / or the one or more modified medical images. For example, the display information may be a package that enables the display device to display the 3D model and / or the one or more modified medical images via a user interface. For example, the planning device may render the 3D model to transform the geometric representation of the 3D model into a visually realistic and informative display, such as by performing shading, lighting, and / or texture mapping, among other examples. The display information may include one or more geometric data parameters (e.g., to identify the vertices, edges, and / or faces of the surface message of the 3D model and associated features showing the one or more surgical actions), one or more rendering parameters (e.g., to control the visualappearance of the 3D model), one or more transformation matrices (e.g., to enable translation, rotation, scaling, and / or perspective projection of the 3D model), among other examples to enable the display device to display the 3D model in an interactive manner via the user interface.

[0078] As shown by reference number 130, the planning device may transmit, and the display device may receive, the display information. For example, the planning device, may provide, for display via the user interface of the display device, the 3D model. As described elsewhere herein, the display device may be a display of the planning device (e.g., may be a component of the planning device) or another device described herein, such as the navigation system. In other aspects, the display device may be a separate device. The display device may include a display screen. As shown by reference number 135, the display device may display, via the user interface, the 3D model (e.g., of the one or more anatomical elements showing the one or more surgical actions of the protocol). The user interface and display of the 3D model is depicted and described in more detail in connection with Figs. 2, 3 A, 3B, and 4. In some aspects, the display device may display multiple versions of the 3D model corresponding to respective protocols of the multiple protocols determined by the planning device. Additionally, or alternatively, the display device may display multiple versions of the 3D model corresponding to respective versions of at least one step of one or more steps included in the protocol.

[0079] In some aspects, the display information may cause the display device to display protocol information (e.g., in addition to the 3D model). The protocol information may include one or more parameters based on the one or more surgical actions included in the protocol. For example, the protocol information may include anatomy stability indication based on the one or more surgical actions. The anatomy stability indication may indicate a level of stability of one or more anatomical elements after and / or during the performance of the one or more surgical actions. As another example, the protocol information may include one or more distances, such as an estimated distance between a surgical tool or implement and one or more anatomical elements (e.g., a bone surface, a sensitive structure, and / or another anatomical element) during the performance of the one or more surgical actions. For example, the one or more distances may include a distance from a facet joint or pars to be preserved, a distance from a ligamentum flavum root (e.g., a nerve root) to be released, a distance between a surgical action to a superior facet (e.g., for a lower vertebra) and an upper end plate of the lower vertebra, and / or a distance from a pedicle canal outer surface and a planned surgical action, among other examples.

[0080] In some aspects, as shown by reference number 140, the display device may display a simulation of the protocol via the 3D model. For example, the 3D model may include features corresponding to respective surgical actions. The display device may display the 3D model at various steps or stages of the protocol by including features in the displayed 3D model for surgical actions to be performed during (or prior to) the various steps or stages. Additionally, or alternatively, the simulation of the protocol may include the display device displaying a simulated 3D model of the one or more anatomical elements simulating material removal that will occur as a result of performing the protocol. For example, the simulated 3D model may be a model of the anatomical elements with material that is expected to be removed as a result of performing the one or more surgical actions of the protocol.

[0081] As shown in Fig. 1C, and by reference number 145, the display device may obtain, via the user interface, one or more user inputs. The one or more user inputs may indicate one or more modifications to at least one of the one or more surgical actions. For example, the user interface may enable a user (e.g., a surgeon, a clinician, or another surgical team member) to provide a user input to modify a size, location, type, trajectory, and / or orientation, among other examples, of a surgical action included in the protocol. In some aspects, the one or more user inputs may include a selection of an area on the 3D model, a free-hand (e.g., a drawn or input selection) input, a surface selection (e.g., a superior and inferior surface selection, or a medial and lateral surface selection), or another type of user input.

[0082] In some aspects, the one or more user inputs may include multiple inputs to different views or orientations of the 3D model (or other medical imaging data provided for display in connection with the 3D model). For example, a first user input may indicate a modification to a surgical action in a first view or orientation of the 3D model. The user may interact with the user interface to rotate, scale, or otherwise modify the 3D model to view a second view or orientation. A second user input may indicate a modification to the surgical action in the second view or orientation. For example, the planning device and / or display device may provide, for display, the 3D model in multiple anatomical views from respective planes. The one or more user inputs may include two or more user inputs to respective anatomical views of the multiple anatomical views.

[0083] As shown by reference number 150, the display device may provide, and the planning device may obtain, an indication of the one or more user inputs. As shown by reference number 155, the planning device may determine a modification for the protocol based on the one or more user inputs. For example, the planning device may determine the modification following information or changes indicated by the one or more user inputs and based on the anatomicalinformation, the one or more outcome parameters, and / or the configuration of the robot, among other examples, in a similar manner as described in more detail elsewhere herein.

[0084] As an example, a user input of the one or more user inputs may be to an area of the 3D model associated with an anatomical element of the one or more anatomical elements. For example, the user input may be a general highlight of an area of the anatomical element. The planning device may identify a structure of the anatomical element based on a distance between the user input and the structure on the three-dimensional model. For example, the structure may be a surface of the anatomical element. The planning device may identify the structure of the anatomical element based on a distance between the user input and the structure on the 3D model in a coordinate system of the user interface. For example, the planning device may determine that the user input is to the structure based on the distance satisfying a threshold. As an example, the planning device may determine that the user input is to the structure based on the distance satisfying a threshold. The planning device may modify the user input to a modified user input based on the structure (e.g., may associate the user input with the surface, rather than the general area highlighted via the user interface). In other words, the planning device may automatically associate the user input with a nearby structure (e.g., bone surface or tissue surface), such in a “snap-to-surface” manner. In some aspects, the planning device may cause the display device to display the modified user input (e.g., to show the user input to the structure, rather than the general area highlighted). In some aspects, a modified surgical action of described herein may be based on the modified user input. For example, the planning device may determine the modified surgical action to be to the structure based on modifying the user input, as described herein.

[0085] In some aspects, the planning device may determine, based on two or more user inputs (e.g., to different views or orientations), a modified surgical action. For example, the planning device may identify, based on the two or more user inputs, a structure of the one or more anatomical elements indicated by the two or more user inputs. For example, the planning device may determine a modified surgical action to be made to the structure based on the two or more user inputs (e.g., in accordance with the two or more user inputs). For example, the planning device may combine or otherwise use information indicated by the two or more user inputs, such as a location, size, depth, orientation, shape, and / or trajectory, among other examples, of a modification to a surgical action to determine the modified surgical action.

[0086] As described elsewhere herein, the planning device may determine the protocol using information indicated by a profile of a surgeon or clinician (e.g., the clinician profile). The planning device may update the profile based on the one or more user inputs. For example, theplanning device may include one or more modifications to the protocol in the profile. This may enable the planning device to store the preference or modifications preferred by a surgeon or clinician (e.g., as indicated by the one or more user inputs) for future protocol determinations for the surgeon or the clinician. This improves the accuracy and / or reliability of the protocols determined by the planning device. Additionally, this may reduce the likelihood of modifications requested by the surgeon or clinician for future protocols, thereby conserving resources (e.g., processing resources, computing resources, memory resources, and / or power resources) that would have otherwise been associated with receiving a modification request, modifying the protocol, and / or presenting the modified protocol for display, among other examples.

[0087] As shown by reference number 160, the planning device may update the 3D model of the one or more anatomical elements to indicate, show, or depict the modified protocol. For example, the planning device may update the 3D model to depict one or more modified surgical actions in accordance with the one or more user inputs. For example, the planning device may update a feature of the 3D model corresponding to a surgical action that was modified or added in accordance with the one or more user inputs. As an example, the planning device may move, resize, and / or relocate a feature and / or may add a feature for a modified surgical action. The planning device may update the 3D model in a similar manner as the generation of the 3D model described in more detail elsewhere herein.

[0088] As shown by reference number 165, the planning device may transmit, and the display device may receive, display information. The display information may enable the display device to display the updated 3D model and / or other updated information, such as one or more updated medical images (e.g., showing one or more modified surgical actions). As shown by reference number 170, the planning device may display the updated 3D model, in a similar manner as described in more detail elsewhere herein. The planning device may update or modify the protocol for the surgical procedure based on user inputs to the updated 3D model, in a similar manner as described in more detail elsewhere herein. For example, a user (e.g., a surgeon, a clinician, or another surgical team member) may incrementally modify the protocol by providing user inputs to the displayed 3D model over one or more instances.

[0089] As shown in Fig. ID, and by reference number 175, the display device may obtain a user input indicating an acceptance of the protocol (e.g., an acceptance indication). For example, a user (e.g., a surgeon, a clinician, or another surgical team member) may view the modeled protocol (e.g., after one or more modifications) via the displayed 3D model and determine that the protocol is acceptable for the material removal procedure. The user mayprovide a user input to the display device indicating that the protocol is approved. As shown by reference number 180, the display device may provide, and the planning device may obtain, the acceptance indication of the protocol. The planning device may perform one or more actions based on obtaining the acceptance indication.

[0090] For example, the planning device may update a profile of the user (e.g., a clinician profile or a surgeon profile) to indicate the accepted protocol for the material removal procedure. For example, the planning device may save the accepted or approved protocol for the material removal procedure. This may enable the planning device to make improved protocol determinations for the user for future procedures that are similar to the material removal procedure.

[0091] As another example, as shown by reference number 185, the planning device may transmit, to the navigation system of the robot, instructions associated with an execution of the one or more planned operations based on the accepted protocol. For example, the planning device may transmit the instructions based on the obtaining the acceptance indication. The instructions may indicate the one or more planned operations for at least one of the surgical actions indicated by the accepted protocol. In some aspects, a surgeon or other surgical team member may control the operation of the robot during the robotically-assisted procedure and the instructions may facilitate the control by provided (e.g., pre-loading) the planned operations for the protocol (e.g., thereby enabling the robot and / or robotic arms to perform one or more automated operations and / or enabling the navigation system to provide enhanced instruction and / or feedback to the surgeon or other surgical team member).

[0092] As shown by reference number 190, the navigation system may transmit, and the robot may receive, navigation instructions that are based on the instructions received from the planning device. For example, the navigation instructions may be configured to cause the one or more robotic arms to perform the one or more planned operations for the protocol. As shown by reference number 195, the robot may perform one or operations for the protocol based on, or in response to, receiving the navigation instructions. For example, the planning device may enable the robot to perform operations during the material removal procedure based on the determined and / or accepted protocol. As described elsewhere herein, movement and / or operations of the robot may be initiated and / or controlled by a surgeon and the one or operations for the protocol may include automated or semi-automated operations that are based on the one or more surgical actions, thereby facilitating and / or improving the accuracy of the user- controlled (e.g., surgeon controlled) operation of the robot. In some aspects, the robot mayautonomously (e.g., without or with limited surgeon control or input) perform at least one (or all) of the one or more surgical actions during the material removal procedure.

[0093] As indicated above, Figs. 1A-1D are provided as an example. Other examples may differ from what is described with regard to Figs. 1A-1D. The number and arrangement of devices shown in Figs. 1A-1D are provided as an example. In practice, there may be additional devices, fewer devices, different devices, or differently arranged devices than those shown in Figs. 1A-1D. Furthermore, two or more devices shown in Figs. 1A-1D may be implemented within a single device, or a single device shown in Figs. 1A-1D may be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one or more devices) shown in Figs. 1A-1D may perform one or more functions described as being performed by another set of devices shown in Figs. 1A-1D.

[0094] Fig. 2 is a diagram of an example implementation 200 associated with a user interface for modeling protocols for material removal procedures. As shown in Fig. 2, the display device (e.g., the display device depicted and described elsewhere herein) may display a user interface 205.

[0095] As shown in Fig. 2, the user interface 205 may include a 3D model 210 of one or more anatomical elements of a patient. For example, the planning device may generate and / or provide for display the 3D model based on the anatomical information of the patient, as described in more detail elsewhere herein. As an example, the anatomical element may be one or more spinal vertebrae, such as an L2 lumbar vertebra as shown in Fig. 2.

[0096] As shown in Fig. 2, the 3D model 210 may be presented for display in an procedural view position in which a user (e.g., a surgeon or other surgical team member) would view the actual anatomical element(s) of the patient during the procedure. For example, a default or initial view of the 3D model 210 may be the anatomical position. As an example, for one or more anatomical elements of a spine, the procedural view position may be a horizontal position. For other anatomical elements, the procedural view position may be a different view or orientation of the 3D model 210. Presenting the 3D model 210 for display in the procedural view position may enable a user (e.g., a surgeon or other surgical team member) to view the 3D model 210 from the same perspective that the user would view the actual anatomical element(s) of the patient during the procedure. This enables the user to quickly and more accurately view, analyze, and / or evaluate the one or more planned surgical actions, as described in more detail elsewhere herein.

[0097] The user interface 205 may include one or more anatomical views 215 of the one or more anatomical elements. The one or more anatomical views 215 may be, or may be basedon, medical imaging data of the patient. For example, the one or more anatomical views 215 may include an axial view (or transverse view), a sagittal view, a coronal view, and / or an oblique view, among other examples.

[0098] The user interface 205 may be a user interface via which a user provides the input information described elsewhere herein. For example, the 3D model 210 may be a model of the one or more anatomical elements without any shown or depicted surgical actions for the material removal procedure. The user interface may include one or more elements to enable a user to provide user inputs (e.g., a click and drag input, as shown in Fig. 2, or another type of input) to provide the input information to be used by the planning device to determine the protocol for the material removal procedure. For example, a user input may include a selection of a structure or anatomical element to be removed or cut as part of the material removal procedure. In some aspects, a user input may be a rough or estimated location of one or more surgical actions (e.g., which may be refined by the planning device, as described in more detail elsewhere herein).

[0099] As indicated above, Fig. 2 is provided as an example. Other examples may differ from what is described with regard to Fig. 2.

[0100] Figs. 3A and 3B are diagrams of an example implementation 300 associated with a user interface for modeling protocols for material removal procedures. As shown in Figs. 3A and 3B, the display device (e.g., the display device depicted and described elsewhere herein) may display a user interface 305. As shown in Figs. 3A and 3B, the user interface 305 may include a 3D model 310 of one or more anatomical elements of a patient (e.g., the one or more anatomical elements depicted by the 3D model 210). The user interface 305 may include one or more anatomical views 315 of the one or more anatomical elements, in a similar manner as described in connection with Fig. 2. The surgical action(s) shown in Figs. 3A and 3B may be for a laminectomy procedure. However, the planning device and / or display device may generate and / or model protocols for other procedures in a similar manner as depicted and described herein.

[0101] As shown in Figs. 3A and 3B, the 3D model 310 may show or depict one or more surgical actions for a protocol associated with the material removal procedure. As an example, the 3D model 310 may include one or more features depicted a surgical action 320a and a surgical action 320b. For example, as shown in Fig. 3A, the 3D model 310 may include a feature depicting a size, location, shape, trajectory, and / or orientation, among other examples of the surgical action 320a on the 3D model of the one or more anatomical elements. Similarly, each anatomical view 315 may include a feature or element showing or depicting the surgicalaction 320a in context of that anatomical view 315. A feature representing a surgical action (e.g., the surgical action 320a and / or the surgical action 320b) may include a tool volume (e.g., representing a surgical tool or implement used to perform the surgical action), one or more lines (e.g., to represent one or more cuts to the anatomical element(s)), one or more curves or splines (e.g., to represent one or more cuts to the anatomical element(s)), a tool path (e.g., representing a path of surgical tool or implement used to perform the surgical action), one or more planes (e.g., to represent a midline of a surgical action, such as a trough), and / or one or more surfaces (e.g., to represent a midline of a surgical action and / or a decortication process performed to an anatomical element), among other examples.

[0102] As shown in Fig. 3B, the 3D model 310 may be rotated to view a different orientation of the 3D model 310. The rotated view of the 3D model 310 may enable the display device to depict or show a surgical action 320b. For example, the surgical action 320a may be a first cut for the material removal procedure and the surgical action 320b may be a second cut for the material removal procedure. The user interface may include one or more navigation controls to enable a user to rotate, move, scale, and / or otherwise modify a view of the 3D model 310 showing the one or more surgical actions (e.g., the surgical action 320a and the surgical action 320b). This may enable a user (e.g., a surgeon or clinician) to view the planned surgical actions in context of patient-specific anatomy and in a 3D view. In some aspects, as the view of the 3D model 310 is updated, the view depicted by the anatomical views 315 may be similarly updated (e.g., to show information depicted by a current view of the 3D model 310 in the one or more anatomical views 315) by the planning device using the medical imaging data and / or the anatomical information.

[0103] As indicated above, Figs. 3A and 3B is provided as an example. Other examples may differ from what is described with regard to Figs. 3 A and 3B.

[0104] Fig. 4 is a diagram of an example implementation 400 associated with a user interface for modeling protocols for material removal procedures. As shown in Fig. 4, the display device (e.g., the display device depicted and described elsewhere herein) may display a user interface 405. As shown in Fig. 4, the display device (e.g., the display device depicted and described elsewhere herein) may display a user interface 405. The user interface 405 may include a 3D model 410 of one or more anatomical elements of a patient (e.g., the one or more anatomical elements depicted by the 3D model 210 and / or the 3D model 310). The user interface 405 may include one or more anatomical views 415 of the one or more anatomical elements, in a similar manner as described in connection with Figs. 2, 3 A, and / or 3B. The surgical action(s) shown in Fig. 4 may be for a laminectomy procedure. However, the planning device and / or displaydevice may generate and / or model protocols for other procedures in a similar manner as depicted and described herein.

[0105] As shown in Fig. 4, the 3D model 410 may include a simulated result 420 of the material removal procedure. The simulated result 420 may be a result of the one or more surgical actions for the procedure, such as the surgical action 320a and / or the surgical action 320b. For example, the simulated result 420 may simulate a material removal from the one or more anatomical elements that is enabled via the one or more surgical actions for the procedure, such as the surgical action 320a and / or the surgical action 320b.

[0106] As indicated above, Fig. 4 is provided as an example. Other examples may differ from what is described with regard to Fig. 4.

[0107] Fig. 5 is a diagram of an example environment 500 in which systems and / or methods described herein may be implemented. As shown in Fig. 5, environment 500 may include a planning device 510, an image database 520, a robot 530 (e.g., that includes one or more robotic arms 535), a navigation system 540, a display device 550, a client device 560, and a network 570. Devices of environment 500 may interconnect via wired connections, wireless connections, or a combination of wired and wireless connections. In some examples, the environment 500 may be a surgical environment, a surgical field, a surgical space, a connected surgical operating room, or a connected surgical ecosystem, among other examples.

[0108] The planning device 510 may include one or more devices capable of receiving, generating, storing, processing, providing, and / or routing information associated with generating and modeling protocols for material removal procedure, as described elsewhere herein. The planning device 510 may include a communication device and / or a computing device. For example, the planning device 510 may include a server, such as an application server, a client server, a web server, a database server, a host server, a proxy server, a virtual server (e.g., executing on computing hardware), or a server in a cloud computing system. In some implementations, the planning device 510 may include computing hardware used in a cloud computing environment. In some examples, the planning device 510 may include a surgical navigation system. In some examples, the planning device 510 may be included in another device in the environment 500, such as the navigation system 540.

[0109] The image database 520 may include one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with generating and modeling protocols for material removal procedure, as described elsewhere herein. The image database 520 may include a communication device and / or a computing device. For example, the image database 520 may include a data structure, a database, a data source, a server, adatabase server, an application server, a client server, a web server, a host server, a proxy server, a virtual server (e.g., executing on computing hardware), a server in a cloud computing system, a device that includes computing hardware used in a cloud computing environment, or a similar type of device. As an example, the image database 520 may store medical imaging data, as described elsewhere herein.

[0110] In some implementations, the medical imaging data may be generated by, captured by, and / or provided by (e.g., to the image database 520) one or more imaging devices. The imaging device may be operable to image anatomical element(s) (e.g., a bone, veins, tissue, and / or other anatomical elements) and / or other aspects of patient anatomy to output medical imaging data (e.g., image data depicting or corresponding to a bone, veins, and / or tissue). “Image data” or “imaging data” as used herein refers to the data generated or captured by an imaging device, including in a machine-readable form, a graphical / visual form, and in any other form. In some examples, medical imaging data may include data corresponding to an anatomical feature of a patient, or to a portion thereof. The image data may be or include a preoperative image, an intraoperative image, a postoperative image, or an image taken independently of any surgical procedure. In some examples, a first imaging device may be used to obtain first image data (e.g., a first image) at a first time, and a second imaging device may be used to obtain second image data (e.g., a second image) at a second time after the first time. The imaging device may be capable of taking a 5D image or a 6D image to yield the image data. The imaging device may be or comprise, for example, an ultrasound scanner (which may comprise, for example, a physically separate transducer and receiver, or a single ultrasound transceiver), an 0-arm, a C-arm, a G-arm, or any other device utilizing X-ray -based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a MRI scanner, an OCT scanner, an endoscope, a microscope, an optical camera, a thermographic camera (e.g., an infrared camera), a radar system (which may include a transmitter, a receiver, a processor, and one or more antennas), or any other imaging device suitable for obtaining images of an anatomical feature of a patient. In some examples, medical imaging data, as described herein, may be considered to be continuous and / or provided as an image data stream (e.g., to the image database 520 via one or more imaging devices), such as if the medical image data represents two or more frames per second.

[0111] The robot 530 may be any surgical robot or surgical robotic system. The robot 530 may include a robotic guidance system. The robot may be configured to position, for example, a retraction assembly and / or an end unit at one or more precise position(s) and orientation(s), and / or to return a retraction assembly and / or an end unit to the same position(s) andorientation(s) at a later point in time. The end unit may include a surgical instrument, a surgical tool, and / or an implement, among other examples. The robot 530 may additionally or alternatively be configured to manipulate a surgical tool or implement (whether based on guidance from the navigation system 540 or not) to accomplish or to assist with a surgical task. In some embodiments, the robot 530 may be configured to hold and / or manipulate an anatomical element during or in connection with a surgical procedure. The robot 530 may comprise one or more robotic arms 535. In some embodiments, the robotic arm 535 may comprise a first robotic arm and a second robotic arm, though the robot 530 may comprise more than two robotic arms. In some embodiments, one or more of the robotic arms 535 may be used to hold and / or maneuver a retraction assembly and / or an end unit. In embodiments where two retraction assemblies and end units are used, one robotic arm 535 may hold one set, and another robotic arm 535 may hold another set. Each robotic arm 535 may be positioned or moved independently of the other robotic arm(s). The robotic arms 535 may be controlled in a single, shared coordinate space, or in separate coordinate spaces.

[0112] The robot 530, together with the robotic arm(s) 535, may have one, two, three, four, five, six, seven, or more degrees of freedom. Further, the robotic arm 535 may be positioned in any pose, plane, and / or focal point. The pose includes a position and an orientation. As a result, the retraction assembly, the end unit, a surgical tool, or other object held by the robot 530 (or by the robotic arm 535) may be precisely positioned in one or more needed and specific positions and orientations. The robotic arm(s) 535 may comprise one or more sensors that enable one or more processors (or one or more processors of the robot 530) to determine a precise pose in space of the robotic arm (as well as any object or element held by or secured to the robotic arm). The one or more sensors may also measure a force at an end of the robotic arm 535. In some implementations, the sensors may measure a force at an end unit disposed at the end of the robotic arm 535. Sensors may be used in any other component in a similar manner. For example, a manually operated surgical tool may comprise a retraction assembly and a sensor for measuring a force at the surgical tool. The retraction assembly may be triggered when the sensor measures a force at the surgical tool that satisfies a force threshold.

[0113] In some examples, reference markers (e.g., navigation markers) may be placed on the robot 530 (including, e.g., on the robotic arm 535), an imaging device, a retraction assembly, an end unit or any other object, device, or component in a surgical space. The reference markers may be tracked by the navigation system 540, and the results of the tracking may be used by the robot 530 and / or by an operator. In some examples, the navigation system 540 may be used to track other components in the environment 500 (e.g., the retraction assembly, the end unit,one or more people, and / or one or more objects) and the one or more components can operate without the use of the robot 530 (e.g., with the surgeon manually manipulating a retraction assembly and an end unit and / or one or more surgical tools, based on information and / or instructions generated by the navigation system 540).

[0114] The navigation system 540 may include one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with generating and modeling protocols for material removal procedure, as described elsewhere herein. The navigation system 540 may include a communication device and / or a computing device. The navigation system 540 may provide navigation for a surgeon and / or a surgical robot during a surgical procedure. The navigation system 540 may include one or more cameras or other sensor(s) for tracking one or more reference markers, navigated trackers, or other objects within the operating room or other room in which some or all of the environment 500 is located. The one or more cameras may be optical cameras, endoscopic cameras, infrared cameras, or other cameras. In some embodiments, the navigation system 540 may include one or more electromagnetic sensors. In some examples, the navigation system 540 may be used to track a position and orientation (e.g., a pose) of an imaging device, a retraction assembly, an end unit, the robot 530 and / or robotic arm 535, and / or one or more surgical tools (e.g., to track a pose of a navigated tracker attached, directly or indirectly, in fixed relation to the one or more of the examples described above). The navigation system 540 may include a display for displaying one or more images from an external source or for displaying an image and / or video stream from the one or more cameras or other sensors of the navigation system 540. In some examples, the display may be, or may include, the display device 550. The navigation system 540 may be configured to provide guidance to a surgeon or other user in the environment 500, to the robot 530, or to any other device or component of the environment 500. The guidance provided by the navigation system 540 may include a pose of one or more anatomical elements, whether or not a tool is in the proper trajectory, and / or how to move a tool into the proper trajectory to carry out a surgical task according to a preoperative or other surgical plan, among other examples.

[0115] The display device 550 may include one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with generating and modeling protocols for material removal procedure, as described elsewhere herein. The display device 550 may include a communication device and / or a computing device. The display device 550 may be any device capable of, or configured to, provide, display, or otherwise output (e.g., via a visual output and / or an audio output) information. For example, the display device550 may include a wireless communication device, a mobile phone, a user equipment, a laptop computer, a tablet computer, a desktop computer, a wearable communication device (e.g., a smart wristwatch, a pair of smart eyeglasses, a head mounted display, or a virtual reality headset), a keyboard, mouse, trackball, monitor, television, screen, touch screen, a speaker, and / or any other device for receiving information from a user and / or for providing information to a user. In some example, the display device 550 may be included in one or more devices or components of the environment 500. For example, the display device 550 may be a component of the planning device 510, and / or the navigation system 540, among other examples. In some examples, multiple devices or components of the environment 500 may include display devices 550. For example, the planning device 510 may include a first display device 550 and the navigation system 540 may include a second display device 550.

[0116] The client device 560 may include one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with generating and modeling protocols for material removal procedure, as described elsewhere herein. The client device 560 may include a communication device and / or a computing device. For example, the client device 560 may include a wireless communication device, a mobile phone, a user equipment, a laptop computer, a tablet computer, a desktop computer, a wearable communication device (e.g., a smart wristwatch, a pair of smart eyeglasses, a head mounted display, or a virtual reality headset), or a similar type of device. In some examples, the client device 560 may be a component of another device or system described herein, such as the planning device 510 or the navigation system 540.

[0117] The network 570 may include one or more wired and / or wireless networks. For example, the network 570 may include a wireless wide area network (e.g., a cellular network or a public land mobile network), a local area network (e.g., a wired local area network or a wireless local area network (WLAN), such as a Wi-Fi network), a personal area network (e.g., a Bluetooth network), a near-field communication network, a telephone network, a private network, the Internet, and / or a combination of these or other types of networks. The network 570 enables communication among the devices of environment 500.

[0118] The number and arrangement of devices and networks shown in Fig. 5 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown in Fig. 5. Furthermore, two or more devices shown in Fig. 5 may be implemented within a single device, or a single device shown in Fig. 5 may be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one ormore devices) of environment 500 may perform one or more functions described as being performed by another set of devices of environment 500.

[0119] Fig. 6 is a diagram of example components of a device 600 associated with generating and modeling protocols for material removal procedure. The device 600 may correspond to the planning device 510, the image database 520, the robot 530, the robotic arm 535, the navigation system 540, and / or the display device 550. In some implementations, the planning device 510, the image database 520, the robot 530, the robotic arm 535, the navigation system 540, the display device 550, and / or the client device 560 may include one or more devices 600 and / or one or more components of the device 600. As shown in Fig. 6, the device 600 may include a bus 610, a processor 620, a memory 630, an input component 640, an output component 650, and / or a communication component 660.

[0120] The bus 610 may include one or more components that enable wired and / or wireless communication among the components of the device 600. The bus 610 may couple together two or more components of Fig. 6, such as via operative coupling, communicative coupling, electronic coupling, and / or electric coupling. For example, the bus 610 may include an electrical connection (e.g., a wire, a trace, and / or a lead) and / or a wireless bus. The processor 620 may include a central processing unit, a graphics processing unit, a microprocessor, a controller, a microcontroller, a digital signal processor, a field-programmable gate array, an application-specific integrated circuit, and / or another type of processing component. The processor 620 may be implemented in hardware, firmware, or a combination of hardware and software. In some implementations, the processor 620 may include one or more processors capable of being programmed to perform one or more operations or processes described elsewhere herein.

[0121] The memory 630 may include volatile and / or nonvolatile memory. For example, the memory 630 may include random access memory (RAM), read only memory (ROM), a hard disk drive, and / or another type of memory (e.g., a flash memory, a magnetic memory, and / or an optical memory). The memory 630 may include internal memory (e.g., RAM, ROM, or a hard disk drive) and / or removable memory (e.g., removable via a universal serial bus connection). The memory 630 may be a non-transitory computer-readable medium. The memory 630 may store information, one or more instructions, and / or software (e.g., one or more software applications) related to the operation of the device 600. In some implementations, the memory 630 may include one or more memories that are coupled (e.g., communicatively coupled) to one or more processors (e.g., processor 620), such as via the bus 610. Communicative coupling between a processor 620 and a memory 630 may enable the processor620 to read and / or process information stored in the memory 630 and / or to store information in the memory 630.

[0122] The input component 640 may enable the device 600 to receive input, such as user input and / or sensed input. For example, the input component 640 may include a touch screen, a keyboard, a keypad, a mouse, a button, a microphone, a switch, a sensor, a global positioning system sensor, a global navigation satellite system sensor, an accelerometer, a gyroscope, and / or an actuator. The output component 650 may enable the device 600 to provide output, such as via a display, a speaker, and / or a light-emitting diode. The communication component 660 may enable the device 600 to communicate with other devices via a wired connection and / or a wireless connection. For example, the communication component 660 may include a receiver, a transmitter, a transceiver, a modem, a network interface card, and / or an antenna.

[0123] The device 600 may perform one or more operations or processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 630) may store a set of instructions (e.g., one or more instructions or code) for execution by the processor 620. The processor 620 may execute the set of instructions to perform one or more operations or processes described herein. In some implementations, execution of the set of instructions, by one or more processors 620, causes the one or more processors 620 and / or the device 600 to perform one or more operations or processes described herein. In some implementations, hardwired circuitry may be used instead of or in combination with the instructions to perform one or more operations or processes described herein. Additionally, or alternatively, the processor 620 may be configured to perform one or more operations or processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0124] The number and arrangement of components shown in Fig. 6 are provided as an example. The device 600 may include additional components, fewer components, different components, or differently arranged components than those shown in Fig. 6. Additionally, or alternatively, a set of components (e.g., one or more components) of the device 600 may perform one or more functions described as being performed by another set of components of the device 600.

[0125] Fig. 7 is a flowchart of an example process 700 associated with generating and modeling protocols for material removal procedures. In some aspects, one or more process blocks of Fig. 7 are performed by a planning device (e.g., planning device 510). In some aspects, one or more process blocks of Fig. 7 are performed by another device or a group of devices separate from or including the planning device, such as the image database 520, therobot 530, a robotic arm 535, the navigation system 540, the display device 550, and / or the client device 560. Additionally, or alternatively, one or more process blocks of Fig. 7 may be performed by one or more components of device 600, such as processor 620, memory 630, input component 640, output component 650, and / or communication component 660.

[0126] As shown in Fig. 7, process 700 may include obtaining input information that indicates one or more outcome parameters for a robotically-assisted procedure that is associated with a robot, and wherein the robotically-assisted procedure is associated with one or more anatomical elements (block 710). For example, the planning device may obtain input information that indicates one or more outcome parameters for a robotically-assisted procedure that is associated with a robot, and wherein the robotically-assisted procedure is associated with one or more anatomical elements, as described above. In some aspects, the robotically-assisted procedure is associated with one or more anatomical elements.

[0127] As further shown in Fig. 7, process 700 may include determining, based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements, and wherein at least one of the one or more surgical actions is configured to be performed via the robot (block 720). For example, the planning device may determine, based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements, and wherein at least one of the one or more surgical actions is configured to be performed via the robot, as described above. In some aspects, the protocol includes one or more surgical actions to be performed to the one or more anatomical elements. In some aspects, at least one of the one or more surgical actions is configured to be performed via the robot.

[0128] As further shown in Fig. 7, process 700 may include generating, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements (block 730). For example, the planning device may generate, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements, as described above. In some aspects, the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements.

[0129] As further shown in Fig. 7, process 700 may include providing, for display via a user interface, the three-dimensional model (block 740). For example, the planning device may provide, for display via a user interface, the three-dimensional model, as described above.

[0130] As further shown in Fig. 7, process 700 may include receiving, via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions (block 750). For example, the planning device may receive, via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions, as described above.

[0131] As further shown in Fig. 7, process 700 may include modifying the three-dimensional model to a modified three-dimensional model that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs (block 760). For example, the planning device may modify the three-dimensional model to a modified three-dimensional model that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs, as described above. In some aspects, the one or more modified surgical actions are based on the one or more user inputs.

[0132] As further shown in Fig. 7, process 700 may include providing, for display via the user interface, the modified three-dimensional model (block 770). For example, the planning device may provide, for display via the user interface, the modified three-dimensional model, as described above.

[0133] Process 700 may include additional aspects, such as any single aspect or any combination of aspects described below and / or in connection with one or more other processes described elsewhere herein.

[0134] In a first aspect, process 700 includes generating one or more instructions for the robot based on the one or more modified surgical actions, where the one or more instructions are configured to cause the robot to perform at least one of the one or more modified surgical actions, and transmitting, to a navigation system of the robot, the one or more instructions.

[0135] In a second aspect, alone or in combination with the first aspect, process 700 includes determining a template protocol based on the type of the robotically-assisted procedure, and determining the protocol based on the template protocol, the protocol includes one or more modifications to the template protocol, and the one or more modifications are based on at least one of the anatomical information or the one or more outcome parameters.

[0136] In a third aspect, alone or in combination with one or more of the first and second aspects, process 700 includes providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective protocols of the multiple protocols.

[0137] In a fourth aspect, alone or in combination with one or more of the first through third aspects, process 700 includes providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective versions of at least one step of the one or more steps.

[0138] In a fifth aspect, alone or in combination with one or more of the first through fourth aspects, process 700 includes identifying a structure of the anatomical element based on a distance between the user input and the structure on the three-dimensional model, and modifying the user input to a modified user input based on the structure, where a modified surgical action of the one or more modified surgical actions is based on the modified user input.

[0139] In a sixth aspect, alone or in combination with one or more of the first through fifth aspects, process 700 includes determining that the user input is to the structure based on the distance satisfying a threshold.

[0140] In a seventh aspect, alone or in combination with one or more of the first through sixth aspects, process 700 includes providing, for display, the three-dimensional model in multiple anatomical views from respective planes, the one or more user inputs include two or more user inputs to respective anatomical views of the multiple anatomical views, and modifying the three-dimensional model includes determining, based on the two or more user inputs, a modified surgical action of the one or more modified surgical actions.

[0141] In an eighth aspect, alone or in combination with one or more of the first through seventh aspects, process 700 includes identifying, based on the two or more user inputs, a structure of the one or more anatomical elements indicated by the two or more user inputs, and determining the modified surgical action to be made to the structure based on the two or more user inputs.

[0142] In a ninth aspect, alone or in combination with one or more of the first through eighth aspects, process 700 includes determining a depth of a surgical action of the one or more surgical actions based on the structure condition information and a safety barrier parameter indicated by the input information, where the surgical action is associated with the structure.

[0143] In a tenth aspect, alone or in combination with one or more of the first through ninth aspects, the anatomical information includes sensitive structure information of one or more sensitive structures included in the one or more anatomical elements, and the protocol is based on the sensitive structure information.

[0144] In an eleventh aspect, alone or in combination with one or more of the first through tenth aspects, process 700 includes determining the protocol based on a profile of a clinician associated with the robotically-assisted procedure, where the profile indicates historical information for one or more previously performed procedures.

[0145] In a twelfth aspect, alone or in combination with one or more of the first through eleventh aspects, process 700 includes updating the profile based on the one or more user inputs.

[0146] In a thirteenth aspect, alone or in combination with one or more of the first through twelfth aspects, process 700 includes determining, using medical imaging data of the patient, the anatomical information.

[0147] Although Fig. 7 shows example blocks of process 700, in some aspects, process 700 includes additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in Fig. 7. Additionally, or alternatively, two or more of the blocks of process 700 may be performed in parallel.

[0148] Fig. 8 is a flowchart of an example process 800 associated with generating and modeling protocols for material removal procedures. In some aspects, one or more process blocks of Fig. 8 are performed by a planning device (e.g., planning device 510). In some aspects, one or more process blocks of Fig. 8 are performed by another device or a group of devices separate from or including the planning device, such as the image database 520, the robot 530, a robotic arm 535, the navigation system 540, the display device 550, and / or the client device 560. Additionally, or alternatively, one or more process blocks of Fig. 8 may be performed by one or more components of device 600, such as processor 620, memory 630, input component 640, output component 650, and / or communication component 660.

[0149] As shown in Fig. 8, process 800 may include obtaining input information that indicates one or more outcome parameters for a procedure, and wherein the procedure is associated with one or more anatomical elements (block 810). For example, the planning device may obtain input information that indicates one or more outcome parameters for a procedure, and wherein the procedure is associated with one or more anatomical elements, as described above. In some aspects, the procedure is associated with one or more anatomical elements.

[0150] As further shown in Fig. 8, process 800 may include determining, based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements (block 820). For example, the planning device may determine, based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements,wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements, as described above. In some aspects, the protocol includes one or more surgical actions to be performed to the one or more anatomical elements.

[0151] As further shown in Fig. 8, process 800 may include generating, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements (block 830). For example, the planning device may generate, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements, as described above. In some aspects, the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements.

[0152] As further shown in Fig. 8, process 800 may include providing the three-dimensional model (block 840). For example, the planning device may provide the three-dimensional model, as described above.

[0153] Process 800 may include additional aspects, such as any single aspect or any combination of aspects described below and / or in connection with one or more other processes described elsewhere herein.

[0154] In a first aspect, process 800 includes receiving, via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions, modifying the three-dimensional model to a modified three-dimensional model that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, the one or more modified surgical actions are based on the one or more user inputs, and providing, for display via the user interface, the modified three-dimensional model.

[0155] In a second aspect, alone or in combination with the first aspect, process 800 includes generating one or more instructions for a robot based on the one or more modified surgical actions, where the one or more instructions are configured to cause the robot to perform at least one of the one or more modified surgical actions, and transmitting, to a navigation system of the robot, the one or more instructions.

[0156] In a third aspect, alone or in combination with one or more of the first and second aspects, the input information indicates a type of the procedure, and determining the protocol includes determining a template protocol based on the type of the procedure, and determining the protocol based on the template protocol, the protocol includes one or more modifications tothe template protocol, and the one or more modifications are based on at least one of the anatomical information or the one or more outcome parameters.

[0157] In a fourth aspect, alone or in combination with one or more of the first through third aspects, a user input of the one or more user inputs is to an area of the three-dimensional model associated with an anatomical element of the one or more anatomical elements, and the process 800 includes identifying a structure of the anatomical element based on a distance between the user input and the structure on the three-dimensional model, and modifying the user input to a modified user input based on the structure, where a modified surgical action of the one or more modified surgical actions is based on the modified user input.

[0158] In a fifth aspect, alone or in combination with one or more of the first through fourth aspects, identifying the structure includes determining that the user input is to the structure based on the distance satisfying a threshold.

[0159] In a sixth aspect, alone or in combination with one or more of the first through fifth aspects, the protocol includes one or more steps, and providing, for display via the user interface, the three-dimensional model includes providing, for display, the three-dimensional model in multiple anatomical views from respective planes, the one or more user inputs include two or more user inputs to respective anatomical views of the multiple anatomical views, and modifying the three-dimensional model includes determining, based on the two or more user inputs, a modified surgical action of the one or more modified surgical actions.

[0160] In a seventh aspect, alone or in combination with one or more of the first through sixth aspects, determining the modified surgical action includes identifying, based on the two or more user inputs, a structure of the one or more anatomical elements indicated by the two or more user inputs, and determining the modified surgical action to be made to the structure based on the two or more user inputs.

[0161] In an eighth aspect, alone or in combination with one or more of the first through seventh aspects, the protocol includes multiple protocols, and providing, for display via the user interface, the three-dimensional model includes providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective protocols of the multiple protocols.

[0162] In a ninth aspect, alone or in combination with one or more of the first through eighth aspects, the protocol includes one or more steps, and providing, for display via the user interface, the three-dimensional model includes providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective versions of at least one step of the one or more steps.

[0163] In a tenth aspect, alone or in combination with one or more of the first through ninth aspects, the anatomical information indicates structure condition information of a structure of the one or more anatomical elements, and determining the protocol includes determining a depth of a surgical action of the one or more surgical actions based on the structure condition information and a safety barrier parameter indicated by the input information, where the surgical action is associated with the structure.

[0164] In an eleventh aspect, alone or in combination with one or more of the first through tenth aspects, the anatomical information includes sensitive structure information of one or more sensitive structures included in the one or more anatomical elements, and the protocol is based on the sensitive structure information.

[0165] In a twelfth aspect, alone or in combination with one or more of the first through eleventh aspects, determining the protocol includes determining the protocol based on a profile of a clinician associated with the procedure, where the profile indicates historical information for one or more previously performed procedures.

[0166] In a thirteenth aspect, alone or in combination with one or more of the first through twelfth aspects, process 800 includes updating the profile based on one or more user inputs to the user interface.

[0167] In a fourteenth aspect, alone or in combination with one or more of the first through thirteenth aspects, process 800 includes determining, using medical imaging data of the patient, the anatomical information.

[0168] Although Fig. 8 shows example blocks of process 800, in some aspects, process 800 includes additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in Fig. 8. Additionally, or alternatively, two or more of the blocks of process 800 may be performed in parallel.

[0169] The following provides an overview of some Examples of the present disclosure:

[0170] Example 1: A method, comprising: obtaining, by a device, input information that indicates one or more outcome parameters for a robotically-assisted procedure that is associated with a robot, and wherein the robotically-assisted procedure is associated with one or more anatomical elements; determining, by the device and based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements, and wherein at least one of the one or more surgical actions is configured to be performed via the robot; generating, by the device and based on the protocol and the anatomical information, a three-dimensional model of the one or moreanatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements; providing, by the device and for display via a user interface, the three-dimensional model; receiving, by the device and via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions; modifying, by the device, the three-dimensional model to a modified three-dimensional model that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs; and providing, by the device and for display via the user interface, the modified three- dimensional model.

[0171] Example 2 : The method of Example 1, further comprising: generating one or more instructions for the robot based on the one or more modified surgical actions, wherein the one or more instructions are configured to cause the robot to perform at least one of the one or more modified surgical actions; and transmitting, to a navigation system of the robot, the one or more instructions.

[0172] Example 3: The method of any one of Examples 1-2, wherein the input information indicates a type of the robotically-assisted procedure, and wherein determining the protocol comprises: determining a template protocol based on the type of the robotically-assisted procedure; and determining the protocol based on the template protocol, wherein the protocol includes one or more modifications to the template protocol, and wherein the one or more modifications are based on at least one of the anatomical information or the one or more outcome parameters.

[0173] Example 4: The method of any one of Examples 1-3, wherein the protocol includes multiple protocols, and wherein providing, for display via the user interface, the three- dimensional model comprises: providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective protocols of the multiple protocols.

[0174] Example 5: The method of any one of Examples 1-4, wherein the protocol includes one or more steps, and wherein providing, for display via the user interface, the three- dimensional model comprises: providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective versions of at least one step of the one or more steps.

[0175] Example 6: The method of any one of Examples 1-5, wherein a user input of the one or more user inputs is to an area of the three-dimensional model associated with an anatomical element of the one or more anatomical elements, and the method further comprising: identifyinga structure of the anatomical element based on a distance between the user input and the structure on the three-dimensional model; and modifying the user input to a modified user input based on the structure, wherein a modified surgical action of the one or more modified surgical actions is based on the modified user input.

[0176] Example 7: The method of Example 6, wherein identifying the structure comprises: determining that the user input is to the structure based on the distance satisfying a threshold.

[0177] Example 8: The method of any one of Examples 1-7, wherein the protocol includes one or more steps, and wherein providing, for display via the user interface, the three- dimensional model comprises: providing, for display, the three-dimensional model in multiple anatomical views from respective planes, wherein the one or more user inputs include two or more user inputs to respective anatomical views of the multiple anatomical views; and wherein modifying the three-dimensional model comprises: determining, based on the two or more user inputs, a modified surgical action of the one or more modified surgical actions.

[0178] Example 9: The method of Example 8, wherein determining the modified surgical action comprises: identifying, based on the two or more user inputs, a structure of the one or more anatomical elements indicated by the two or more user inputs; and determining the modified surgical action to be made to the structure based on the two or more user inputs.

[0179] Example 10: The method of any one of Examples 1-9, wherein the anatomical information indicates structure condition information of a structure of the one or more anatomical elements, and wherein determining the protocol comprises: determining a depth of a surgical action of the one or more surgical actions based on the structure condition information and a safety barrier parameter indicated by the input information, wherein the surgical action is associated with the structure.

[0180] Example 11: The method of any one of Examples 1-10, wherein the anatomical information includes sensitive structure information of one or more sensitive structures included in the one or more anatomical elements, and wherein the protocol is based on the sensitive structure information.

[0181] Example 12: The method of any one of Examples 1-11, wherein determining the protocol comprises: determining the protocol based on a profile of a clinician associated with the robotically-assisted procedure, wherein the profile indicates historical information for one or more previously performed procedures.

[0182] Example 13: The method of Example 12, further comprising: updating the profile based on the one or more user inputs.

[0183] Example 14: The method of any one of Examples 1-13, further comprising: determining, using medical imaging data of the patient, the anatomical information.

[0184] Example 15: A method, comprising: obtaining, by a device, input information that indicates one or more outcome parameters for a procedure, and wherein the procedure is associated with one or more anatomical elements; determining, by the device and based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements; and generating, by the device and based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements; and providing, by the device for display via a user interface, the three-dimensional model.

[0185] Example 16: The method of Example 15, further comprising: receiving, via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions; modifying the three-dimensional model to a modified three- dimensional model that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs; and providing, for display via the user interface, the modified three-dimensional model.

[0186] Example 17: The method of Example 16, further comprising: generating one or more instructions for a robot based on the one or more modified surgical actions, wherein the one or more instructions are configured to cause the robot to perform at least one of the one or more modified surgical actions; and transmitting, to a navigation system of the robot, the one or more instructions.

[0187] Example 18: The method of any one of Examples 16-17, wherein the input information indicates a type of the procedure, and wherein determining the protocol comprises: determining a template protocol based on the type of the procedure; and determining the protocol based on the template protocol, wherein the protocol includes one or more modifications to the template protocol, and wherein the one or more modifications are based on at least one of the anatomical information or the one or more outcome parameters.

[0188] Example 19: The method of any one of Examples 16-18, wherein a user input of the one or more user inputs is to an area of the three-dimensional model associated with an anatomical element of the one or more anatomical elements, and the method further comprising: identifying a structure of the anatomical element based on a distance between the user inputand the structure on the three-dimensional model; and modifying the user input to a modified user input based on the structure, wherein a modified surgical action of the one or more modified surgical actions is based on the modified user input.

[0189] Example 20: The method of Example 19, wherein identifying the structure comprises: determining that the user input is to the structure based on the distance satisfying a threshold.

[0190] Example 21 : The method of any one of Examples 16-20, wherein the protocol includes one or more steps, and wherein providing, for display via the user interface, the three- dimensional model comprises: providing, for display, the three-dimensional model in multiple anatomical views from respective planes, wherein the one or more user inputs include two or more user inputs to respective anatomical views of the multiple anatomical views; and wherein modifying the three-dimensional model comprises: determining, based on the two or more user inputs, a modified surgical action of the one or more modified surgical actions.

[0191] Example 22: The method of Example 21, wherein determining the modified surgical action comprises: identifying, based on the two or more user inputs, a structure of the one or more anatomical elements indicated by the two or more user inputs; and determining the modified surgical action to be made to the structure based on the two or more user inputs.

[0192] Example 23 : The method of any one of Examples 15-22, wherein the protocol includes multiple protocols, and wherein providing, for display via the user interface, the three- dimensional model comprises: providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective protocols of the multiple protocols.

[0193] Example 24: The method of any one of Examples 15-23, wherein the protocol includes one or more steps, and wherein providing, for display via the user interface, the three- dimensional model comprises: providing, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective versions of at least one step of the one or more steps.

[0194] Example 25: The method of any one of Examples 15-24, wherein the anatomical information indicates structure condition information of a structure of the one or more anatomical elements, and wherein determining the protocol comprises: determining a depth of a surgical action of the one or more surgical actions based on the structure condition information and a safety barrier parameter indicated by the input information, wherein the surgical action is associated with the structure.

[0195] Example 26: The method of any one of Examples 15-25, wherein the anatomical information includes sensitive structure information of one or more sensitive structuresincluded in the one or more anatomical elements, and wherein the protocol is based on the sensitive structure information.

[0196] Example 27: The method of any one of Examples 15-26, wherein determining the protocol comprises: determining the protocol based on a profile of a clinician associated with the procedure, wherein the profile indicates historical information for one or more previously performed procedures.

[0197] Example 28: The method of Example 27, further comprising: updating the profile based on one or more user inputs to the user interface.

[0198] Example 29: The method of any one of Examples 15-28, further comprising: determining, using medical imaging data of the patient, the anatomical information.

[0199] Example 30: A system configured to perform one or more operations recited in one or more of Examples 1-29.

[0200] Example 31: An apparatus comprising means for performing one or more operations recited in one or more of Examples 1-29.

[0201] Example 32: A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising one or more instructions that, when executed by a device, cause the device to perform one or more operations recited in one or more of Examples 1-29.

[0202] Example 33: A computer program product comprising instructions or code for executing one or more operations recited in one or more of Examples 1-29.

[0203] Example 34: A device comprising one or more memories; and one or more processors, communicatively coupled to the one or more memories; configured to perform one or more operations recited in one or more of Examples 1-29.

[0204] Example 35: A device, comprising: one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured to: obtain input information that indicates one or more outcome parameters for a robotically-assisted procedure that is associated with a robot, wherein the robotically-assisted procedure is associated with one or more anatomical elements; determine, based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements, and wherein at least one of the one or more surgical actions is configured to be performed via the robot; generate, based on the protocol and the anatomical information, a three-dimensional model of the one or more anatomical elements, wherein the three-dimensional model shows the one or more surgical actions made to the one or more anatomical elements; provide, for display via a user interface, the three-dimensional model;receive, via the user interface, one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions; modify the three-dimensional model to a modified three-dimensional model that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs; and provide, for display via the user interface, the modified three-dimensional model.

[0205] Example 36: The device of claim Example 35, wherein the one or more processors are further configured to: generate one or more instructions for the robot based on the one or more modified surgical actions, wherein the one or more instructions are configured to cause the robot to perform at least one of the one or more modified surgical actions; and transmit, to a navigation system of the robot, the one or more instructions.

[0206] Example 37: The device of Examples 35 or 36, wherein the input information indicates a type of the robotically-assisted procedure, and wherein the one or more processors, to determine the protocol, are configured to: determine a template protocol based on the type of the robotically-assisted procedure; and determine the protocol based on the template protocol, wherein the protocol includes one or more modifications to the template protocol, and wherein the one or more modifications are based on at least one of the anatomical information or the one or more outcome parameters.

[0207] Example 38: The device of any one of Examples 35-37, wherein the protocol includes multiple protocols, and wherein the one or more processors, to provide, for display via the user interface, the three-dimensional model, are configured to provide, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective protocols of the multiple protocols.

[0208] Example 39: The device of any one of Examples 35-38, wherein the protocol includes one or more steps, and wherein the one or more processors, to provide, for display via the user interface, the three-dimensional model, are configured to provide, for display via the user interface, multiple versions of the three-dimensional model corresponding to respective versions of at least one step of the one or more steps.

[0209] Example 40: The device of any one of Examples 35-40, wherein a user input of the one or more user inputs is to an area of the three-dimensional model associated with an anatomical element of the one or more anatomical elements, and wherein the one or more processors are further configured to: identify a structure of the anatomical element based on a distance between the user input and the structure on the three-dimensional model; and modifythe user input to a modified user input based on the structure, wherein a modified surgical action of the one or more modified surgical actions is based on the modified user input.

[0210] Example 41: The device of Example 40, wherein the one or more processors, to identify the structure, are configured to: determine that the user input is to the structure based on the distance satisfying a threshold.

[0211] Example 42: The device of any one of Examples 35-41, wherein the protocol includes one or more steps, and wherein the one or more processors, to provide, for display via the user interface, the three-dimensional model, are configured to provide, for display, the three- dimensional model in multiple anatomical views from respective planes, wherein the one or more user inputs include two or more user inputs to respective anatomical views of the multiple anatomical views; and wherein the one or more processors, to modify the three-dimensional model, are configured to: determine, based on the two or more user inputs, a modified surgical action of the one or more modified surgical actions.

[0212] Example 43: The device of Example 42, wherein the one or more processors, to determine the modified surgical action, are configured to: identify, based on the two or more user inputs, a structure of the one or more anatomical elements indicated by the two or more user inputs; and determine the modified surgical action to be made to the structure based on the two or more user inputs.

[0213] Example 44: The device of any one of Examples 35-43, wherein the anatomical information indicates structure condition information of a structure of the one or more anatomical elements, and wherein the one or more processors, to determine the protocol, are configured to determine a depth of a surgical action of the one or more surgical actions based on the structure condition information and a safety barrier parameter indicated by the input information, wherein the surgical action is associated with the structure.

[0214] Example 45: The device of any one of Examples 35-44, wherein the anatomical information includes sensitive structure information of one or more sensitive structures included in the one or more anatomical elements, and wherein the protocol is based on the sensitive structure information.

[0215] Example 46: The device of any one of Examples 35-45, wherein one or more processors, to determine the protocol, are configured to determine the protocol based on a profile of a clinician associated with the robotically-assisted procedure, wherein the profile indicates historical information for one or more previously performed procedures.

[0216] Example 47: The device of claim 46, wherein the one or more processors are further configured to update the profile based on the one or more user inputs.

[0217] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the aspects described herein to the precise forms that are described. Modifications and variations may be made in light of the above description or may be acquired from practice of the aspects described herein.

[0218] As used herein, the term “component” is intended to be broadly construed as hardware, firmware, and / or a combination of hardware and software. It will be apparent that systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not limiting of the aspects described herein. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code — it being understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0219] As used herein, satisfying a threshold may, depending on the context, refer to a value being greater than the threshold, greater than or equal to the threshold, less than the threshold, less than or equal to the threshold, equal to the threshold, not equal to the threshold, or the like.

[0220] Even though particular combinations of features are recited in the claims and / or described in the specification, these combinations are not intended to limit the aspects described herein. In fact, many of these features may be combined in ways not specifically recited in the claims and / or described in the specification. Although each dependent claim listed below may directly depend on only one claim, the description includes each dependent claim in combination with every other claim in the claim set. As used herein, a phrase referring to “at least one of’ a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiple of the same item.

[0221] When “a component” or “one or more components” (or another element, such as “a controller” or “one or more controllers”) is described or claimed (within a single claim or across multiple claims) as performing multiple operations or being configured to perform multiple operations, this language is intended to broadly cover a variety of architectures and environments. For example, unless explicitly claimed otherwise (e.g., via the use of “first component” and “second component” or other language that differentiates components in the claims), this language is intended to cover a single component performing or being configured to perform all of the operations, a group of components collectively performing or being configured to perform all of the operations, a first component performing or being configuredto perform a first operation and a second component performing or being configured to perform a second operation, or any combination of components performing or being configured to perform the operations. For example, when a claim has the form “one or more components configured to: perform X; perform Y ; and perform Z,” that claim should be interpreted to mean “one or more components configured to perform X; one or more (possibly different) components configured to perform Y ; and one or more (also possibly different) components configured to perform Z.”

[0222] No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more.” Further, as used herein, the article “the” is intended to include one or more items referenced in connection with the article “the” and may be used interchangeably with “the one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, or a combination of related and unrelated items,), and may be used interchangeably with “one or more.” Where only one item is intended, the phrase “only one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise. Also, as used herein, the term “or” is intended to be inclusive when used in a series and may be used interchangeably with “and / or,” unless explicitly stated otherwise (e.g., if used in combination with “either” or “only one of’).

Claims

WHAT IS CLAIMED IS:

1. A device (510, 600), comprising: one or more memories (630); and one or more processors (620), communicatively coupled to the one or more memories (630), configured to: obtain input information that indicates one or more outcome parameters for a robotically-assisted procedure that is associated with a robot (530), wherein the robotically-assisted procedure is associated with one or more anatomical elements; determine, based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements, and wherein at least one of the one or more surgical actions is configured to be performed via the robot (530); generate, based on the protocol and the anatomical information, a three- dimensional model (210, 310, 410) of the one or more anatomical elements, wherein the three-dimensional model (210, 310, 410) shows the one or more surgical actions made to the one or more anatomical elements; provide, for display via a user interface (200, 300, 400), the three-dimensional model (210, 310, 410); receive, via the user interface (200, 300, 400), one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions; modify the three-dimensional model (210, 310, 410) to a modified three- dimensional model (210, 310, 410) that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs; and provide, for display via the user interface (200, 300, 400), the modified three- dimensional model (210, 310, 410).

2. The device (510, 600) of claim 1, wherein the one or more processors (620) are further configured to: generate one or more instructions for the robot (530) based on the one or more modified surgical actions,wherein the one or more instructions are configured to cause the robot (530) to perform at least one of the one or more modified surgical actions; and transmit, to a navigation system of the robot (530), the one or more instructions.

3. The device (510, 600) of any one of claims 1-2, wherein the input information indicates a type of the robotically-assisted procedure, and wherein the one or more processors (620), to determine the protocol, are configured to: determine a template protocol based on the type of the robotically-assisted procedure; and determine the protocol based on the template protocol, wherein the protocol includes one or more modifications to the template protocol, and wherein the one or more modifications are based on at least one of the anatomical information or the one or more outcome parameters.

4. The device (510, 600) of any one of claims 1-3, wherein the protocol includes multiple protocols, and wherein the one or more processors (620), to provide, for display via the user interface (200, 300, 400), the three-dimensional model (210, 310, 410), are configured to: provide, for display via the user interface (200, 300, 400), multiple versions of the three- dimensional model (210, 310, 410) corresponding to respective protocols of the multiple protocols.

5. The device (510, 600) of any one of claims 1-4, wherein a user input of the one or more user inputs is to an area of the three-dimensional model (210, 310, 410) associated with an anatomical element of the one or more anatomical elements, and wherein the one or more processors (620) are further configured to: identify a structure of the anatomical element based on a distance between the user input and the structure on the three-dimensional model (210, 310, 410); and modify the user input to a modified user input based on the structure, wherein a modified surgical action of the one or more modified surgical actions is based on the modified user input.

6. The device (510, 600) of claim 5, wherein the one or more processors (620), to identify the structure, are configured to:determine that the user input is to the structure based on the distance satisfying a threshold.

7. The device (510, 600) of any one of claims 1-6, wherein the protocol includes one or more steps, and wherein the one or more processors (620), to provide, for display via the user interface (200, 300, 400), the three-dimensional model (210, 310, 410), are configured to: provide, for display, the three-dimensional model (210, 310, 410) in multiple anatomical views from respective planes, wherein the one or more user inputs include two or more user inputs to respective anatomical views of the multiple anatomical views; and wherein the one or more processors (620), to modify the three-dimensional model (210, 310, 410), are configured to: determine, based on the two or more user inputs, a modified surgical action of the one or more modified surgical actions.

8. The device (510, 600) of claim 7, wherein the one or more processors (620), to determine the modified surgical action, are configured to: identify, based on the two or more user inputs, a structure of the one or more anatomical elements indicated by the two or more user inputs; and determine the modified surgical action to be made to the structure based on the two or more user inputs.

9. The device (510, 600) of any one of claims 1-8, wherein the anatomical information indicates structure condition information of a structure of the one or more anatomical elements, and wherein the one or more processors (620), to determine the protocol, are configured to: determine a depth of a surgical action of the one or more surgical actions based on the structure condition information and a safety barrier parameter indicated by the input information, wherein the surgical action is associated with the structure.

10. The device (510, 600) of any one of claims 1-9, wherein the anatomical information includes sensitive structure information of one or more sensitive structures included in the one or more anatomical elements, and wherein the protocol is based on the sensitive structure information.

11. A method, comprising: obtaining, by a device (510, 600), input information that indicates one or more outcome parameters for a procedure, and wherein the procedure is associated with one or more anatomical elements; determining, by the device (510, 600) and based on the one or more outcome parameters and anatomical information of a patient, a protocol to be performed to the one or more anatomical elements, wherein the protocol includes one or more surgical actions to be performed to the one or more anatomical elements; and generating, by the device (510, 600) and based on the protocol and the anatomical information, a three-dimensional model (210, 310, 410) of the one or more anatomical elements, wherein the three-dimensional model (210, 310, 410) shows the one or more surgical actions made to the one or more anatomical elements; and providing, by the device (510, 600) for display via a user interface (200, 300, 400), the three-dimensional model (210, 310, 410).

12. The method of claim 11, further comprising: receiving, via the user interface (200, 300, 400), one or more user inputs indicating one or more modifications to at least one of the one or more surgical actions; modifying the three-dimensional model (210, 310, 410) to a modified three-dimensional model (210, 310, 410) that includes the one or more anatomical elements and shows one or more modified surgical actions made to the one or more anatomical elements, wherein the one or more modified surgical actions are based on the one or more user inputs; and providing, for display via the user interface (200, 300, 400), the modified three- dimensional model (210, 310, 410).

13. The method of claim 12, further comprising: generating one or more instructions for a robot (530) based on the one or more modified surgical actions, wherein the one or more instructions are configured to cause the robot (530) to perform at least one of the one or more modified surgical actions; andtransmitting, to a navigation system of the robot (530), the one or more instructions.

14. The method of any one of claims 11-13, wherein the input information indicates a type of the procedure, and wherein determining the protocol comprises: determining a template protocol based on the type of the procedure; and determining the protocol based on the template protocol, wherein the protocol includes one or more modifications to the template protocol, and wherein the one or more modifications are based on at least one of the anatomical information or the one or more outcome parameters.

15. The method of any one of claims 11-14, wherein a user input of the one or more user inputs is to an area of the three-dimensional model (210, 310, 410) associated with an anatomical element of the one or more anatomical elements, and the method further comprising: identifying a structure of the anatomical element based on a distance between the user input and the structure on the three-dimensional model (210, 310, 410); and modifying the user input to a modified user input based on the structure, wherein a modified surgical action of the one or more modified surgical actions is based on the modified user input.

Images