Robotic surgical system with plan-dependent surgical visualization

Abstract

A surgical system comprising: a tool operable to modify a bone; and circuitry programmed to orient a voxel grid relative to a model of the bone based on a planned bone modification, and to generate a visualization of progress of the tool while performing the planned bone modification using the voxel grid based on tracking data indicative of a location of the tool.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 357,118, filed on June 30, 2022, the entire disclosure of which is incorporated herein by reference. Background Art

[0003] The present disclosure relates generally to methods and systems for providing a visual representation of a cutting process, and more particularly to computer-implemented methods and systems for providing a visual representation of material removed from an object during a cutting process.

[0004] Converting the cutting process into a visual representation is of interest in many applications such as surgery, machining, and milling. For example, there is interest in developing and improving systems and methods for providing a visual representation of the surgical resection process as a cutting tool (e.g., a drill, saw, reamer, etc.) penetrates a patient's bone or tissue. More specifically, providing real-time (or perceived real-time) updates of the progress of bone or tissue removal on a display screen allows a surgeon or other medical personnel to visualize the material removed from the bone or tissue as the cutting tool penetrates the bone or tissue.

[0005] Visualization of the resection process can be computationally expensive, causing delays in some cases, resulting in a lag between the actual progress of bone or tissue removal and the graphical updates on the display screen. In addition, even if the cutting tool follows a smooth, continuous path during the resection process, the visualization often has visual artifacts, such as angular ridge patterns that do not accurately represent the resected surface.

[0006] Therefore, innovations that provide a visual representation of material removal during cutting procedures, such as bone resection, that reduce both computational expense and visualization artifacts are desired. Summary of the invention

[0007] One implementation of the present disclosure is a method. The method includes: providing a virtual skeletal model and a surgical plan defining a planned cut relative to the virtual skeletal model; orienting a voxel grid relative to the skeletal model based on the planned cut; obtaining tracking data indicating the position of a cutting tool; and providing guidance for the execution of the planned cut by generating a visualization based on the voxel grid and the tracking data.

[0008] Another implementation of the present disclosure is one or more non-transitory computer-readable media storing program instructions that, when executed by one or more processors, cause the one or more processors to perform operations. These operations include: providing a skeletal model and a surgical plan defining a planned cut relative to a virtual skeletal model; and orienting the skeletal model relative to a graphics coordinate system based on the planned cut so that the planned cut is orthogonal to a first dimension of the graphics coordinate system, parallel to a second dimension of the graphics coordinate system, and parallel to a third dimension of the graphics coordinate system. These operations also include: generating a visualization of the execution of the planned cut based on the graphics coordinate system, the skeletal model, and tracking data indicating the position of a cutting tool during the execution of the planned cut.

[0009] Another implementation of the present disclosure is a surgical system. The surgical system includes a tool operable to modify a bone. The surgical system also includes a circuit system programmed to: orient a voxel grid relative to a model of the bone based on a planned bone modification, and to generate a visualization of the tool's progress in performing the planned bone modification using the voxel grid based on tracking data indicating the position of the tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a perspective view of a system constructed in accordance with the present disclosure, the system being configured to provide a visual representation of material being removed from an object by a cutting tool during a cutting process.

[0011] Figure 2 is a flow chart of a series of steps involved in generating a visual representation of a cutting process according to a method of the present disclosure.

[0012] Figure 3 is a schematic diagram of the concept of shape sweeping of continuous poses of a cutting tool according to the present disclosure.

[0013] Figure 4 is a schematic diagram of a CSG subtraction operation according to the present disclosure.

[0014] Figure 5 is a schematic diagram of a CSG merging operation according to the present disclosure.

[0015] Figure 6 is a schematic diagram of the CSG intersection operation according to the present disclosure.

[0016] Figure 7 is a flow chart of the steps involved in generating an empty voxelized constructive solid geometry (CSG) mesh for accumulating a union of cutting trajectories according to the method of the present disclosure.

[0017] Figure 8 According to the method of the present disclosure Figure 2 A flow chart of a series of steps that may be involved in a method.

[0018] Fig. 9 is based on Figure 2 A flow chart of a series of steps that may be involved in performing CSG synthesis.

[0019] Fig.10 is a schematic diagram of the concept of defining a valid ray range for non-polygonal isosurface ray marching according to the method of the present disclosure.

[0020] Fig.11 is a schematic diagram of non-polygonal isosurface ray marching to compute the union of cut trajectories subtracted from an object according to the method of the present disclosure.

[0021] FIG. 12 is a schematic diagram of subtracting an unswept cutting trajectory from an object according to the prior art.

[0022] Fig.13 is a schematic diagram of subtracting a swept cut trajectory from an object to produce a smooth cut surface in a visual representation according to the present disclosure.

[0023] Fig.14 is a flow diagram of a plan-dependent visualization process according to some embodiments.

[0024] Fig.15 According to some embodiments Fig.14 Illustration of the skeleton models, cutting planes, and bounding boxes used in some examples of the plan-dependent visualization process.

[0025] Fig.16 According to some embodiments Fig.14 Schematic diagram of the plan-dependent visualization process.

[0026] Fig.17 According to some embodiments Fig.14 Another schematic diagram of the plan-dependent visualization process.

[0027] Fig.18 is a graph showing experimental results according to some embodiments compared with another method of application Fig.14 An illustration of the results of a planned visualization process.

[0028] Fig.19 is a flow diagram of another plan-dependent visualization process according to some embodiments.

[0029] Fig. 20 is an illustration of an application according to some embodiments Fig.19 An illustration of the results of a planned visualization process.

[0030] Fig.21 is an illustration of the use of procedural geometry according to some embodiments.

[0031] Fig. 22 is a flowchart of a process for generating a visualization using procedural geometry, according to some embodiments.

[0032] Fig.23 is another flow chart related to generating visualizations using procedural geometry, according to some embodiments.

[0033] Fig.24 is a flow diagram of a process for providing a visualization with transparency, according to some embodiments.

[0034] Fig.25 is a diagram illustrating features for providing visualization with transparency according to some embodiments.

[0035] It should be understood that the drawings are not necessarily drawn to scale. It should also be understood that the following detailed description is merely exemplary in nature and is not intended to limit the invention or its application and use. Therefore, although the present disclosure is depicted and described as certain illustrative embodiments for ease of explanation, it should be understood that the present disclosure can be implemented in various other types of embodiments and in various other systems and environments. DETAILED DESCRIPTION

[0036] The present disclosure relates to methods and systems for providing a visual representation of material removal from an object by a cutting tool during a cutting process. For example, the methods and systems disclosed herein can be used to provide a visual representation of the progress of removing bone material from a bone during a surgical bone removal process. In other non-limiting examples, the methods and systems disclosed herein can be used to provide a visual representation of material removal in a machining or milling operation. The visual representation can be provided on a display that is constantly updated as the cutting tool penetrates the object so that a user (e.g., a surgeon, etc.) can monitor the progress of material removal in real time (perceived real time). This allows the user to confirm that the planned volume of the object has been removed, and to obtain a notification (and visualization of such penetration) if the cutting tool penetrates beyond the planned volume. The following is a list of terms and phrases used throughout the present disclosure and their meanings as used herein, many or all of which are apparent to those of ordinary skill in the art.

[0037] Constructive Solid Geometry (CSG): A technique for displaying complex geometric models by combining multiple models using Boolean set operations such as union, intersection, and subtraction.

[0038] CSG operations: Mathematical operations used to combine multiple models to display complex geometric models, such as Boolean set operations such as union, intersection, and subtraction.

[0039] Voxel: Volume pixel (the volume equivalent of a pixel); each element in the array of discrete elements into which a three-dimensional object representation is divided. Each voxel has a 3D coordinate and holds a color at that coordinate.

[0040] Object space: represented in a 3D voxelized grid.

[0041] Rasterization: A concept in computer graphics that describes how polygonal geometry is converted into a raster (pixel or voxel) image.

[0042] A-buffer rasterization: A visible surface detection technique that identifies what is visible in a scene from a specific viewpoint of a virtual camera; an extension of the Z-buffer method that adds transparency. A-buffer rasterization creates an A-buffer that contains values ​​that describe the depth complexity of the scene.

[0043] Fragment: One of several surfaces that make up a pixel and contribute to the pixel's color.

[0044] Cutting trajectory: The path followed by the cutting tool during the cutting process; the volume that the cutting tool sweeps through the object up to the current point in time.

[0045] Referring now to the drawings, and in particular to Figure 1 , a system 10 for performing a cutting process on an object and providing a visual representation of material removed from the object during the cutting process is shown. As a non-limiting example, the cutting process may be a surgical tissue or bone removal process, such as hip or knee arthroplasty. Alternatively, the cutting process may be a machining or milling process. The system 10 may include a robotic system 12 having a robotic arm 14 that holds and moves a cutting tool 16 (e.g., a drill cone, a saw, a reamer, etc.) during the cutting process. The system 10 may also include a tracking device 18 having an optical positioning system 19 (including an optical positioning camera) that tracks the position (position / orientation) of the cutting tool 16 and the object to be cut (e.g., a bone, etc.) during the cutting process. The tracking device 18 may be positioned at a support or at another location separate from the robotic system 12, or it may be positioned at the robotic system 12 or incorporated into the robotic system. In addition, the system 10 may include one or more display screens 20 for displaying a visual representation of the cutting process and a navigation system 22 that communicates with the tracking device 18 and the display screen 20. The navigation system 22 may include a computer 24 having a computer processing unit (CPU) 26 configured to execute one or more algorithms that calculate and render visual representations and output the visual representations to the display screen 20, as described in further detail below. In some arrangements, the CPU 26 may include a graphics processing unit (GPU) 28 to assist the CPU 26 in performing rendering calculations.

[0046] Now go to Figure 2 , a method 30 for generating a visual representation of a cutting process that can be executed by the CPU 26 is shown. Figures 8 to 9 Further details of the steps of the method 30 are provided. In a first block 32, a continuous pose of the cutting tool 16 following its cutting trajectory during the cutting process can be tracked based on input from the optical positioning system. Each continuous pose in the continuous pose represents a discrete position / orientation of the cutting tool 16 at a specific point in time captured by the optical positioning system.

[0047] The shape sweep of the continuous poses of the cutting tool 16 up to a certain point in time may be performed according to block 34. Figure 3 The concept of shape sweeping performed during block 34 is schematically depicted in FIG. 1 . Although the cutting tool 16 may actually follow a smooth cutting path / trajectory during the cutting process, the tracked continuous poses 36 of the cutting tool 16 may be detected as discrete shapes due to the sparse temporal sampling of the poses 36 by the tracking device 18. For example, if the cutting tool 16 is a drilling cone, the tracked continuous poses 36 may be different spherical shapes, such as Figure 3 shown.

[0048] In order to correct for the temporal resolution of the tracking device 18 and provide a smoother visual output that more closely reflects the smooth cutting path followed by the cutting tool 16, a shape sweep 34 can be applied to the continuous pose 36, as shown. The shape sweep 34 can enclose the continuous pose 36 in a smooth surface 38 to provide a smooth representation of the cutting trajectory 40 followed by the cutting tool 16. The shape sweep process provides a smoother visual representation of the cut object, as described below with reference to Fig.13 Further explained.

[0049] exist Figure 3 In the example of , shape sweep 34 combines spherical poses 36 into a capsule-like structure with a smooth surface 38. However, it should be noted that shape sweeping as disclosed herein can be applied to a variety of cutting tools with different shapes and can be performed on any closed pose geometry, such as but not limited to drill cone-ellipsoid / sphere, saw-rectangular cuboid, and reamer / hemisphere shapes. The swept shape can be generated in any suitable manner, such as via piecewise linear or high-order (e.g., spline) approximation.

[0050] Back to Figure 2, the union of the cutting trajectories 40 may be accumulated in a three-dimensional voxelized constructed solid geometry (CSG) mesh to produce an object space representation of the union of the cutting trajectories 40 up to a particular point in time during the cutting process (block 42). As the cutting process proceeds, the voxelized CSG mesh may be continuously updated with the union of the cutting trajectories. That is, the voxelized CSG mesh may be iteratively updated with the current cutting trajectory as the position of the tool changes and more material is removed from the object during the cutting process (see below).

[0051] According to the next block 44, CSG synthesis may be performed. Fig. 9 In addition to the other steps further described, the CSG synthesis step may involve performing a CSG operation on an object space representation of the union of the cutting trajectory 40 and the three-dimensional model 41 of the object being cut. For example, if the object is a bone, a three-dimensional model of the bone can be generated by segmenting a computed tomography (CT) scan of the bone. If the object being cut has softer properties, such as meat or a tumor, the three-dimensional model can change slightly between consecutive renderings. The CSG operation can be performed using non-polygonal isosurface ray stepping, as further described below with reference to FIG. 12.

[0052] In the case of a surgical bone resection procedure, for example, the CSG operation performed during frame 44 may be to subtract the union of the cutting trajectories 40 from the bone model to produce a 2D image of the resected bone. However, depending on the procedure performed on the object, other types of CSG operations, such as merging, intersecting, and drawing, may be performed. Further descriptions of certain types of CSG operations that may be performed during frame 44 are provided below.

[0053] In CSG subtraction (see Figure 4 ), subtracting an object from a master object. The subtraction operation produces a solid with a boundary consisting of (a) all external points of the master object that are outside the subtracted object and (b) all external points of the subtracted object that are inside the master object. For example, the subtraction 46 of a cube 48 that intersects one quarter of a sphere 50 may be a three-quarter sphere 52 (see Figure 4 ). In the CSG merge operation (see Figure 5 ), creates an object that combines two or more primitive objects. Points that are on the boundary of one of the combined objects but also inside the other objects are not visible in the combined object. Figure 5 The union 54 of two spheres 50 is schematically depicted in FIG. In a CSG intersection operation, an object is created from primitive objects, where the boundary consists of all points outside one object and inside the other object. For example, Figure 6 The result of the intersection 56 of the sphere 50 and the cube 48 is depicted.

[0054] Reference again Figure 2 When the frame 44 is completed, the CPU 26 can output the 2D image of the result of the CSG operation to the display screen 20 (frame 60). By repeating the method 30 and continuously displaying the updated 2D image on the display screen 20, the 2D image can be continuously updated during the entire cutting process. Figure 2 The method may be repeated with sufficient frequency so that the 2D image is updated in real time, or at least with sufficient frequency so that the user perceives that the updates are occurring in real time (perceived real time).

[0055] Before accumulating the cutting trajectory 40 into the voxelized CSG grid, it can be based on Figure 7 An empty voxelized CSG grid is generated by method 62 of the CSG operation. Starting from box 64, the spatial boundary of the CSG operation can be defined as the intersection of the bounding box (geometric surface) of the object being cut and the bounding box of the allowed volume of objects that can be removed during the cutting process. Optionally, an outer margin can be added to the intersection to allow cutting performed outside the allowed volume to be visualized on the display screen. Once the spatial boundary of the CSG operation has been defined by box 64, an empty voxelized CSG grid is generated within the defined spatial boundary according to box 66.

[0056] Now go to Figure 8 , schematically depicting Figure 2 At block 70, the cutting trajectory 40 may be calculated using a shape sweep 34 that takes into account the shape of the cutting tool 16, the tracked continuous pose 36, and the parameters of the shape sweep (see also Figure 3 ). The cutting trajectory 40 may then be rasterized into a voxelized CSG grid using the GPU 28 or a combination of the CPU 26 and the GPU 28. For example, the rasterization may be performed on the CPU 26 according to box 72, and only the subvolume of the cutting trajectory 40 that has changed since the last update may be transferred to the GPU 28 according to box 74. Alternatively, the rasterization of the cutting trajectory 40 may be performed on the GPU 28 according to box 76. If performed on the GPU 28, the rasterization may be performed using layered rendering (layer-by-layer rasterization) because the GPU 28 may only be capable of rendering 2D images. Therefore, it may be decided whether to perform the rasterization by the boxes 72 and 74 or by the box 76, depending on which strategy is more efficient. Figure 8 Also shown is the option of using a procedural geometry mesh (block 84), as described below.

[0057] After rasterization of the cutting trajectory 40 is completed, the voxelized CSG grid may be updated with the changed subvolume of the cutting trajectory 40 according to the next block 78. By continuously repeating blocks 70, 72 and 74 (or 76) and 78 throughout the cutting process, the union of the cutting trajectory 40 may be iteratively updated in the voxelized CSG grid until the current point in time of the cutting process.

[0058] In addition, the A buffer (block 80) of the object model 41 can be created by performing a single-pass A-buffer rasterization (block 82) on the object model 41 without depth testing. Single-pass A-buffer rasterization can reduce the cost of rasterization because more passes increase the cost of the calculation. The A-buffer can be a data structure that describes the depth complexity of an object in image space from the viewpoint of the virtual camera. The A-buffer and the updated voxelized CSG grid can be used in the CSG synthesis step (block 44), as described in further detail below.

[0059] In an embodiment using a procedural geometry mesh (i.e., executing block 84), the cut trajectory accumulated using the shape sweep in block 70 is provided to block 84. Block 84 includes building a procedural geometry mesh on the GPU 28 at block 86 and then ray traversing the procedural geometry mesh on the GPU 28. As used herein, procedural geometry refers to shapes represented mathematically (e.g., general primitives) and may include, for example, spheres, capsules, cylinders, boxes, planes, cones, wedges, etc. For example, a sphere may be mathematically represented by a center point and a radius (e.g., rather than a surface of a tessellation or other method). A procedural geometry mesh refers to a three-dimensional bounding box (e.g., an axis-aligned bounding box) divided into voxels, where each voxel is associated with a list (e.g., a linked list) of procedural geometry. Fig.21 An example illustration of a procedural geometry mesh 2100 with associated procedural geometry is shown, shown as a first sphere 2102, a second sphere 2104, a third sphere 2106, and a fourth sphere 2108. As shown, the first sphere 2102 is associated with the first voxel 2110, while the second sphere 2104 and the third sphere 2106 are associated with the second voxel 2112, and the fourth sphere 2108 is associated with the third voxel 2114. In various embodiments, different numbers of different procedural geometry shapes may occupy voxels. This method allows mathematical ideals of such spheres and other shapes to be directly represented in the method used for visualization in block 84. Further details of this method are described below. Figure 22 to Figure 23 Thus, according to some embodiments, block 84 is adapted to provide a mesh that can be used in the CSG synthesis step of block 44.

[0060] Now go to Fig. 9 , shows the steps that may be involved in performing CSG synthesis (block 44). At a first block 90, a classification of fragments from the object model A buffer may be performed. Specifically, block 90 may involve reading fragments from the A buffer and classifying the fragments according to their depth.

[0061] At block 92, an effective ray range for non-polygonal isosurface ray marching may be defined to reduce the effective ray range for ray marching and improve computational efficiency. Fig.10 A schematic diagram of defining the effective light range performed during block 92 is shown in FIG. Fig.10 A voxelized CSG grid 94, a union of a cutting trajectory 96, and a bounding box 98 of an allowed volume of an object 100 (e.g., a bone) that is allowed to be cut during the cutting process are shown. In non-polygonal isosurface ray marching, multiple rays (e.g., rays 102 and 104) from a particular viewpoint of a virtual camera can be cast or marched through the voxelized CSG grid 94 to iteratively search for intersections. The intersections can be used to generate an image space (2D) surface representation of the cutting process displayed on the display screen 20. Each of the rays 102 and 104 can correspond to a pixel in the generated 2D image. Defining a valid ray range according to box 92 limits the search for ray marching and improves the efficiency of CSG operations. More specifically, according to box 92, if the bounding box 98 of the object 100 and the voxelized CSG grid 94 do not overlap along the ray (see the dashed portion of the exemplary rays 102 and 104 corresponding to the "invalid" ray range), then the ray marching can be skipped (or the ray marching may be "invalid"). Within the "invalid" ray range, no calculations are performed. Otherwise, the ray is iterated within the "valid" ray range, as shown by the solid portion of ray 104. Within the "valid" ray range, the start of the step is the closest surface of the object 100 (from the A-buffer) or where the ray enters the voxelized CSG grid 94, whichever is farther (Equation 1). Ray stepping ends at the farthest intersection of the ray with the object 100 (from the A-buffer), or if the CSG grid 94 is exited, whichever occurs first (Equation 2). For example, in Fig.10 , the dashed portions of exemplary rays 102 and 104 are skipped during stepping because they are outside of object 100, wherein ray 102 terminates immediately upon reaching object 100 because CSG grid 94 is not set at this time.

[0062] t start =max(t object_closest ,t grid_entry )(Equation 1)

[0063] t end =min(t grid _ exit t object_furthest )(Equation 2)

[0064] With the valid ray range defined as described above, a CSG operation may be performed using non-polygonal isosurface ray marching on the union of the object 100 (from the A-buffer) and the cut trajectory 96 (block 106). It will be appreciated that other CSG operations may be implemented, such as surface rendering, intersection, etc. Fig.11An example of using non-polygonal isosurface ray walking to display the result of subtracting the union of cutting trajectories 96 from an object 100 is depicted. For each pixel, from the viewpoint of the virtual camera, a ray 104 is stepped through the valid ray range (the solid portion of the ray 104), and an intersection point 108 is determined as the closest intersection point that is inside the object 100 and outside the union of the cutting trajectories 96. The intersection point 108 represents the cutting surface of the object 100 and is used to generate a 2D surface representation that is displayed on the display screen 20.

[0065] If the union of the object 100 and the cutting trajectory 96 does not overlap along the ray (indicating that the object is not cut at that location), the object surface is displayed as not cut in the 2D image. Iteratively searching the intersection of the voxelized CSG grid 94 in this manner produces a 2D surface representation of the object as it is cut during the cutting process. For example, in the case of bone resection, block 106 can produce a 2D surface representation of the resected bone surface.

[0066] Reference again Fig. 9 , the CSG synthesis block 44 may also include discarding or masking segments of the 2D surface representation resulting from the CSG operation to refine the 2D image (block 110). Specifically, during block 110, a machining calculation may be applied that determines the shading of each segment and determines the red, green, and blue contributions of the segment at each pixel. In the case of a surgical bone resection procedure, the bone area to be resected may be colored green, while the area where the cutting tool 16 has penetrated beyond the planned area may be colored red. Block 110 may also involve discarding segments that are not visible. The refined 2D image may be displayed on the display screen 20 ( Figure 2 60).

[0067] In another embodiment, a display of the total removed material, the total remaining material to be removed, or both may be generated to provide guidance to the user. For example, the display may indicate that 80% of the material to be removed has been removed, while 20% of the material to be removed remains on the object.

[0068] The present application applies shape sweeping 34 to produce a smoother visual representation of the cutting process. Figure 12 shows the object surface 114 produced after subtracting the unswept cutting pose 110 from the object model 112 according to the prior art. It can be seen that due to the temporal resolution of the tracking method used to track the pose of the cutting tool 16, the pose 110 can be detected as a discrete shape representing the shape of the cutting tool 16. As a result, after subtracting 115 the unswept cutting pose 110, the displayed object surface 114 can have angular, wavy ridges 116 that are artifacts of the sparse temporal sampling of the cutting pose 110. In contrast, subtracting the swept cutting pose 118 from the object model 112 according to the present disclosure avoids the remaining ridges and produces a smoother visual representation of the cut object surface 114.

[0069] Thus, it can be generally seen that the technology disclosed herein can be applied to a variety of settings, such as, but not limited to, surgery and machining. More specifically, the present disclosure can be applied to a variety of applications that can benefit from real-time visualization of the progress of material removal from an object during a manual or automated cutting process. Although robotic bone resection is given as an example, the technology disclosed herein can also be applied to manual surgery, such as manual bone resection, cutting or milling operations, and other types of operations.

[0070] Current methods for generating visual representations of bone removal processes can be computationally expensive due to the need to subtract a large number of tool positions from the bone model. The computational cost of current methods that rely on depth peeling can increase as the depth complexity of the scene increases. Furthermore, if the virtual camera is moved or rotated, the computational results may be invalid and need to be recomputed.

[0071] In contrast, the above method has a constant computational cost relative to the depth complexity of the scene. The calculation of the union of the cutting trajectories as disclosed herein occurs incrementally at a constant computational cost by rasterizing the model of the cutting trajectories into a voxelized CSG grid that (spatially) covers the planned resection volume (with optional outer margins). Creating image space fragments describing the voxelized model in pixels also causes the constant computational cost of ray stepping through the voxelized CSG grid to occur. The resolution of the voxelized CSG grid determines the computational cost and is fixed in advance. The voxelized CSG representation also eliminates the burden of handling overlapping or repeated positions. In addition, during non-polygonal isosurface ray stepping, high visual quality of the resected surface is achieved by subvoxel accurate filtering. The initial isosurface intersection is refined in a binary search manner by reducing the step size.

[0072] As disclosed herein, a voxelized CSG representation can be rasterized on the CPU or GPU by rendering the CSG mesh hierarchically into slices, depending on which rasterization strategy is more efficient. If rasterization occurs on the CPU, only the subvolume of the cutting trajectory that has changed since the last update is uploaded to the GPU to further improve efficiency. The subvolume is defined by the bounding box of the updated cutting pose of the cutting trajectory.

[0073] The present disclosure also provides another improvement in the quality of visual representation relative to existing methods. In the current technology, due to sparse time sampling of the tool position tracking device, the continuous poses of the cutting tool may be too far apart, resulting in the remaining ridges in the displayed image may not accurately reflect the smooth cutting path followed by the cutting tool. The present disclosure applies shape sweeping to represent the continuous poses within the smooth bounding surface to prevent ridge artifacts from appearing in the visual representation of the cutting process. Advantageously, this results in a visually smoother resection surface in the displayed image, which can more closely represent the true appearance of the resected object.

[0074] Further technical advantages can be obtained by additionally or alternatively applying the following references Figures 14 to 19 The features described are implemented. As described in detail below, computational overhead can be further reduced by orienting a graphics coordinate system (e.g., a voxel grid, a voxelized CSG grid 94) based on a cut plan (e.g., a surgical plan) and / or defining a volume for visualization calculations (e.g., defining a bounding box 98), for example by reducing the total volume used for rasterization and CSG operations while capturing areas of interest to a user. This approach can also reduce visualization artifacts, for example by aligning the voxel grid (e.g., a voxelized CSG grid 94) with the planned cut so that successfully performing the planned cut results in surfaces in the aligned voxel grid being inherently smooth. Therefore, as will be apparent from the following description, technical improvements in graphics visualization techniques, particularly useful in the field of surgical navigation and guidance technology, and similar applications are provided by Figures 14 to 19 The features shown in and described in detail below are provided.

[0075] Reference now Fig.14 , a flow chart of process 1400 according to some embodiments is shown. In various embodiments, process 1400 can be performed by a surgical system, such as a robotic surgical system, a surgical navigation system, etc. For example, in some embodiments, process 1400 can be performed by system 10, such as by or using navigation system 22 (e.g., by CPU 26). In some embodiments, process 1400 can optionally be performed in conjunction with method 30 and / or method 62 described above. For example, Fig.14 The discussion referred to Fig.15 , Fig.15 An illustration of a skeletal model 1500, a planned skeletal modification plane 1502, and a bounding box 1504 is shown in accordance with some embodiments.

[0076] At step 1402, a skeletal model is provided. The skeletal model is a three-dimensional virtual representation of a skeleton. For example, in some embodiments, the skeletal model is generated from a segmented CT image (or other type of medical imaging) of the patient's skeleton. In some embodiments, the skeletal model is a generic (standard, non-patient-specific) model of the skeleton. As an example, Fig.15 A skeletal model 1500 of a femur is shown.

[0077] In step 1404, a surgical plan is provided, which indicates a plan (planned cuts, cutting plans, planned bone modifications) of bone modifications (cutting, grinding, drilling, carving, etc.) relative to the bone model. The surgical plan may also indicate one or more additional bone modifications (additional planned cuts), such as a series of bone modifications planned to prepare the bone surface to receive an implant. In some embodiments, step 1404 includes automatically generating a surgical plan based on patient characteristics and / or providing a graphical user interface that allows a user to adjust the surgical plan (e.g., by adjusting the planned implant position relative to the bone model). In some examples, the planned cuts indicated by the surgical plan are flat (or planar) and are defined relative to the bone model. For example, a plane with an orientation and position relative to the position and orientation of the bone model can be provided in step 1404. As an example, Fig.15 The planned bone modification plane 1502 is shown positioned and oriented relative to the bone model 1500.

[0078] In step 1406, a voxel grid (e.g., voxelized CSG grid 94) or other graphics coordinate system is oriented relative to the skeletal model based on the planned skeletal modification. Orienting the graphics coordinate system in step 1406 may be performed such that the planned skeletal modification has a constant value in one dimension of the three-dimensional graphics coordinate system. For example, where the planned skeletal modification is planar, step 1406 may include aligning the voxel grid with the planned skeletal modification such that a first dimension of the voxel grid is perpendicular to the planned skeletal modification, a second dimension of the voxel grid is parallel to the planned skeletal modification, and a third dimension of the voxel grid is parallel to the planned skeletal modification. Fig.15 In the example of FIG. 1406 , step 1406 includes orienting the voxel grid based on the planned skeleton modification plane 1502 so that the voxel grid (which itself has orthogonal dimensions, such as Fig.15 The cubic voxel in the example of ) has two coordinate dimensions parallel to the planned bone modification plane 1502 and one coordinate dimension perpendicular to the planned bone modification plane 1502, i.e., such that the planned bone modification plane 1502 is a constant value in one dimension of the voxel grid and extends in all directions in the other two dimensions of the voxel grid.

[0079] Orienting the voxel grid or other graphical coordinate system relative to the skeletal model in step 1406 can also be characterized as orienting the skeletal model relative to the voxel grid. Coordinate transformations can be used to translate between a voxel grid aligned with a planned skeletal modification and other coordinate systems used (e.g., by system 10) during navigation or control (e.g., used by a tracking system, for robotic control, etc.).

[0080] In step 1408, a bounding box is defined based on the planned skeleton modification and voxel grid (or other graphics coordinate system). The bounding box can be a cube or rectangular box defined as a square relative to the voxel grid (e.g., each surface of the bounding box is planar and has a constant value in the dimension of the graphics coordinate system), which can simplify calculations compared to defining the bounding box in some other ways. Step 1408 may include spatially locating the bounding box, wherein one surface is located at or slightly close to the first side of the planned skeleton modification (e.g., distance one millimeter, two millimeters, three millimeters, etc.), and the remainder of the bounding box is located to the second opposite side of the planned skeleton modification. Step 1408 may also include adjusting the size of the bounding box so that all parts of the skeleton model on the second side of the planned skeleton modification (i.e., the part of the planned excision) are enclosed in the bounding box, and in some embodiments, the scope of the bounding box is limited to the ends of these parts of the skeleton model on each coordinate dimension (e.g., with a slight offset).

[0081] Fig.15 An example is shown in which a bounding box 1504 is positioned at a planned bone modification plane 1502 and is sized to enclose a portion 1506 of the planned resection of the bone model 1500 (i.e., a portion separated from the rest of the bone model 1500 by the planned bone modification plane 1502). Thus, the bounding box 1504 is narrowly defined to include portions of the bone model 1500 that are intended to be affected during the execution of the surgical plan for the planned bone modification, and does not include substantial portions of the bone model that are not intended to be affected by the planned bone modification. Thus, the volume of the bounding box 1504 can be minimized while capturing all areas of interest to the user for visualization. Therefore, the bounding box 1504 defined based on the planned bone modification and voxel grid in step 1408 is limited to minimize the amount of rasterization, CSG calculations, ray casting, and / or other calculations required to generate the visualization, thereby reducing computational overhead and improving computational efficiency.

[0082] In step 1410, a real-time visualization of the execution of the planned skeletal modification is generated using the skeletal model, the voxel grid, and, in some embodiments, data indicating the position of a tool used to perform the planned skeletal modification (e.g., from a tracking system). In some embodiments, step 1410 includes determining (e.g., using a CSG operation as described above) voxels of the voxel grid that are shared by the skeletal model and the accumulation (e.g., sweep) of the tool positions, and updating the visualization to remove the visualization of such voxels (or to color, shade, make them transparent, etc.). Step 1410 may include generating a representation of the surface of the skeletal model that is produced by removing the virtual bone from the voxels shared by the skeletal model and the accumulation of the tool positions. For example, step 1410 may use the above reference to Figures 1 to 13 Describe the method to perform.

[0083] Because the planned skeletal modifications are aligned to the voxel grid (or other graphics coordinate system), the visualization generated in step 1410 is in an inherently smooth surface when the skeletal modifications are performed according to the plan. Figure 16 to Figure 17 A schematic diagram illustrating this effect is included. Fig.16 A cutting tool 1600 is shown relative to the skeletal model 1500, along with a plane alignment box 1602 and a bone axis alignment box 1604. The plane alignment box 1602 shows the orientation of the voxel grid arranged relative to the skeletal model 1500 according to the steps of the process 1400, while the bone axis alignment box 1604 shows the voxel grid aligned with the skeletal model 1500 regardless of the planned skeletal modification. Fig.16 As shown, a surgeon following the surgical plan will operate cutting tool 1600 in a direction parallel to the surface of plane alignment box 1602 (e.g., along the coordinate direction of the voxel grid), that is, cutting tool 1600 is aligned with the plane and therefore with box 1602 as expected. This alignment and parallel movement can simplify calculations and reduce visualization artifacts. In contrast, surgical tool 1600 is shown as being at an angle relative to all surfaces and coordinate directions of bone axis alignment box 1604, which may result in visualization artifacts, such as steps between voxels, such as Fig.17 shown.

[0084] Fig.17 A grid 1700 is shown (shown in two dimensions for purposes of explanation, wherein the principles shown can be extended to three or more dimensions). Fig.17 As shown, first line 1702 extends through grid 1700 in a direction that is not parallel to any (any) coordinate dimension of grid 1700 . Fig.17The boxes 1704 (voxels) of the grid 1700 below the first line 1702 are also shown as shaded / dashed lines. Because the first line 1702 is angled relative to the grid 1700, the boxes 1704 below the first line 1702 form steps (ridges, jumps, artifacts, etc.). Therefore, a visualization generated using the boxes 1704 will include visualization artifacts, i.e., steps, thereby inaccurately representing the straight line 1702.

[0085] Fig.17 Also shown is a second line 1706 aligned with the grid 1700. Since the grid 1700 is oriented based on the surgical plan in process 1400, the second line 1706 can be aligned with the grid 1700. Because the second line 1704 is aligned with the grid 1700, the box 1708 below the second line 1706 can be highlighted without steps, jumps, discontinuities, etc. Therefore, the generated visualization of the box 1708 accurately represents the smooth linear contour of the second line 1706 without visualization artifacts. In the case of process 1400, the line 1704 can correspond to the position of the tool when the planned bone modification is performed according to the surgical plan. Therefore, when the tool moves along such a line (surface, etc.), the visualization based on the tracking data can inherently result in no visualization artifacts or limited visualization artifacts as an effect of aligning the voxel grid based on the surgical plan in step 1406.

[0086] In some such embodiments, no or only limited post-processing (e.g., smoothing, interpolation, etc.) is required to achieve a smooth appearance of the visualization, so that the visualization corresponds more closely to the underlying data (e.g., tracking data) than an approach in which some post-processing method is applied to hide the steps between boxes 1704. Computational efficiency may also be improved by minimizing the need for such post-processing.

[0087] Fig.18 An example visualization based on experimental results is shown. Specifically, Fig.18 1800 and a second illustration 1802 generated after process 1400 (e.g., including aligning the voxel grid with the planned bone modification). The first illustration 1800 includes visualization artifacts caused by the lack of alignment with the voxel grid, resulting in the appearance of a rough surface 1804. The second illustration 1802 does not have such visualization artifacts because the bone modification caused the voxel grid to be aligned with a smooth surface 1806. Therefore, Fig.18 Experimental results illustrating the improvements in visualization quality produced by the techniques described herein, including by performing process 1400 , are provided.

[0088] Reference again Fig.14 , Fig.14It is shown that process 1400 can return to step 1406 from step 1410. Specifically, for processes involving multiple planned bone modifications (e.g., making multiple cuts at different angles to prepare the femur to receive an implant), process 1400 may include responding to an indication that a first (previous, etc.) planned bone modification is completed (e.g., based on user input, based on detecting from tracking data that the tool has completed the full range of planned cuts, etc.), based on additional (subsequent, second, etc.) planned bone modifications to re-orient the voxel grid. Re-orienting the voxels may include changing the voxel grid from being aligned with the first planned bone modification to being aligned with the additional planned bone modification (e.g., so that the first dimension of the voxel grid becomes perpendicular to the additional planned bone modification, while the second and third dimensions of the voxel grid are parallel to the additional planned bone modification). Thus, the voxel grid is updated (re-orientated, moved, changed, rotated, etc.) for different stages of the surgical procedure (e.g., different cuts, different bone modifications, different sub-portions of continuous bone modifications, etc.) to provide efficient, artifact-free visualization for multiple stages of the surgical procedure.

[0089] Reference now Fig.19 , a flowchart of a process 1900 for providing a plan-dependent visualization feature according to some embodiments is shown. In some embodiments, process 1900 can be used together with process 1400, for example as Fig.14 The step 1410 of using the skeletal model and tool position to generate a portion of the real-time visualization of the execution of the planned skeletal modification. Process 1900 may also be used in an implementation that omits some or all of process 1400. Process 1900 may be performed by one or more processors executing instructions stored in a non-transitory computer-readable medium, such as by system 10, such as by navigation system 22 or using the navigation system (e.g., by CPU 26). Process 1900 may advantageously use the surgical plan to eliminate visualization artifacts of the bone surface.

[0090] In step 1902, surface normals of the modified bone surface are determined at a plurality of points. The plurality of points may be points where the intersection 108 is near a threshold of the planned bone modification (e.g., the planned cutting plane). For example, the points used in step 1902 may be selected as || n plan (po plan )||<∈, where p is the position of the intersection point 108, n plan is the normal vector of the planned bone modification (e.g., the planned resection plane), and o planis the origin of the planned bone modification (e.g., the planned resection plane), and ∈ is a fixed threshold selected, for example, based on the expected amount of deviation in the cut, tolerances in the cutting workflow, etc. For such a point, the normal of the surface of the bone modification (e.g., the actual resection surface) can be determined, for example, by using a volume gradient method applied to the voxel representation of the bone remaining after the CSG operation described herein. The normal of the bone modification surface calculated from the voxel representation of a given point is denoted herein as n grid .

[0091] In step 1904, based on n plan (normal vector of the planned bone modification) and n grid Calculate the adjusted normal (based on the normal of the modified bone surface calculated from the voxel grid representation). The adjustment can be an interpolation between the two normals. In some embodiments, the adjusted normal n adjusted is calculated as n adjusted =(n grid *α)+(n plan *(1-α)), where α is a weighting parameter. In some embodiments, α has a value between 0.7 and 0.9, such as about 0.85. In some embodiments, α is a fixed value. In some embodiments, α is a function of the distance from the intersection 108 to the planned bone modification.

[0092] To apply normal modification when the resection involves multiple resection planes, some embodiments of process 1900 include tracking the associated resection planes during raycasting of the raster volume. Since each raster volume can have a unique resection plane associated with it, information identifying the corresponding plan surface can be stored with intersection point 108 (added to the intersection point, labeled to the intersection point, etc.). During the CSG subtraction step described above, this information is used in some embodiments to identify which n is used in the instance of steps 1902 and 1904. plan and which o plan .

[0093] In step 1906, the adjusted normal (i.e., n adjusted The normal after adjustment can be used in the surface shading method, for example, using Phong shading to produce visualization. The normal after adjustment can be used in step 1906 to reduce or eliminate the artifacts that appear on the modified bone surface, so that the surface looks flat, smooth, etc. when displayed to the user.

[0094] Fig. 20A comparison of a first visualization 2000 prepared without process 1900 and a second visualization 2002 prepared by applying process 1900 is shown. In the first visualization 2000, several steps (steps between levels of the voxel grid) are shown as a result of the voxel representation. Advantageously, the second visualization 2002 prepared by applying process 1900 shows a smooth modified bone surface without such artifacts. This allows the display of a representation of a cut bone surface with a smooth surface, as a user (e.g., a surgeon) might expect to result from bone modification. Thus, the plan-dependent visualization techniques as described herein provide technical solutions in the field of graphical visualization technology for computer-assisted surgery, including technical solutions for robotic surgical systems.

[0095] Reference now Fig. 22 , a flowchart of a process 2200 for providing visualization using procedural geometry according to some embodiments is shown. The process 2200 may be performed by one or more processors executing instructions stored in a non-transitory computer-readable medium, such as by the system 10, such as by the navigation system 22 or using the navigation system (e.g., by the GPU 28).

[0096] In step 2202, a skeletal model and a planned resection are provided. The skeletal model may be provided as described above, such as based on segmentation of preoperative images (e.g., MRI, CT, etc.) of the patient's bones, based on a library of stored skeletal models, based on intraoperative tracking of a probe contacting the bone surface, etc. The planned resection may be provided based on the results of a surgical planning workflow provided, for example, via a graphical user interface of system 10. The planned resection may be a surface (e.g., a curved surface, a plane) and / or a volume with a planned posture (position and orientation) relative to the skeletal model. The data related to the planned resection provided in step 2202 may include the selection of the tool used in the planned resection, the planned posture that the tool will be in when performing the planned resection, and / or other information related to the tool to be used in the planned resection.

[0097] At step 2204, a mesh bounding box is defined based on the planned resection. Step 2204 may include generating a mesh based on the planned resection, e.g., divided into axis-aligned bounding box voxels (e.g., in three dimensions). The mesh may be sized to include the planned resection (e.g., with a surrounding region of interest), e.g., provided as a minimum size suitable for including the planned resection and the surrounding region of interest. In some embodiments, the mesh is oriented based on the planned resection, e.g., as described above with reference to Fig.14 Step 2204 may include providing each voxel with a linked list to which procedural geometry data may be added.

[0098] At step 2206, mesh voxel parameters are defined based on the planned resection. For example, voxel parameters such as voxel size may be determined based on the shape of the planned resection to provide sufficient accuracy, resolution, etc. while improving computational efficiency.

[0099] In step 2208, program geometry parameters are defined based on at least one cutting tool and the pose of at least one cutting tool associated with the planned resection. The data related to the planned resection provided in step 2202 may include information about the tool to be used, which information may be associated with a specific program geometry in step 2208. For example, a particular size of drill cone may be planned to be used in the planned resection, and step 2208 may include defining the program geometry to include a sphere with a drill cone radius suitable for the planned resection. As another example, it may be indicated to use a router cutting tool for the planned resection, where the scenario step 2208 may include defining the program geometry to include a half-capsule shape of a size corresponding to the identified router cutting tool, where the planned orientation of the tool is used to orient the half-capsule shape in the program geometry. Various such examples may appropriately represent different tools that can be used, tools of different sizes, different tool orientations, etc. Therefore, program geometry parameters can be dynamically assigned based on the planned resection to best represent the specific tool selected for the specific planned resection.

[0100] In step 2210, a visualization of the execution of the planned resection is generated using the skeletal model, mesh bounding box, mesh voxel parameters, and procedural geometry. Step 2210 may include ray casting and / or CSG operations, such as Fig.23 2200 is shown and described in detail below with reference thereto. Step 2210 may include providing surgical guidance and navigation by updating a visualization based on tracking of a tool as the planned resection is performed, such that the visualization shows the actual performance of the resection relative to the planned resection and the skeletal model. For example, the position taken by the tool during performance of the resection may be represented in the procedural geometry. Advantageously, the visualization is adapted to the details of the planned resection, including the tool used in performing the planned resection, such that the visualization may accurately represent the planned resection (and the performance of the planned resection based on tracking data) in a manner that smoothly represents the cut performed by the tool without artifacts that might otherwise be left by rasterization methods. Thus, process 2200 may be particularly suitable for providing a visualization of curved tools (e.g., spherical drilling cones) and / or curved resections, such as for guiding surgical procedures on the shoulder or hip.

[0101] Process 2200 can be performed for different planned resections, for example, such that the procedural geometry parameters are updated according to the different planned resections. For example, a first planned resection can be associated with a first procedural geometry parameter, and the first procedural geometry parameter can be used to provide guidance for the first planned resection. The surgical procedure can then also include a second planned resection associated with a second procedural geometry parameter (e.g., due to the use of a second tool, based on a different orientation of the tool relative to the first resection when performing the second resection, etc.), such that the procedural geometry parameters can be updated from the first procedural geometry parameters to the second procedural geometry parameters for providing guidance for the second planned resection. Therefore, the parameters used in the visualization are plan-dependent, that is, the planned resection based on the surgical plan is variable.

[0102] Reference now Fig.23 , a process 2300 for providing visualization using procedural geometry is shown according to some embodiments. The process 2300 can be performed by one or more processors executing instructions stored in a non-transitory computer-readable medium, such as by the system 10, such as through the navigation system 22 or using the navigation system (e.g., by the GPU 28). In some embodiments, the process 2300 can be performed as part of step 2210 of the process 2200.

[0103] At step 2302, stencil mask optimization is performed. Stencil mask optimization may include determining areas for ray casting, such as areas where the skeletal model overlaps with the planned resection. Step 2302 may include, for example, providing a stencil mask (stencil buffer) to adjust the number of rays to be cast based on a solution to an optimization problem that minimizes the number of rays or optimizes some other objective to improve the computational efficiency of process 2300.

[0104] In step 2304, a virtual camera ray is created. The virtual camera ray extends from the viewpoint of the virtual camera viewing the bone model and the planned resection, for example, with reference to Fig.11 The viewpoint of the virtual camera as described above. Step 2304 may include creating enough virtual rays to view the area defined by the stencil mask from step 2302, e.g., each ray is associated with a pixel of a display screen used for visualization. Each ray may have a direction defined by a corresponding pixel coordinate in screen space and is directed to intersect the procedural geometry mesh (e.g., from process 2200).

[0105] At step 2306, ray casting is performed. The ray created in step 2304 is cast and traverses the procedural geometry mesh (e.g., the mesh generated according to process 220). Various methods for ray casting mesh traversal can be implemented. Ray casting can determine the points where the procedural geometry (e.g., its surface) intersects with the ray.

[0106] At step 2308, a union between the intersections of the rays and the procedural geometry is established. Step 2308 may include determining the region associated with the intersection of the rays and the procedural geometry. For example, where the procedural geometry represents the tracked position of a tool during the performance of ablation, the union between the intersections of the rays and the procedural geometry may represent the ablation performed by the tool. The union may then be used to select the colors of the pixels associated with the various rays. In some embodiments, the output of step 88 may then be provided to Figure 8 Box 44 for CSG synthesis or other steps for providing final visualization via a display screen. Thus, procedural geometry (e.g., defined in a plan-dependent manner in process 2300) can be used in a ray casting approach to facilitate visualization of the surgical procedure.

[0107] Reference now Figure 24 to Figure 25 , illustrating teachings related to providing transparency in visualization for surgical planning and navigation according to some embodiments. Fig.24 Process 2400 is shown, and Fig.25 A simplified example illustration is provided to facilitate process 2400. Process 2400 may be performed by one or more processors executing instructions stored in a non-transitory computer-readable medium, such as by system 10, such as by navigation system 22 or using the navigation system (e.g., by GPU 28). Process 2400 may be performed in combination with any of the various other processes and methods described herein, such as when transparency of one or more surfaces or objects represented in the visualization herein is desired (e.g., transparency of a bone model, transparency of an implant, transparency of a cutting tool, transparency of a planned resection, etc.).

[0108] At step 2400, segments are identified along the ray. Fig.25 An example of identifying fragments of object 2500 along ray 2502 is shown. Ray 2502 is cast from virtual camera 2504 and is associated with pixel 2506 at plane 2508. Due to the geometry of object 2500, ray 2502 crosses the boundary of object 2500 four times, where the intersections are identified as fragments. Specifically, as Fig.25 As shown, process 2400 will identify four segments: first segment 2510, second segment 2512, third segment 2514, and fourth segment 2516. Various other geometric shapes will result in other segments being identified, and in most cases the ray will identify segments at least on the front surface of the object and on the back surface of the object, potentially with intermediate segments for hollow objects, objects that intersect other objects, etc.

[0109] At step 2404, the segments are evaluated based on the selected transparency rules. The transparency rules may indicate which segments should be retained and / or discarded to provide the desired transparency visualization. Fig.24 In the example of , an example set of transparency rules may indicate that the second fragment 2514 and the third fragment 2516 should be discarded to avoid cluttering the visualization of the representation with the interior surfaces, such as by providing rules based on the relationship between multiple points (e.g., a rule to discard any point that has points closer and farther from the camera 2504). Step 2404 may also include associating opacity levels (e.g., completely opaque, 60% opaque, 20% opaque, completely transparent, etc.) with different fragments. Depending on the specific visual effect desired to be achieved using process 2400, various rules may be implemented in step 2402. Advantageously, such rules may be applied using the results of step 2402 without requiring prior knowledge or data stored with object 2500 to indicate which surfaces are front surfaces, back surfaces, interior surfaces, etc. In some embodiments, after evaluation by step 2404, the fragment regions are appended to a linked list of an order-independent transparency algorithm.

[0110] At step 2406, the fragments are sorted. Sorting the fragments may include determining the order in which the fragments intersect the rays. Fig.25 In the example of , step 2406 may include determining, for example, based on the determined depths of the fragments along the ray 2502, that the ray 2502 first intersects the first fragment 2510, then the second fragment 2512, then the third fragment 2514, and then the fourth fragment 2516. Step 2406 may determine that the first fragment 2510 is closer to the virtual camera 2504 than the last fragment 2516. Various sorting and / or sorting operations may be performed in step 2406.

[0111] At step 2408, a pixel color is determined based on blending the fragments according to the order. For example, based on the transparency rules evaluated in 2404, a set of related fragments can be provided, each fragment having an associated color (in various embodiments, the color is also based on the transparency level). Based on the order of such fragments (e.g., whether the first fragment is in front of or behind the second fragment as determined in step 2406), the associated colors can be blended (combined, merged, etc.) to provide the color of the pixel associated with the light. By repeating such operations for different rays projected on the object according to the teachings described elsewhere herein, a visualization with transparent features can be provided, thereby enabling a user to see objects inside other objects, internal features of an object, etc. in various embodiments.

[0112] This can provide various visualizations. Fig.25, if the segment 2510 is assigned full opacity, the object 2500 will appear as a solid object without visualization of the other segments. As another option, all segments 2510-2516 can be assigned less than full opacity (e.g., 60% opacity), in which case the visualization can provide the ability to see all surfaces. As another option, the first segment 2510 and the fourth segment 2516 can have partial transparency, while the second segment 2512 and / or the third segment 2514 are shown as opaque, thereby facilitating viewing of the internal structure of the object 2500. As another option, the fourth segment 2516 can be provided as opaque, while the first segment 2510 is provided as transparent (e.g., 60% opacity), thereby facilitating viewing of the back surface of the object 2500. Various such examples are possible, such as can be selected by the user to view features of interest in the visualization contemplated by the present disclosure.

[0113] The construction and arrangement of the system and method as shown in various exemplary embodiments are only illustrative. Although only several embodiments are described in detail in the present disclosure, many modifications are possible (for example, the size, size, structure, shape and ratio, parameter value, installation arrangement, material use, color, orientation, etc. of various elements). For example, the position of the element can be reversed or otherwise changed, and the property or quantity or position of the discrete element can be changed or changed. Therefore, all such modifications are intended to be included in the scope of the present disclosure. The order or sequence of any process or method step can be changed according to alternative embodiments, or the order or sequence is reordered. Without departing from the scope of the present disclosure, other replacements, modifications, changes and omissions can be made in the design, operating conditions and arrangement of the exemplary embodiments.

[0114] The present disclosure contemplates a program product on a method, system, and any machine-readable medium for completing various operations. The embodiments of the present disclosure can be implemented using an existing computer processor, or by a special-purpose computer processor for an appropriate system combined for this or another purpose, or by a hard-wired system. The embodiments within the scope of the present disclosure include a program product, which includes a machine-readable medium for carrying or storing machine-executable instructions or data structures thereon. Such machine-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer or other machine with a processor. For example, such machine-readable media can include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or can be used to carry or store the desired program code in the form of machine-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer or other machine with a processor. The above combination is also included in the scope of machine-readable media. Machine-executable instructions include instructions and data that, for example, enable a general-purpose computer, a special-purpose computer, or a special-purpose processing machine to perform a specific function or a group of functions.

[0115] Although the drawings show a specific order of method steps, the order of steps may be different from the order depicted. Two or more steps may also be performed simultaneously or partially simultaneously. Such variations will depend on the selected software and hardware systems and the designer's choice. All such variations are within the scope of the present disclosure. Similarly, software implementations may be accomplished with standard programming techniques with rule-based logic and other logic to accomplish various connection steps, processing steps, comparison steps, and decision steps.

Claims

1. A method, comprising: providing a bone model and a surgical plan defining planned cuts relative to the bone model; orienting a voxel grid relative to the skeletal model based on the planned cut; obtaining tracking data indicating a position of a cutting tool; as well as Guidance is provided for the execution of the planned cut by generating a visualization based on the voxel grid and the tracking data.

2. The method of claim 1, wherein orienting the voxel grid relative to the skeletal model based on the planned cut comprises orienting a first dimension of the voxel grid orthogonal to the planned cut.

3. The method of claim 2, wherein orienting the voxel grid relative to the skeletal model further comprises orienting a second dimension and a third dimension of the voxel grid parallel to the planned cut.

4. The method of claim 1 , wherein the surgical plan further defines additional planned cuts, the method further comprising: responsive to completion of the planned cuts, reorienting the voxel grid based on the additional planned cuts; as well as Guidance is provided for the performance of the additional planned cuts by generating an additional visualization based on the voxel grid and the tracking data.

5. The method of claim 4, wherein reorienting the voxel grid based on the additional planned cut comprises changing the voxel grid from a first orientation in which a first dimension of the voxel grid is orthogonal to the planned cut to a second orientation in which the first dimension of the voxel grid is orthogonal to the additional planned cut.

6. The method of claim 1 , wherein generating the visualization based on the voxel grid and the tracking data comprises: representing, in the voxel grid and based on the tracking data, an accumulation of the position of the cutting tool as the cutting tool performs the planned cut; as well as determining voxels of the voxel grid that are shared by the accumulation of the skeletal model and the position of the cutting tool; Wherein the visualization comprises a representation of a surface of the skeletal model, the surface being produced by removing virtual bone from the voxels shared by the accumulation of the skeletal model and the position of the cutting tool.

7. The method of claim 6, wherein generating the visualization comprises smoothing the representation of the surface based on the planned cuts by: obtaining adjusted normals by adjusting normals of the representation of the surface of the skeletal model in the voxel grid based on the normals of the planned cut; and The visualization is shaded using the adjusted normals.

8. One or more non-transitory computer-readable media storing program instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: providing a bone model and a surgical plan defining planned cuts relative to the bone model; orienting the skeletal model relative to a graphics coordinate system based on the planned cut, such that the planned cut is orthogonal to a first dimension of the graphics coordinate system, parallel to a second dimension of the graphics coordinate system, and parallel to a third dimension of the graphics coordinate system; A visualization of the execution of the planned cut is generated based on the graphical coordinate system, the skeletal model, and tracking data indicating a position of a cutting tool during execution of the planned cut.

9. The one or more non-transitory computer-readable media of claim 8, wherein the graphics coordinate system defines a voxel grid.

10. One or more non-transitory computer-readable media as described in claim 9, wherein generating the visualization includes determining voxels of the voxel grid that are shared by the accumulation of the skeletal model and the position of the cutting tool; and wherein the visualization comprises a surface resulting from bone removal by the cutting tool, wherein the operation comprises determining the surface based on the voxels shared by the accumulation of the bone model and the position of the cutting tool.

11. One or more non-transitory computer-readable media as described in claim 10, wherein the operations further include smoothing the surface based on the normal of the planned cut by shading the visualization using an adjusted normal, wherein the adjusted normal is a function of the normal of the planned cut and a normal calculated from a representation of a result of the execution of the planned cut in the graphics coordinate system.

12. A surgical system, comprising: Tools that can be manipulated to modify the skeleton; A circuit system, the circuit system being programmed to: defining a visualization mesh based on a planned skeletal modification; as well as Based on the tracking data indicating the position of the tool, a visualization of the progress of the tool in performing the planned bone modification is generated using the visualization grid.

13. The surgical system of claim 12, wherein the circuit system is further programmed to: In response to an indication that the planned skeletal modification is complete, updating the visualization mesh based on a subsequent planned skeletal modification; and A second visualization is generated using the updated visualization mesh, the second visualization showing the progress of the tool as it performs additional planned bone modifications.

14. The surgical system of claim 12, wherein: The visualization includes a representation of a bone surface resulting from the progress of the tool in performing the planned bone modification; and The circuit system is programmed to obtain the representation of the bone surface by determining voxels of the voxel grid that are shared by the model of the bone and the sweep of the position of the tool.

15. A surgical system as described in claim 12, wherein the circuit system is further programmed to smooth the representation of the bone surface based on the planned bone modification.

16. The surgical system of claim 12, wherein the circuit system is programmed to define the visualization mesh by defining mesh voxel parameters based on the planned bone modification.

17. The surgical system of claim 12, wherein the circuitry is further programmed to define a bounding box of the visualization grid based on a planned resection.

18. The surgical system of claim 12, wherein the visualization grid comprises axis-aligned bounding box voxels, each voxel being associated with a linked list providing procedural geometry data.

19. The surgical system of claim 18, wherein the circuit system is programmed to provide the procedural geometry data based on a shape of the tool.

20. The surgical system of claim 18, wherein the circuit system is programmed to provide the procedural geometry data.

21. The surgical system of claim 12, wherein the surgical system is further programmed to generate a visualization with transparency by providing colors to pixels in the following manner: identifying a front surface and a back surface intersected by a ray associated with the pixel; and The front surface is provided with a first opacity and the rear surface is provided with a second opacity.

Images