Automatic prediction of a surgical guide using point groups

Abstract

A method for predicting tool alignment, the method comprising: obtaining, by a computing system, a first point cloud representing one or more bones of a patient; applying, by the computing system, a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points indicating tool alignment; and determining, by the computing system, tool alignment based on the points indicating tool alignment.

Description

Technical Field

[0001]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 350,785, filed on Jun. 9, 2022, the entire content of which is incorporated herein by reference.

Background Art

[0002]

[0002] Orthopedic surgery often involves implanting one or more orthopedic prostheses into a patient. For example, in total shoulder replacement surgery, a surgeon may attach an orthopedic prosthesis to the scapula and humerus of the patient. In ankle replacement surgery, a surgeon may attach an orthopedic prosthesis to the tibia and talus of the patient. When planning orthopedic surgery, it can be important for a surgeon to select an appropriate tool alignment, such as a drilling axis, a cutting plane, a pin insertion axis, etc. Selecting an inappropriate tool alignment can lead to an undesirably restricted range of motion, an increased probability of failure of the orthopedic prosthesis, intraoperative complications, and other adverse health outcomes.

Summary of the Invention

[0003]

[0003] This disclosure describes exemplary techniques for tool alignment and automatic prediction of tool alignment guides for orthopedic surgery. As described in this disclosure, a computing system obtains a first point cloud representing one or more bones of a patient. The computing system may then apply a point cloud neural network to generate a second point cloud based on the first point cloud. In some examples, the second point cloud comprises points indicating a tool alignment. The computing system may determine the tool alignment based on the points indicating the tool alignment. In some examples, the second point cloud comprises points representing a tool alignment guide for aligning a tool during surgery.

[0004]

[0004] In one example, the present disclosure describes a method for predicting tool alignment, the method comprising: obtaining, by a computing system, a first point cloud representing one or more bones of a patient; applying, by the computing system, a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points indicative of tool alignment; and determining, by the computing system, the tool alignment based on the points indicative of tool alignment.

[0005]

[0005] In another example, the present disclosure describes a system comprising: a memory system configured to store a first point cloud representing one or more bones of a patient; and a processing circuit configured to apply a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points indicative of tool alignment, and to determine the tool alignment based on the points indicative of tool alignment.

[0006]

[0006] In another example, the present disclosure describes a method for predicting a tool alignment guide, the method comprising: obtaining, by a computing system, a first point cloud representing one or more bones of a patient; applying, by the computing system, a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points representing a tool alignment guide configured to guide a tool along a tool alignment to a target bone among the one or more bones of the patient.

[0007]

[0007] In another example, the present disclosure describes a system for predicting a tool alignment guide, the system comprising a memory system configured to store a first point cloud representing one or more bones of a patient, and a processing circuit configured to apply a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points configured to guide a tool along a tool alignment to a target bone of one or more bones of the patient.

[0008]

[0008] In other examples, the present disclosure describes a system comprising means for performing the methods of the present disclosure, and a computer-readable storage medium storing instructions that, when executed, cause a computing system to perform the methods of the present disclosure.

[0009]

[0009] Details of various examples of the present disclosure are set forth in the accompanying drawings and the following description. Various features, objects, and advantages will become apparent from this description, the drawings, and the claims.

Brief Description of the Drawings

[0010]

Figure 1

[0010] FIG. 1 is a block diagram illustrating an example system that can be used to implement the techniques of the present disclosure.

Figure 2

[0011] FIG. 2 is a block diagram illustrating example components of a planning system according to one or more techniques of the present disclosure.

Figure 3

[0012] FIG. 3 is a conceptual diagram illustrating an example point cloud neural network (PCNN) according to one or more techniques of the present disclosure.

Figure 4

[0013] FIG. 4 is a flowchart illustrating an example architecture of a T-Net model according to one or more techniques of the present disclosure.

Figure 5

[0014] FIG. 5 is a conceptual diagram illustrating an example three-dimensional (3D) image representing a predicted tool alignment according to one or more techniques of the present disclosure.

Figure 6

[0015] Figure 6 is a conceptual diagram illustrating an example of a patient-specific guide according to one or more techniques of the present disclosure.

Figure 7

[0016] Figure 7 is a flowchart illustrating an example process for predicting tool alignment according to one or more techniques of the present disclosure.

Figure 8

[0017] Figure 8 is a flowchart illustrating an example process for predicting a tool alignment guide according to one or more techniques of the present disclosure. **DETAILED DESCRIPTION**

[0011]

[0018] When planning orthopedic surgery, it can be important for a surgeon to select an appropriate tool alignment, such as a drilling axis, a cutting plane, or a pin insertion axis. Selecting an inappropriate tool alignment can lead to an inappropriate range of motion, an increased probability of failure of an orthopedic prosthesis, intraoperative complications, and other adverse health outcomes. Due to the importance of selecting an appropriate tool alignment, planning systems have been developed to assist surgeons in selecting orthopedic prostheses. For example, in some instances, a planning system applies a set of deterministic rules, such as based on the geometric shape of a patient's bone, to recommend a tool alignment for a patient. However, the accuracy of such planning systems can be insufficient, and a surgeon may not be able to rely on the predictions generated by such an automated planning system. Some of the reasons for the insufficient accuracy and lack of surgeon confidence are that the surgeon may not be convinced that the orthopedic prosthesis recommended by the automated planning system is based on cases similar to the patient the surgeon is planning to treat. A further challenge relates to ensuring the accuracy of a tool alignment guide.

[0012]

[0019] The present disclosure describes techniques that can address one or more problems associated with planning systems for predicting tool alignment. For example, according to one or more techniques of the present disclosure, a computing system may obtain a first point cloud representing one or more bones of a patient. The computing system may apply a point cloud neural network (PCNN) to generate a second point cloud based on the first point cloud. In some examples, the second point cloud comprises points indicating tool alignment. The computing system may determine tool alignment based on the second point cloud. The use of point clouds and PCNNs can result in improved accuracy of tool alignment and tool alignment guides, for example, by training the PCNN based on similar patients and experienced surgeons. In some examples, the second point cloud comprises points representing a tool alignment guide for aligning a tool during surgery.

[0013]

[0020] FIG. 1 is a block diagram illustrating an example system 100 that may be used to implement the techniques of the present disclosure. FIG. 1 illustrates a computing system 102, which is an example of one or more computing devices configured to perform one or more example techniques described in the present disclosure. The computing system 102 may include various types of computing devices, such as a server computer, a personal computer, a smartphone, a laptop computer, and other types of computing devices. In some examples, the computing system 102 includes a plurality of computing devices that communicate with each other. In other examples, the computing system 102 includes only a single computing device. The computing system 102 includes a processing circuit 104, a memory system 106, a display 108, and a communication interface 110. The display 108 is optional, for example, in an example where the computing system 102 is a server computer.

[0014]

[0021] Examples of the processing circuit 104 include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. In general, the processing circuit 104 can be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. A fixed function circuit refers to a circuit that provides a specific function and is preset with the operations that can be performed. A programmable circuit refers to a circuit that can be programmed to perform various tasks and provides a flexible function in the operations that can be performed. For example, a programmable circuit can execute software or firmware that operates the programmable circuit as defined by instructions of the software or firmware. A fixed function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operations performed by the fixed function circuit is generally invariant. In some examples, one or more of these units can be individual circuit blocks (fixed function or programmable), and in some examples, one or more units can be integrated circuits. In some examples, the processing circuit 104 is distributed among a plurality of computing devices in the visualization device 114 and the computing system 102. In some examples, the processing circuit 104 is included within a single computing device of the computing system 102.

[0015]

[0022] The processing circuit 104 can include an arithmetic logic unit (ALU), an elementary function unit (EFU), a digital circuit, an analog circuit, and / or a programmable core formed from a programmable circuit. In an example where the operations of the processing circuit 104 are performed using software executed by a programmable circuit, the memory system 106 can store the object code of the software that the processing circuit 104 receives and executes, or another memory (not shown) within the processing circuit 104 can store such instructions. Examples of software include software designed for surgical planning.

[0016]

[0023] The memory system 106 can be formed by any of various memory devices such as a dynamic random access memory (DRAM) including a synchronous DRAM (SDRAM), a magnetoresistive RAM (MRAM), a resistive RAM (RRAM (registered trademark)), or other types of memory devices. Examples of the display 108 include a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. In some examples, the memory system 106 can include multiple separate memory devices such as multiple disk drives, memory modules, etc., which can be distributed among multiple computing devices or included within the same computing device.

[0017]

[0024] The communication interface 110 enables the computing system 102 to communicate with other devices via the network 112. For example, the computing system 102 can output medical images, images of segmentation masks, and other information for display. The communication interface 110 can include a hardware circuit that enables the computing system 102 to communicate (e.g., wirelessly or using wires) with other computing systems and devices such as the visualization device 114 and the imaging system 116. The network 112 can include various types of communication networks including one or more wide area networks such as the Internet, a local area network, etc. In some examples, the network 112 can include wired and / or wireless communication links.

[0018]

[0025] Visualization device 114 can utilize various visualization techniques to display image content to the surgeon. In some examples, visualization device 114 is a computer monitor or display screen. In some examples, visualization device 114 can be a mixed reality (MR) visualization device, a virtual reality (VR) visualization device, a holographic projector, or other device for presenting extended reality (XR) visualization. For example, in some examples, visualization device 114 can be the Microsoft HOLOLENS (trademark) headset available from Microsoft Corporation of Redmond, Washington, USA, or a similar device such as a similar MR visualization device including a waveguide. The HOLOLENS (trademark) device can be used to present 3D virtual objects through a holographic lens or waveguide while allowing the user to view real-world scenes, i.e., actual objects in a real-world environment, through the holographic lens. In some examples, there can be multiple visualization devices for multiple users.

[0019]

[0026] The visualization device 114 may utilize visualization tools available for generating a three-dimensional model of the bone contour, a segmentation mask, or other data to facilitate preoperative planning using patient image data. These tools may enable a surgeon to design and / or select surgical guides and implant components that closely conform to the patient's anatomical structure. These tools may improve surgical outcomes by customizing the surgical plan for each patient. An example of such a visualization tool is the BLUEPRINT™ system available from Stryker Corp. A surgeon may use the BLUEPRINT™ system to select, design, or modify appropriate implant components, determine how best to position and orient the implant components, and how to shape the surface of the bone that will receive the components, and design, select, or modify tool alignment guides(s) or instruments to execute the surgical plan. The information generated by the BLUEPRINT™ system may be incorporated into a preoperative surgical plan for the patient stored in a database in a suitable location, such as the memory system 106, that can be accessed by the surgeon or other healthcare provider, including during actual pre-operative and intra-operative procedures.

[0020]

[0027] The imaging system 116 may comprise one or more devices configured to generate medical image data. For example, the imaging system 116 may include a device for generating CT images. In some examples, the imaging system 116 may include a device for generating MRI images. Further, in some examples, the imaging system 116 may include one or more computing devices configured to process data from the imaging device to generate medical image data. For example, the medical image data may include 3D images of one or more bones of a patient. In this example, the imaging system 116 may include one or more computing devices configured to generate 3D images based on CT or MRI images.

[0021]

[0028] Computing system 102 may obtain a point cloud representing one or more bones of a patient. The point cloud may be generated based on medical image data generated by imaging system 116. In some examples, imaging system 116 may include one or more computing devices configured to generate the point cloud. Imaging system 116 or computing system 102 may generate the point cloud by identifying the surface of one or more bones in the image and sampling points on the identified surface. Each point in the point cloud may correspond to a set of 3D coordinates of a point on the surface of the patient's bone. In other examples, computing system 102 may include one or more computing devices configured to generate medical image data based on data from the devices in imaging system 116.

[0022]

[0029] The memory system 106 of computing system 102 may store instructions that, when executed by processing circuit 104, cause computing system 102 to perform various activities. For example, in the example of FIG. 1, memory system 106 may store instructions that, when executed by processing circuit 104, cause computing system 102 to perform activities associated with planning system 118. For ease of explanation, rather than discussing that computing system 102 performs an activity when processing circuit 104 executes an instruction, the present disclosure may simply refer to planning system 118 or its components as performing the activity, or may directly describe computing system 102 as performing the activity.

[0023]

[0030] In the example of FIG. 1, the memory system 106 stores a surgical plan 120. The surgical plan 120 can correspond to an individual patient. A surgical plan corresponding to a patient can include data associated with a planned or completed orthopedic surgery for the corresponding patient. A surgical plan corresponding to a patient can include the patient's medical image data 126, point cloud data 128, and the patient's tool alignment data 130. The medical image data 126 can include a computed tomography (CT) image of the patient's bone or a 3D image of the patient's bone based on the CT image. In the present disclosure, the term "bone" can refer to an entire bone or a bone fragment. In some examples, the medical image data 126 can include a magnetic resonance imaging (MRI) image of one or more of the patient's bones or a 3D image based on the MRI image of one or more of the patient's bones. In some examples, the medical image data 126 can include an ultrasonic image of one or more of the patient's bones. The point cloud data 128 can include a point cloud representing the patient's bone. The tool alignment data 130 can include data representing one or more tool alignments for use in the surgery. In the example of FIG. 1, the memory system 106 can also store tool guide data 132 including data representing a tool alignment guide. In some examples, the tool guide data 132 can be included in the surgical plan 120.

[0024]

[0031] The planning system 118 can be configured to assist a surgeon by planning an orthopedic surgery with proper alignment of tools such as a saw, drill, reamer, punch, or other type of tool. According to one or more techniques of the present disclosure, the planning system 118 can apply a point cloud neural network (PCNN) to generate an output point cloud based on an input point cloud. The point cloud data 128 can include the input point cloud and / or the output point cloud. The input point cloud represents one or more bones of the patient. In some examples, the output point cloud includes points indicating tool alignment. The planning system 118 can determine the tool alignment based on the points indicating tool alignment. In some examples, the output point cloud can include points representing a tool alignment guide configured to guide a tool along the tool alignment to a target bone among one or more bones of the patient during the surgery.

[0025]

[0032] In the example of FIG. 1, the system 100 includes a manufacturing system 140. The manufacturing system 140 may manufacture a patient-specific tool alignment guide configured to guide a tool along a tool alignment to a target bone among one or more bones represented in an input point cloud. For example, the manufacturing system 140 may include an additive manufacturing device (e.g., a 3D printer) configured to generate a patient-specific tool alignment guide. In other examples, the manufacturing system 140 may include other types of devices, such as a reductive manufacturing device, a forming device, or other types of devices for generating a patient-specific tool alignment guide.

[0026]

[0033] In an example where the tool alignment corresponds to a cutting plane of a reciprocating saw, the patient-specific tool alignment guide may define a slot for the reciprocating saw. When the patient-specific tool alignment guide is correctly positioned on the patient's bone, the slot is aligned with the determined tool alignment. Accordingly, the surgeon may use the reciprocating saw with the determined tool alignment by inserting the reciprocating saw into the slot of the patient-specific tool alignment guide. In an example where the tool alignment corresponds to a drilling axis or a pin insertion axis, the patient-specific tool alignment guide may define a channel for a drill bit or a pin. When the patient-specific tool alignment guide is correctly positioned on the patient's bone, the channel is aligned with the determined tool alignment. Accordingly, the surgeon may drill a hole or insert a pin by inserting the drill bit or the pin into the channel of the patient-specific tool alignment guide.

[0027]

[0034] FIG. 2 is a block diagram illustrating example components of the planning system 118 according to one or more techniques of the present disclosure. In the example of FIG. 2, the components of the planning system 118 include a PCNN 200, a prediction unit 202, a training unit 204, and a recommendation unit 206. In other examples, the planning system 118 may be implemented using more, fewer, or different components. For example, if the PCNN 200 is already trained, the training unit 204 may be omitted. In some examples, one or more components of the planning system 118 are implemented as software modules. Further, the components of FIG. 2 are provided by way of example, and the planning system 118 may be implemented in other ways.

[0028]

[0035] The prediction unit 202 may apply the PCNN 200 to generate an output point cloud based on the input point cloud. The input point cloud represents one or more bones of a patient. In some examples, the output point cloud includes points indicating tool alignment. In some examples, the output point cloud includes points representing a tool alignment guide for aligning a tool during surgery. The prediction unit 202 may obtain the input point cloud in one of various ways. For example, the prediction unit 202 may generate the input point cloud based on medical image data (e.g., the medical image data 126 of FIG. 1). The medical image data of the patient may include a plurality of input images (e.g., CT images or MRI images, etc.). In this example, each of the input images may have a width dimension and a height dimension, and each of the input images may correspond to different depth dimension layers in a plurality of depth dimension layers. In other words, the plurality of input images may be conceptualized as a stack of 2D images, and the position of an individual 2D image in the stack corresponds to the depth dimension. As part of generating the point cloud, the prediction unit 202 may perform an edge detection algorithm (e.g., Canny edge detection, phase stretch transform (PST), etc.) on the 2D image (or a 3D image based on the 2D image). The prediction unit 202 may select points on the detected edges as points in the input point cloud. In other examples, the prediction unit 202 may obtain the input point cloud from one or more devices external to the computing system 102.

[0029]

[0036] As described above, in some examples, the output point cloud may include points indicating tool alignment. In some such examples, the output point cloud is limited to points indicating tool alignment. In other words, the output point cloud does not include points representing the patient's bone or other tissue. In some examples, the output point cloud includes points indicating tool alignment and points representing other objects such as the patient's bone or tissue. In an example where the tool alignment indicates the cutting plane of a vibrating saw, the points indicating the tool alignment may form a plane that is oriented and positioned in the coordinate space to correspond to the proper alignment of the vibrating saw when cutting bone. In an example where the tool alignment indicates the insertion axis of a tool (e.g., a drill bit, a surgical pin, etc.), the points indicating the tool alignment may form a line that is oriented and positioned in the coordinate space to correspond to the proper alignment of the tool.

[0030]

[0037] In some examples, the output point cloud includes points representing a tool alignment guide for aligning a tool during surgery. In some such examples, the output point cloud is limited to points representing the tool alignment guide. In other words, the output point cloud does not include points representing the patient's bone or other tissue. In some examples, the output point cloud includes points representing the tool alignment guide and points representing other objects such as the patient's bone or tissue. In some examples where the tool alignment guide includes a slot corresponding to the cutting plane of a vibrating saw, the output point cloud does not include points at positions corresponding to the cutting plane. In an example where the tool alignment guide includes a channel for the insertion axis of a tool (e.g., a drill bit, a surgical pin, etc.), the output point cloud does not include points at positions corresponding to the channel.

[0031]

[0038] PCNN200 is implemented using a d-based architecture. An architecture based on a point cloud learning model (e.g., a point cloud learning model) is a neural network-based architecture that receives one or more point clouds as input and generates one or more point clouds as output. Example point cloud learning models include PointNet, PointTransformer, etc. An example architecture based on a point cloud learning model based on PointNet will be described below with respect to Figure 3.

[0032]

[0039] The planning system 118 may include different sets of PCNNs for different surgical types. A set of PCNNs for a surgical type may include one or more PCNNs that correspond to different instances where a surgeon aligns a tool with a patient's bone during a surgery belonging to the surgical type. For example, a set of PCNNs for a total ankle replacement surgery may include a first PCNN that generates an output point cloud including points indicating the alignment of a vibrating saw when removing a part of the patient's distal talus (or points representing a tool alignment guide that defines a slot for aligning the vibrating saw for removing a part of the patient's distal talus). In this example, a second PCNN of the set of PCNNs for a total ankle replacement surgery may generate an output point cloud including points indicating an axis for inserting a guide pin for attaching a cutting guide (or points representing a tool alignment guide that defines a channel for inserting a guide pin for attaching a cutting guide).

[0033]

[0040] The training unit 204 can train the PCNN 200. For example, the training unit 204 can generate a plurality of training data sets. Each of the training data sets can correspond to different past patients among a plurality of past patients. The past patients can include patients for whom a surgical plan was developed. For example, the surgical plan 120 (FIG. 1) can include a surgical plan for a past patient. In some examples, the surgical plan can be limited to those developed by a specialist surgeon, for example, to ensure high-quality training data. In some examples, the past patients can be selected for relevance. The surgical plan can include data indicating the planned tool alignment. For example, the surgical plan can include data indicating that a vibrating saw enters the patient's bone at a specific position and a specific angle. In some examples, the training data set can include a point cloud representing a tool alignment guide used during surgery on a past patient.

[0034]

[0041] The training data set for a past patient can include training input data and predicted output data. The training input data can include a point cloud representing one or more bones of the patient. In an example where the PCNN 200 generates an output point cloud indicating tool alignment, the predicted output data comprises a point cloud including points indicating the tool alignment used during surgery on the past patient. In an example where the PCNN 200 generates an output point cloud representing a tool alignment guide, the predicted output data can comprise a point cloud representing the tool alignment guide used during surgery on the past patient. In some examples, the training unit 204 can generate the training input data based on the medical image data stored in the surgical plan of the past patient. In some examples, the training unit 204 can generate the predicted output data based on the tool alignment in the surgical plan of the past patient. For example, the surgical plan of the past patient can include information indicating the angle of tool alignment and the bone contact position. The training unit 204 can generate points in the training input point cloud along the indicated angle from the bone contact position.

[0035]

[0042] In some examples, the surgical plan includes post-operative medical image data. The post-operative medical image data can be generated after completion of some or all steps of an actual surgery on a past patient. The training unit 204 can analyze the post-operative medical image data to determine tool alignment. The training unit 204 can generate a training input point cloud based on the determined tool alignment. For example, the training unit 204 can determine that a vibrating saw followed a particular cutting plane while removing a portion of bone. In this example, the training unit 204 can determine a training input point cloud based on the determined cutting plane. In some examples, the training unit 204 can receive an indication of user input to indicate a region (e.g., a plane where bone was cut with a saw, a hole drilled with a drill, etc.) in the post-operative medical image data representing a portion of bone corresponding to the tool alignment. The training unit 204 can sample points within the indicated region and then fit a plane or axis to the sampled points. The training unit 204 can extrapolate these planes or axes away from the bone. The training unit 204 can populate the extrapolated region of the plane or axis as the tool alignment to form a training input point cloud. In some examples where the PCNN 200 generates an output point cloud representing a tool alignment guide, the training unit 204 can use the tool alignment determined using the PCNN 200 to generate a point cloud representing the tool alignment guide. For example, the training unit 204 can generate a tool alignment guide that defines a slot or channel corresponding to the determined tool alignment.

[0036]

[0043] The training unit 204 can train the PCNN 200 based on a training data set. Since the training unit 204 generates the training data set based on how an actual surgeon actually planned and / or performed a surgery on a past patient, a surgeon who ultimately uses the tool alignment recommendation or the tool alignment guide recommendation generated by the planning system 118 can have confidence that the recommendation is based on how other actual surgeons selected a tool alignment or a tool alignment guide for actual past patients.

[0037]

[0044] In some examples, as part of training the PCNN200, the training unit 204 may use the input point cloud of the training dataset as the input to the PCNN200 to perform a forward pass on the PCNN200. Next, the training unit 204 may perform a process of comparing the output point cloud obtained as a result generated by the PCNN200 with the corresponding predicted output point cloud. In other words, the training unit 204 may use a loss function to calculate a loss value based on the output point cloud generated by the PCNN200 and the corresponding predicted output point cloud. In some examples, the loss function aims to minimize the difference between the output point cloud generated by the PCNN200 and the corresponding predicted output point cloud. Examples of the loss function may include the Chamfer Distance (CD) and the Earth Mover’s Distance (EMD). CD can be given by the average of the first average and the second average. The first average is the average of the distances between each point in the output point cloud generated by the PCNN200 and its nearest point in the predicted output point cloud. The second average is the average of the distances between each point in the predicted output point cloud and its nearest point in the output point cloud generated by the PCNN200. CD can be defined as follows:

[0038]

Number

[0039] In the above formula, S1 is the output point cloud generated by the PCNN200, S2 is the predicted output point cloud, |..| is the element indicating the number of elements, and ||..|| indicates the absolute value.

[0040]

[0045] Next, the training unit 204 can perform a backpropagation process based on the loss value to adjust the parameters of the PCNN 200 (e.g., the weights of the neurons of the PCNN 200). In some examples, the training unit 204 can determine an average loss value based on loss values calculated from output point clouds generated by performing multiple forward passes through the PCNN 200 using different input point clouds of the training data. In such examples, the training unit 204 can perform a backpropagation process using the average loss value to adjust the parameters of the PCNN 200. The training unit 204 can repeat this process during multiple training epochs.

[0041]

[0046] During use of the PCNN 200 (e.g., after training of the PCNN 200), the prediction unit 202 of the planning system 118 can apply the PCNN 200 to generate an output point cloud for a patient based on an input point cloud representing one or more bones of the patient. In some examples, the recommendation unit 206 can determine a tool alignment based on the output point cloud. For example, in an example where the tool alignment corresponds to a cutting plane, the points of the output point cloud may not be perfectly positioned within the cutting plane. In such an example, the recommendation unit 206 can determine the tool alignment by fitting a plane to the points in the output point cloud indicating the tool alignment. In an example where the tool alignment corresponds to a tool insertion axis (e.g., a drill axis, a pin insertion axis, etc.), the points of the output point cloud may not be perfectly aligned along the tool insertion axis. Thus, in such an example, the recommendation unit 206 can fit a line to the points of the output point cloud representing the tool alignment (e.g., using a regression process).

[0042]

[0047] In some examples, the recommendation unit 206 may determine a tool alignment guide based on the output point cloud. For example, the recommendation unit 206 may perform a 3D reconstruction algorithm, such as a Poisson reconstruction algorithm or Point2Mesh CNN, to generate a 3D mesh based on the output point cloud. The 3D reconstruction algorithm may generate a 3D mesh by deforming at least partially a template input guide mesh to fit the points of the output point cloud. In some examples, before performing the 3D reconstruction algorithm, the recommendation unit 206 may register the output point cloud with a model of one or more bones of the patient (e.g., a model based on the input point cloud, or a model that the input point cloud is based on) using the output point cloud. The recommendation unit 206 may then exclude any points of the output point cloud that are inside the bone model from the output point cloud.

[0043]

[0048] In some examples, such as an example where the output point cloud represents a tool alignment guide, the recommendation unit 206 may determine one or more parameters of the tool alignment guide based on the output point cloud. The parameters of the tool alignment guide may characterize the tool alignment guide such that the tool alignment guide can be selected or manufactured based on the parameters of the tool alignment guide. For example, the recommendation unit 206 may determine the width of the tool alignment guide, the curvature of the arm of the tool alignment guide, etc. The recommendation unit 206 may determine the width of the tool alignment guide based on the distance between the outermost point and the innermost point in the output point cloud. The recommendation unit 206 may determine the curvature of the arm of the tool alignment guide by applying a regression to the points corresponding to the arm.

[0044]

[0049] In some examples, the recommendation unit 206 may output one or more images (e.g., one or more 2D or 3D images) or models indicating tool alignment for display. For example, the recommendation unit 206 may output an image indicating tool alignment with respect to a model of one or more bones of a patient for display. In some such examples, the output point cloud generated by the PCNN 200 and the input point cloud (representing one or more bones of the patient) are in the same coordinate system. Thus, the recommendation unit 206 may position the tool alignment determined by the recommendation unit 206 based on the output point cloud within the coordinate system of the input point cloud. The recommendation unit 206 may then reconstruct the bone model from the points of the input point cloud (e.g., by using the points of the input point cloud as vertices of a polygon that forms the outer skin of the bone model). In some examples, the recommendation unit 206 may output one or more images or models indicating a tool alignment guide for display.

[0045]

[0050] In some examples, the recommendation unit 206 may generate an MR visualization indicating tool alignment based on the output point cloud. In an example where the visualization device 114 (FIG. 1) is an MR visualization device, the visualization device 114 may display the MR visualization. In some examples, the visualization device 114 may display the MR visualization during the surgical planning phase. In such examples, the recommendation unit 206 may generate the MR visualization as a 3D image in space. The recommendation unit 206 may generate the 3D image in the same manner as described above for generating a 3D image. In some examples, the recommendation unit 206 may generate an MR visualization of a tool alignment guide based on the output point cloud.

[0046]

[0051] In some examples, the MR visualization is intraoperative MR visualization. In other words, the visualization device 114 can display MR visualization during surgery. In some examples, the visualization device 114 can perform a registration process that registers the MR visualization with the patient's physical bone. Thus, in such examples, a surgeon wearing the visualization device 114 may be able to view tool alignment with respect to the patient's bone. For example, the surgeon may be able to view a virtual cutting plane that extends away from the patient's bone along a determined tool alignment. In another example, the surgeon may be able to view a virtual drilling axis that extends away from the patient's bone along a determined tool alignment. This can enable the surgeon to use a tool (e.g., a vibrating saw, a drill, etc.) without using a physical patient-specific tool alignment guide. In some examples where the output point cloud represents a tool alignment guide, the recommendation unit 206 can generate an MR visualization representing the tool alignment guide during surgery based on the output point cloud.

[0047]

[0052] In some examples, the computing system 102 can control the operation of the tool based on the alignment of the tool with a determined tool alignment. For example, the visualization device 114 can perform a registration process to associate the positions of the tool, the bone, and the tool alignment with each other. During a surgical stage in which the surgeon is to use the tool at a determined tool alignment, the computing system 102 can determine whether the tool is aligned with the determined tool alignment. If the tool is not aligned with the determined tool alignment, the computing system 102 can communicate with the tool to prevent the tool from operating. For example, the computing system 102 can prevent the tool from operating if the deviation of the tool from the tool alignment is greater than 1 degree or displaced by more than 1 millimeter.

[0048]

[0053] FIG. 3 is a conceptual diagram illustrating an exemplary point cloud learning model 300 according to one or more techniques of the present disclosure. The point cloud learning model 300 can receive an input point cloud. The input point cloud is a set of points. The points in the set of points are not necessarily arranged in any particular order. Thus, the input point cloud can have an unstructured representation.

[0049]

[0054] In the example of FIG. 3, the point cloud learning model 300 includes an encoder network 301 and a decoder network 302. The encoder network 301 receives an array 303 of n points. The points within the array 303 can be the input point cloud of the point cloud learning model 300. In the example of FIG. 3, each of the points within the array 303 has a dimensionality of 3. For example, in a Cartesian coordinate system, each of the points can have an x-coordinate, a y-coordinate, and a z-coordinate.

[0050]

[0055] The encoder network 301 may apply an input transformation 304 to the points in the array 303 to generate an array 305. The encoder network 301 may then use a first shared multi-layer perceptron (MLP) 306 to map each of the n points in the array 305 from 3 dimensions to a larger number of dimensions a (e.g., a = 64 in the example of FIG. 3), thereby generating an array 307 of n×a (e.g., n×64 values). For ease of explanation, the following description of FIG. 3 assumes that a is equal to 64, but in other examples, other values of a may be used. The encoder network 301 may then apply a feature transformation 308 to the values in the array 307 to generate an array 309 of n×64 values. For each of the n points in the array 309, the encoder network 301 uses a second shared MLP 310 to map the n points from a dimensions to b dimensions (e.g., b = 1024 in the example of FIG. 3), thereby generating an array 311 of n×b (e.g., n×1024 values). For ease of explanation, the following description of FIG. 3 assumes that b is equal to 1024, but in other examples, other values of b may be used. The encoder network 301 applies a max pooling layer 312 to generate a global feature vector 313. In the example of FIG. 3, each of the points n in the global feature vector 313 has 1024 dimensions.

[0051]

[0056] Accordingly, as part of applying the PCNN 200, the computing system 102 applies an input transformation (e.g., input transformation 304) to a first array (e.g., array 303) comprising a point cloud to generate a second array (e.g., array 305), the input transformation being implemented using a first T-Net model (e.g., T-Net model 326), applies a first MLP (e.g., MLP 306) to the second array to generate a third array (e.g., array 307), applies a feature transformation (e.g., feature transformation 308) to the third array to generate a fourth array (e.g., array 309), where the input transformation is implemented using a second T-Net model (e.g., T-Net model 330), applies a second MLP (e.g., MLP 310) to the fourth array to generate a fifth array (e.g., array 311), and may apply a max pooling layer (e.g., max pooling layer 312) to the fifth array to generate a global feature vector (e.g., global feature vector 313).

[0052]

[0057] The fully connected network 314 may map the global feature vector 313 to k output classification scores. The value k is an integer indicating the number of classes. Each of the output classification scores corresponds to a different class. The output classification score corresponding to a class may indicate the confidence that the input point cloud as a whole corresponds to the class. The fully connected network 314 includes a neural network having two or more layers of neurons, where each neuron in a layer is connected to each neuron in a subsequent layer. In the example of FIG. 3, the fully connected network 314 includes an input layer having 512 neurons, an intermediate layer having 256 neurons, and an output layer having k neurons. In some examples, the fully connected network 314 may be omitted from the encoder network 301.

[0053]

[0058] In some examples, the input 316 to the decoder network 302 can be formed by concatenating the n 64-dimensional points of the array 309 with the global feature vector 313. In other words, for each of the n points in the array 309, the corresponding 64 dimensions of the point are concatenated with the 1024 features in the global feature vector 313. In some examples, the array 309 is not concatenated with the global feature vector 313.

[0054]

[0059] The decoder network 302 can sample N points in a two-dimensional unit square. Thus, the decoder network 302 can randomly determine N points having an x - coordinate in the range of [0,1] and a y - coordinate in the range of [0,1]. For each of the N points, the decoder network 302 can obtain each input vector by concatenating the respective point with the global feature vector 313. Thus, in an example where the array 309 is not concatenated with the global feature vector 313, each of the input vectors can have 1026 features. For each of the respective input vectors, the decoder network 302 can apply each of K MLPs 318 (where K is an integer greater than or equal to 1) to the respective input vector. Each of the MLPs 318 can correspond to a different patch (e.g., region) of the output point cloud. When the decoder network 302 applies an MLP to an input vector, the MLP can generate 3D points in the patch (e.g., region) corresponding to the MLP. Thus, each of the MLPs 318 can reduce the number of features from 1026 to 3. The three features can correspond to the three coordinates of the points of the output point cloud. For example, for each sampled point n in N, the MLP 318 can reduce the features in the order of 1026, 512, 256, 128, 64, 3. Thus, the decoder network 302 can generate a KxNx3 vector including the output point cloud 320. In some examples, K = 16 and N = 512, resulting in a second point cloud having 8192 3D points. In other examples, other values of K and N can be used. In some examples, as part of training the MLP of the decoder network 302, the decoder network 302 can calculate a chamfer loss of the output point cloud with respect to the ground - truth point cloud. The decoder network 302 can use the chamfer loss in the error back - propagation process to adjust the parameters of the MLP. In this way, the planning system 118 can apply a decoder (e.g., decoder network 302) to generate a premorbid bone model based on the global feature vector.

[0055]

[0060] In some examples, MLP318 may include a series of four fully-connected layers of neurons. For each of the MLP318, the decoder network 302 may pass an input vector of 1026 features to the input layer of the MLP. The fully-connected layers may reduce the number of features in the order of 1026 to 512, 256, and 3.

[0056]

[0061] The input transformation 304 and the feature transformation 308 in the encoder network 301 may provide transformation invariance. In other words, the point cloud learning model 300 may be able to generate the same output point cloud (e.g., the output bone model) regardless of how the input point cloud (e.g., the input bone model) is rotated, scaled, or translated. The fact that the point cloud learning model 300 provides transformation invariance may be advantageous because it can reduce the sensitivity of the point cloud learning model 300 to positioning / scaling-based errors in the pathological bone model. As shown in the example of FIG. 3, the input transformation 304 may be implemented using the T-Net model 326 and the matrix multiplication operation 328. The T-Net model 326 generates a 3×3 transformation matrix based on the array 303. The matrix multiplication operation 328 multiplies the array 303 by the 3×3 transformation matrix. Similarly, the feature transformation 308 may be implemented using the T-Net model 330 and the matrix multiplication operation 332. The T-Net model 330 may generate a 64×64 transformation matrix based on the array 307. The matrix multiplication operation 328 multiplies the array 307 by the 64×64 transformation matrix.

[0057]

[0062] FIG. 4 is a block diagram illustrating an example architecture of a T-Net model 400 according to one or more techniques of the present disclosure. The T-Net model 400 may implement the T-Net model 326 used in the input transformation 304. In the example of FIG. 4, the T-Net model 400 receives an array 402 as input. The array 402 includes n points. Each of the points has a dimensionality of 3. A first shared MLP maps each of the n points in the array 402 from 3 dimensions to 64 dimensions, thereby generating an array 404. A second shared MLP maps each of the n points in the array 404 from 64 dimensions to 128 dimensions, thereby generating an array 406. A third shared MLP maps each of the n points in the array 406 from 128 dimensions to 1024 dimensions, thereby generating an array 408. Next, the T-Net model 400 applies a max pooling operation to the array 408, resulting in an array 810 of 1024 values. A first fully-connected neural network maps the array 410 to an array 812 of 512 values. A second fully-connected neural network maps the array 412 to an array 414 of 256 values. The T-Net model 400 applies a matrix multiplication operation 416 to a matrix 418 of trainable weights. The matrix 418 of trainable weights has a dimension of 256×9. Thus, multiplying the array 414 by the matrix 418 of trainable weights results in an array 820 of size 1×9. Next, the T-Net model 400 may add a trainable bias 422 to the values in the array 420. A reshaping operation 424 may remap the values obtained from adding the trainable bias 422 to a 3×3 transformation matrix. In other examples, the sizes of the matrices and arrays may be different.

[0058]

[0063] The T-Net model 330 (FIG. 3) can be implemented in a similar manner to the T-Net model 400 to perform the feature transformation 308. However, in this example, the trainable weight matrix 418 is 256×4096, and the trainable bias 422 has a bias value of size 1×4096 instead of 9. Thus, the T-Net model for performing the feature transformation 308 can generate a transformation matrix of size 64×64. In other examples, the sizes of the matrices and arrays can be different.

[0059]

[0064] FIG. 5 is a conceptual diagram illustrating an example 3D image 500 representing predicted tool alignments according to one or more techniques of the present disclosure. The 3D image 500 shows the patient's distal tibia 502. Additionally, the 3D image 500 shows three tool alignments 504A, 504B, and 504C (collectively “tool alignments 504”). The tool alignments 504 represent cutting planes for excising sections of the distal tibia 502 as part of a total ankle replacement surgery. The planning system 118 can obtain a point cloud representing the distal tibia 502. Additionally, the prediction unit 202 of the planning system 118 can apply the PCNN 200 to generate one or more output point clouds indicating the tool alignments 504. The recommendation unit 206 of the planning system 118 can determine the tool alignments 504 based on the output point clouds generated by the PCNN 200.

[0060]

[0065] FIG. 6 is a conceptual diagram illustrating an example of a patient-specific guide 600 according to one or more techniques of the present disclosure. In the example of FIG. 6, the patient-specific guide 600 is attached to the patient's distal tibia 502 using guide pins 604A, 604B (collectively “guide pins 604”). The patient-specific guide 600 defines slots 606A, 606B, and 606C (collectively “slots 606”). The slots 606 are aligned with the tool alignment 504 of FIG. 5. Thus, during surgery, the surgeon may use a vibrating saw to cut the distal tibia 502 along the tool alignment 504 by inserting the vibrating saw into the slots 606. In some examples, the patient-specific guide 600 may be manufactured based on a predicted tool alignment. In some examples, the PCNN 200 may generate a point cloud representing the patient-specific guide 600.

[0061]

[0066] FIG. 7 is a flowchart illustrating an example process for predicting tool alignment according to one or more techniques of the present disclosure. In the example of FIG. 7, the computing system 102 may obtain a first point cloud representing one or more bones of a patient (700). In some examples, the computing system 102 may obtain the first point cloud by generating the first point cloud based on one or more medical images. In some examples, the computing system 102 may obtain the first point cloud by receiving the first point cloud from one or more other computing devices or systems.

[0062]

[0067] In addition, the computing system 102 may apply the PCNN 200 to generate a second point cloud based on the first point cloud, the second point cloud comprising points indicating tool alignment (702). For example, the computing system 102 may perform a forward pass through the PCNN 200 using the first input point cloud as an input to the input layer of the PCNN 200. The output layer of the PCNN 200 outputs the second point cloud. In some examples, the second point cloud may include points representing the target bone of the patient (i.e., the bone affected by the use of the tool) and points indicating tool alignment.

[0063]

[0068] Computing system 102 may determine tool alignment (704) based on points indicating tool alignment. For example, computing system 102 may fit a plane or a line to the points indicating tool alignment. The alignment of the tool corresponds to the fitted plane or line. In some examples, to facilitate fitting of the plane or line, computing system 102 may remove outlier points from the second point cloud. An outlier point may be a point whose distance from the nearest neighboring point is greater than a particular amount. The particular amount may be defined by a multiplier of the standard deviation of the distances between the points and their nearest neighboring points.

[0064]

[0069] FIG. 8 is a flowchart illustrating an example process for predicting a tool alignment guide according to one or more techniques of the present disclosure. In the example of FIG. 8, computing system 102 may obtain (800) a first point cloud representing one or more bones of a patient. In some examples, computing system 102 may obtain the first point cloud by generating the first point cloud based on one or more medical images. In some examples, computing system 102 may obtain the first point cloud by receiving the first point cloud from one or more other computing devices or systems.

[0065]

[0070] In addition, computing system 102 may apply PCNN 200 to generate a second point cloud based on the first point cloud, and the second point cloud comprises points configured to guide a tool (e.g., a drill bit, a pin, a vibrating saw, etc.) along tool alignment to a target bone among one or more bones of a patient (802). For example, computing system 102 may perform a forward pass through PCNN 200 using the first input point cloud as an input to the input layer of PCNN 200. The output layer of PCNN 200 outputs the second point cloud. In some examples, the second point cloud may include points representing a target bone (i.e., the bone affected by the use of the tool) of the patient and points representing the tool alignment guide. In some such examples, the spatial arrangement of the points representing the target bone and the points representing the tool alignment guide may indicate proper positioning of the tool alignment guide and the target bone during use of the tool alignment guide. The tool alignment guide may be configured to guide the tool along one or more of a cutting plane, a drilling axis, or a pin insertion axis.

[0066]

[0071] In the example of FIG. 8, computing system 102 may generate a 3D mesh of the tool alignment guide based on the second point cloud (804). For example, computing system 102 may generate the 3D mesh by deforming a template input guide mesh at least partially to fit the points of the second point cloud. After generating the 3D mesh of the tool alignment guide, the 3D mesh may be used as a basis for manufacturing the tool alignment guide using an additive manufacturing process such as 3D printing. In other examples, computing system 102 may not generate a 3D mesh of the tool alignment guide and may use the second point cloud for other purposes.

[0067]

[0072] The following is a non-limiting list of clauses according to one or more techniques of the present disclosure.

[0068]

[0073] Clause 1. A method for predicting tool alignment, the method comprising: obtaining, by a computing system, a first point cloud representing one or more bones of a patient; applying, by the computing system, a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points indicating tool alignment; and determining, by the computing system, tool alignment based on the points indicating tool alignment.

[0069]

[0074] Clause 2. The method according to clause 1, wherein the tool alignment is one of a cutting plane, a drilling axis, or a pin insertion axis.

[0070]

[0075] Clause 3. The method according to clause 1 or 2, further comprising manufacturing a patient-specific tool alignment guide configured to guide a tool along the tool alignment to a target bone among the one or more bones of the patient.

[0071]

[0076] Clause 4. The method according to any one of clauses 1 to 3, further comprising generating, by the computing system, an augmented reality visualization indicating tool alignment based on the second point cloud.

[0072]

[0077] Clause 5. The method according to any one of clauses 1 to 4, further comprising controlling, by the computing system, the operation of the tool based on the alignment of the tool with the tool alignment.

[0073]

[0078] Clause 6. The method according to any one of clauses 1 to 5, wherein the second point cloud comprises points representing a target bone from one or more bones of the patient and points indicating tool alignment.

[0074]

[0079] Clause 7. Determining tool alignment based on the second point cloud comprises fitting a line or a plane to a set of points in the second point cloud.

[0075]

[0080] Clause 8. Applying a point cloud neural network involves applying an input transformation to a first array comprising a first point cloud to generate a second array, where the input transformation is performed using a first T-Net model, applying a first multi-layer perceptron (MLP) to the second array to generate a third array, applying a feature transformation to the third array to generate a fourth array, where the input transformation is performed using a second T-Net model, applying a second MLP to the fourth array to generate a fifth array, applying a max pooling layer to the fifth array to generate a global feature vector, sampling N points in a unit square in two dimensions, concatenating the sampled points with the global feature vector to obtain a combined vector, and applying one or more third MLPs to generate points in a second point cloud, the method according to any one of Clauses 1 to 7.

[0076]

[0081] Clause 9. Further comprising training a PCNN, where training the PCNN involves generating a training dataset based on the surgical plans of past patients and training the PCNN using the training dataset, the method according to any one of Clauses 1 to 8.

[0077]

[0082] Clause 10. A system comprising a memory system configured to store a first point cloud representing one or more bones of a patient, and a processing circuit configured to apply a point cloud neural network to generate a second point cloud based on the first point cloud, where the second point cloud comprises points indicating tool alignment and determine the tool alignment based on the points indicating tool alignment.

[0078]

[0083] Clause 11. The system according to Clause 10, where the tool alignment is one of a cutting plane, a drilling axis, or a pin insertion axis.

[0079]

[0084] System according to clause 10 or 11, further comprising a manufacturing system configured to manufacture a patient-specific tool alignment guide configured to guide a tool along a tool alignment to a target bone among one or more bones of a patient.

[0080]

[0085] System according to any one of clauses 10 to 12, wherein the processing circuit is further configured to generate a mixed reality visualization indicating the tool alignment based on a second point cloud.

[0081]

[0086] System according to any one of clauses 10 to 13, wherein the processing circuit is further configured to control the operation of the tool based on the alignment of the tool with the tool alignment.

[0082]

[0087] System according to any one of clauses 10 to 14, wherein the second point cloud includes points representing a target bone among one or more bones of a patient and points indicating the tool alignment.

[0083]

[0088] System according to any one of clauses 10 to 15, wherein the processing circuit is configured to fit a line or a plane to a set of points in the second point cloud as part of determining the tool alignment based on the second point cloud.

[0084]

[0089] Clause 17. The processing circuit, as part of applying a point cloud neural network, applies an input transformation to a first array comprising a first point cloud to generate a second array, wherein the input transformation is performed using a first T-Net model, applies a first multi-layer perceptron (MLP) to the second array to generate a third array, applies a feature transformation to the third array to generate a fourth array, wherein the input transformation is performed using a second T-Net model, applies a second MLP to the fourth array to generate a fifth array, applies a max pooling layer to the fifth array to generate a global feature vector, samples N points in a unit square in two dimensions, concatenates the sampled points with the global feature vector to obtain a combined vector, and applies one or more third MLPs to generate points in a second point cloud, and is configured to perform the operations of any one of Clauses 10 to 16 of the system described above.

[0085]

[0090] Clause 18. The processing circuit is further configured to train a point cloud neural network. The processing circuit is configured to generate a training dataset based on the surgical plans of past patients and use the training dataset to train a PCNN as part of training the PCNN, and is a system according to any one of Clauses 10 to 17 described above.

[0086]

[0091] A method for predicting a tool alignment guide, the method comprising obtaining, by a computing system, a first point cloud representing one or more bones of a patient, and applying, by the computing system, a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points configured to guide a tool along a tool alignment to a target bone among the one or more bones of the patient.

[0087]

[0092] Clause 20. The method according to clause 19, wherein the tool alignment guide is configured to guide the tool along one of the cutting plane, the drilling axis, or the pin insertion axis.

[0088]

[0093] Clause 21. The method according to clause 19 or 20, further comprising manufacturing a tool alignment guide.

[0089]

[0094] Clause 22. The method according to any one of clauses 19 to 21, further comprising generating, by a computing system, an augmented reality visualization representing a tool alignment guide based on a second point cloud.

[0090]

[0095] Clause 23. The method according to any one of clauses 19 to 22, wherein the second point cloud includes points representing a target bone and points representing a tool alignment guide.

[0091]

[0096] Clause 24. Applying a point cloud neural network to generate the second point cloud comprises applying an input transformation to a first array comprising the first point cloud to generate a second array, wherein the input transformation is performed using a first T-Net model; applying a first multi-layer perceptron (MLP) to the second array to generate a third array; applying a feature transformation to the third array to generate a fourth array, wherein the input transformation is performed using a second T-Net model; applying a second MLP to the fourth array to generate a fifth array; applying a max pooling layer to the fifth array to generate a global feature vector; sampling N points in a two-dimensional unit square; concatenating the sampled points with the global feature vector to obtain a combined vector; and applying one or more third MLPs to generate points in the second point cloud. The method according to any one of clauses 19 to 23.

[0092]

[0097] Clause 25. Further comprising training the PCNN, wherein training the PCNN comprises generating a training dataset based on the surgical plans of past patients and training the PCNN using the training dataset, the method according to any one of Clauses 19 to 24.

[0093]

[0098] Clause 26. A system for predicting a tool alignment guide, the system comprising a memory system configured to store a first point cloud representing one or more bones of a patient, and a processing circuit configured to apply a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points configured to guide a tool along a tool alignment to a target bone among the one or more bones of the patient.

[0094]

[0099] Clause 27. The system according to Clause 26, wherein the tool alignment guide is configured to guide the tool along one of a cutting plane, a drilling axis, or a pin insertion axis.

[0095]

[0100] Clause 28. The system according to Clause 26 or 27, further comprising a manufacturing system configured to manufacture the tool alignment guide.

[0096]

[0101] Clause 29. The system according to any one of Clauses 26 to 28, wherein the processing circuit is further configured to generate a mixed reality visualization representing the tool alignment guide based on the second point cloud.

[0097]

[0102] Clause 30. The system according to any one of Clauses 26 to 29, wherein the second point cloud comprises points representing a target bone from one or more bones of the patient and points representing the tool alignment guide.

[0098]

[0103] Clause 31. The processing circuit, as part of applying a point cloud neural network to generate a second point cloud, applies an input transformation to a first array comprising a first point cloud to generate a second array, wherein the input transformation is performed using a first T-Net model, applies a first multi-layer perceptron (MLP) to the second array to generate a third array, applies a feature transformation to the third array to generate a fourth array, wherein the input transformation is performed using a second T-Net model, applies a second MLP to the fourth array to generate a fifth array, applies a max pooling layer to the fifth array to generate a global feature vector, samples N points in a unit square in two dimensions, concatenates the sampled points with the global feature vector to obtain a combined vector, and applies one or more third MLPs to generate points in the second point cloud, and is configured to perform the system according to any one of Clauses 26 to 30.

[0099]

[0104] Clause 32. The processing circuit is further configured to train a PCNN. As part of training the PCNN, the processing circuit is configured to generate a training dataset based on the surgical plans of past patients and use the training dataset to train the PCNN, and is configured to perform the system according to any one of Clauses 26 to 31.

[0100]

[0105] Clause 33. A system comprising means for performing the method according to any one of Clauses 1 to 9 or 19 to 25.

[0101]

[0106] Clause 34. One or more non-transitory computer-readable storage media storing instructions that, when executed, cause a computing system to perform the method according to any one of Clauses 1 to 9 or Clauses 19 to 25.

[0102]

[0107] Although this technique has been disclosed in relation to a limited number of examples, those skilled in the art having the benefit of this disclosure will appreciate numerous modifications and variations therefrom. For example, any reasonable combination of the described examples is contemplated to be possible. The appended claims are intended to cover such modifications and variations as being within the true spirit and scope of the invention.

[0103]

[0108] It should be recognized that, depending on the example, certain operations or events of any of the techniques described herein may be performed in a different order, may be added, combined, or entirely omitted (e.g., not all described operations or events are necessarily required for the implementation of the technique). Further, in certain examples, operations or events may be performed not sequentially but, for example, concurrently, through multi-threading, interrupt processing, or multiple processors.

[0104]

[0109] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or may include a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. Thus, the computer-readable medium generally can correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. The data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0105]

[0110] By way of example and not limitation, such a computer-readable storage medium can comprise RAM, ROM, EEPROM (registered trademark), CD-ROM or other optical disk storage device, magnetic disk storage device, or other magnetic storage device, flash memory, or any other medium that can be used to store the desired program code in the form of data structures or instructions and that can be accessed by a computer. Also, any connection is not strictly speaking a computer-readable medium. For example, when instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather are directed to non-transient tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc (registered trademark), optical disc, digital versatile disc (DVD), floppy (registered trademark) disc, and Blu-ray (registered trademark) disc, where disks typically reproduce data magnetically and discs reproduce data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0106]

[0111] The operations described in this disclosure may be performed by one or more processors, which may be implemented as fixed-function processing circuits, programmable circuits, or combinations thereof, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Fixed-function circuits refer to circuits that provide a specific function and are preset with the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide a flexible function in the operations that can be performed. For example, a programmable circuit may execute instructions specified by software or firmware that cause the programmable circuit to operate as defined by the instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive or output parameters), but the type of operations performed by fixed-function circuits is generally invariant. Thus, as used herein, the terms "processor" and "processing circuit" may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein.

Claims

1. A method for predicting tool alignment, the method comprising: obtaining, by a computing system, a first point cloud representing one or more bones of a patient; applying, by the computing system, a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points indicating the tool alignment; determining, by the computing system, the tool alignment based on the points indicating the tool alignment; A method comprising the steps of:

2. The method according to claim 1, wherein the tool alignment is one of a cutting plane, a drilling axis, or a pin insertion axis.

3. The method according to claim 1, further comprising manufacturing a patient-specific tool alignment guide configured to guide a tool along the tool alignment to a target bone among the one or more bones of the patient.

4. The method according to claim 1, further comprising generating, by the computing system, an augmented reality visualization indicating the tool alignment based on the second point cloud.

5. The method according to claim 1, further comprising controlling, by the computing system, the operation of the tool based on the alignment of the tool with the tool alignment.

6. The method according to claim 1, wherein the second point cloud comprises points representing a target bone from the one or more bones of the patient and the points indicating the tool alignment.

7. Determining the tool alignment based on the second point cloud comprises fitting a line or a plane to a set of points in the second point cloud.

8. Applying the point cloud neural network comprises: applying an input transformation to a first array comprising the first point cloud to generate a second array, wherein the input transformation is performed using a first T-Net model; applying a first multi-layer perceptron (MLP) to the second array to generate a third array; applying a feature transformation to the third array to generate a fourth array, wherein the input transformation is performed using a second T-Net model; applying a second MLP to the fourth array to generate a fifth array; To generate a global feature vector, applying a max pooling layer to the fifth array; Sampling N points in a unit square in two dimensions; To obtain a combined vector, concatenating the sampled points with the global feature vector; Applying one or more third MLPs to generate points in the second point cloud; The method according to claim 1, comprising the above steps.

9. Further comprising training the point cloud neural network, and training the point cloud neural network includes: Generating a training dataset based on the surgical plans of past patients; Training the point cloud neural network using the training dataset; The method according to claim 1, comprising the above steps.

10. A system comprising: A memory system configured to store a first point cloud representing one or more bones of a patient; Applying a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points indicating tool alignment; Determining the tool alignment based on the points indicating the tool alignment; A processing circuit configured to perform the above operations; The system comprising the above components.

11. The system according to claim 10, wherein the tool alignment is one of a cutting plane, a drilling axis, or a pin insertion axis.

12. The system according to claim 10, further comprising a manufacturing system configured to manufacture a patient-specific tool alignment guide configured to guide a tool along the tool alignment to a target bone among the one or more bones of the patient.

13. The system according to claim 10, wherein the processing circuit is further configured to generate an augmented reality visualization indicating the tool alignment based on the second point cloud.

14. The system according to claim 10, wherein the processing circuit is further configured to control the operation of the tool based on the alignment of the tool with the tool alignment.

15. The system according to claim 10, wherein the second point cloud includes points representing a target bone among the one or more bones of the patient and the points indicating the tool alignment.

16. The system according to claim 10, wherein the processing circuit is configured to fit a line or a plane to a set of points in the second point cloud as part of determining the tool alignment based on the second point cloud.

17. As part of applying the point cloud neural network, the processing circuit applies an input transformation to a first array comprising the first point cloud to generate a second array, wherein the input transformation is performed using a first T-Net model, applies a first multi-layer perceptron (MLP) to the second array to generate a third array, applies a feature transformation to the third array to generate a fourth array, wherein the input transformation is performed using a second T-Net model, applies a second MLP to the fourth array to generate a fifth array, applies a max pooling layer to the fifth array to generate a global feature vector, samples N points in a unit square in two dimensions, concatenates the sampled points with the global feature vector to obtain a combined vector, applies one or more third MLPs to generate points in the second point cloud, The system according to claim 10, which is configured to perform the above.

18. The processing circuit is further configured to train the point cloud neural network. As part of training the point cloud neural network, the processing circuit generates a training dataset based on the surgical plans of past patients, uses the training dataset to train the point cloud neural network, The system according to claim 10, which is configured to perform the above.

19. A method for predicting a tool alignment guide, the method comprising: obtaining, by a computing system, a first point cloud representing one or more bones of a patient; applying, by the computing system, a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points representing a tool alignment guide configured to guide a tool along a tool alignment to a target bone among the one or more bones of the patient. A method comprising

20. The method according to claim 19, wherein the tool alignment guide is configured to guide the tool along one of a cutting plane, a drilling axis, or a pin insertion axis.

21. The method according to claim 19, further comprising manufacturing the tool alignment guide.

22. The method according to claim 19, further comprising generating, by the computing system, an augmented reality visualization representing the tool alignment guide based on the second point cloud.

23. The method according to claim 19, wherein the second point cloud includes points representing the target bone and points representing the tool alignment guide.

24. Applying the point cloud neural network to generate the second point cloud comprises: Applying an input transformation to a first array comprising the first point cloud to generate a second array, wherein the input transformation is performed using a first T-Net model; Applying a first multi-layer perceptron (MLP) to the second array to generate a third array; Applying a feature transformation to the third array to generate a fourth array, wherein the input transformation is performed using a second T-Net model; Applying a second MLP to the fourth array to generate a fifth array; Applying a max pooling layer to the fifth array to generate a global feature vector; Sampling N points in a unit square in two dimensions; Concatenating the sampled points with the global feature vector to obtain a combined vector; Applying one or more third MLPs to generate points in the second point cloud; The method according to claim 19, comprising

25. The method according to claim 19, further comprising training the point cloud neural network, wherein training the point cloud neural network comprises: Generating a training dataset based on surgical plans of past patients; Training the point cloud neural network using the training dataset. The method according to claim 19, comprising

26. A system for predicting a tool alignment guide, the system comprising: A storage system configured to store a first point cloud representing one or more bones of a patient; A processing circuit configured to apply a point cloud neural network to generate a second point cloud based on the first point cloud, wherein the second point cloud comprises points configured to guide a tool along a tool alignment to a target bone among the one or more bones of the patient. A system comprising the same. **Claim 27** The system according to claim 26, wherein the tool alignment guide is configured to guide the tool along one of a cutting plane, a drilling axis, or a pin insertion axis. **Claim 28** The system according to claim 26, further comprising a manufacturing system configured to manufacture the tool alignment guide. **Claim 29** The system according to claim 26, wherein the processing circuit is further configured to generate a mixed reality visualization representing the tool alignment guide based on the second point cloud. **Claim 30** The system according to claim 26, wherein the second point cloud includes points representing the target bone and the points representing the tool alignment guide. **Claim 31** To generate the second point cloud, as part of applying the point cloud neural network, the processing circuit applies an input transformation to a first array comprising the first point cloud to generate a second array, wherein the input transformation is implemented using a first T-Net model; applies a first multi-layer perceptron (MLP) to the second array to generate a third array; applies a feature transformation to the third array to generate a fourth array, wherein the input transformation is implemented using a second T-Net model; applies a second MLP to the fourth array to generate a fifth array; applies a max pooling layer to the fifth array to generate a global feature vector; samples N points in a two-dimensional unit square; concatenates the sampled points with the global feature vector to obtain a combined vector; applies one or more third MLPs to generate points in the second point cloud; and is configured to perform the above operations. The system according to claim 26. **Claim 32** The processing circuit is further configured to train the point cloud neural network, and as part of training the point cloud neural network, the processing circuit generates a training data set based on surgical plans of past patients, and uses the training data set to train the point cloud neural network, and The system according to claim 26, which is configured to perform. **Claim 33** A system comprising means for executing the method according to any one of claims 1 to 9 or 19 to 25. **Claim 34** One or more non-transitory computer-readable storage media storing instructions that, when executed, cause a computing system to execute the method according to any one of claims 1 to 9 or claims 19 to 25.

Images