Automated pre-onset characterization of a patient's anatomical structure using point clouds

Abstract

A method for pre-disease characterization of a patient's anatomical structure includes obtaining a first point cloud representing a diseased state of the patient's bone, generating information indicating at least one of a pathological part of the first point cloud or a non-pathological part of the first point cloud, wherein the pathological part of the first point cloud corresponds to the pathological part of the diseased state of the bone and the non-pathological part of the first point cloud corresponds to the non-pathological part of the diseased state of the bone, generating a second point cloud that includes points corresponding to the non-pathological part but does not include points corresponding to the pathological part, generating a third point cloud representing a pre-disease state of the bone based on the second point cloud, and outputting information indicating the third point cloud representing the pre-disease state of the bone.

Description

Technical Field

[0001]

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 350,732, filed on Jun. 9, 2022, the entire content of which is incorporated herein by reference.

Background Art

[0002]

[0002] Surgical joint repair procedures include the repair and / or replacement of damaged or diseased joints. A surgical joint repair procedure, such as arthroplasty as an example, may include replacing a damaged joint with a prosthesis implanted into the patient's bone. The proper selection or design of a properly sized and formed prosthesis and the proper placement of that prosthesis are important to ensure an optimal surgical outcome. A surgeon may analyze the damaged bone to assist in selecting, designing, and / or placing the prosthesis and, further, the surgical procedures to prepare the bone or tissue to receive or interact with the prosthesis.

Summary of the Invention

[0003]

[0003] This disclosure describes exemplary techniques for performing pre-morbid characterization of patient anatomical structures, such as one or more anatomical objects. Pre-morbid characterization refers to determining a predictive model that predicts the characteristics (e.g., size, shape, location) of a patient anatomical structure as an anatomical structure that existed prior to damage to the patient anatomical structure or disease progression of the anatomical structure. In the examples described in this disclosure, the predictive model can be a point cloud of the pre-morbid anatomical structure of a pathological anatomical structure. A processing circuit can be configured to further process the point cloud of the pre-morbid anatomical structure. For example, the processing circuit can generate a graphic shape model of the pre-morbid anatomical structure that a surgeon can view to assist in planning a surgical procedure (e.g., for repairing or replacing an orthopedic joint).

[0004]

[0004] In one or more examples, the processing circuit may be configured to utilize a point cloud neural network (PCNN) to generate a point cloud of the pre-disease anatomical structure. For example, the processing circuit may apply a first PCNN to a point cloud representation of the diseased state of the anatomical structure to identify pathological (e.g., deformed) and non-pathological (e.g., non-deformed) portions of the anatomical structure. The pathological portion of the point cloud is the portion of the point cloud corresponding to the pathological portion of the diseased state of the anatomical structure, and the non-pathological portion of the point cloud is the portion of the point cloud corresponding to the non-pathological portion of the diseased state of the anatomical structure.

[0005]

[0005] The processing circuit may apply a second PCNN to the non-pathological portion to remove the pathological portion from the point cloud and then generate a point cloud representing the pre-disease state of the anatomical structure (e.g., pre-disease characterization of the patient's anatomical structure). In this way, the exemplary technique utilizes point cloud processing using a neural network that may improve the accuracy of determining the pre-disease characterization of the patient's anatomical structure.

[0006]

[0006] In one or more examples, it may not be necessary to utilize both the first PCNN and the second PCNN. For example, the processing circuit may use techniques that do not necessarily rely on PCNNs, such as surgeon input, comparison with point clouds of other similar patients with non-diseased anatomical structures, a statistical shape model (SSM), etc., to determine the pathological and non-pathological portions of the anatomical structure in the diseased state. In such examples, after removal of the pathological portion, the processing circuit may utilize a PCNN to generate a point cloud representing the pre-disease state of the anatomical structure. As another example, the processing circuit may apply a PCNN to a first point cloud to identify the pathological and non-pathological portions of the anatomical structure in the diseased state. After removal of the pathological portion, the processing circuit may generate a point cloud representing the pre-disease state of the anatomical structure without necessarily using a PCNN, based on surgeon input, comparison with point clouds of other similar patients with non-diseased anatomical structures, SSM, etc.

[0007]

[0007] In the above example, techniques for pre-operative characterization to assist in surgical planning are described. The exemplary techniques described in this disclosure are not so limited. In some examples, the processing circuit may be configured to perform the exemplary techniques described in this disclosure for revision surgery. A patient may have a prosthesis implanted. However, at some point in the future, surgery may be needed to address disease progression, movement of the prosthesis, or because the prosthesis has reached the end of its useful life. Surgery to replace the current prosthesis with another prosthesis is called revision surgery.

[0008]

[0008] In one or more examples, the processing circuit may obtain a first point cloud representing a patient anatomical structure having a prosthesis implanted during a first surgery. The processing circuit may generate a second point cloud representing the patient's anatomical structure prior to the first surgery based on the first point cloud. For example, the processing circuit may generate the second point cloud by applying a point cloud neural network (PCNN) to the first point cloud. In this example, the PCNN may be trained to generate the patient anatomical structure prior to the first surgery (e.g., in the diseased or damaged state to be operated on). The processing circuit may output information indicative of the second point cloud (e.g., representing the patient's anatomical structure prior to the first surgery).

[0009]

[0009] In one example, the present disclosure describes a method for pre-pathological characterization of a patient's anatomical structure, the method comprising: obtaining, by a computing system, a first point cloud representing a pathological condition of a patient's bone; generating, by the computing system, information indicating at least one of a pathological portion of the first point cloud or a non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to the pathological portion of the bone's pathological condition and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to the non-pathological portion of the bone's pathological condition; generating, by the computing system, a second point cloud that includes points corresponding to the non-pathological portion of the bone's pathological condition but does not include points corresponding to the pathological portion of the bone's pathological condition; generating, by the computing system, a third point cloud representing a pre-pathological state of the bone based on the second point cloud; and outputting, by the computing system, information indicating the third point cloud representing the pre-pathological state of the bone.

[0010]

[0010] In one example, the present disclosure describes a method for preoperative characterization of a patient's anatomical structure for a replacement surgery, the method comprising: obtaining, by a computing system, a first point cloud representing a patient's bone having a prosthesis implanted during a first surgery; generating, by the computing system, a second point cloud representing the patient's bone at the time of the first surgery based on the first point cloud; and outputting, by the computing system, information indicating the second point cloud.

[0011]

[0011] In one example, the present disclosure describes a system comprising a memory system configured to store a first point cloud representing a pathological condition of a patient's bone, and a processing circuit. The processing circuit is configured to obtain a first point cloud representing a pathological condition of a patient's bone; generate information indicating at least one of a pathological part of the first point cloud or a non-pathological part of the first point cloud, wherein the pathological part of the first point cloud is a part of the first point cloud corresponding to the pathological part of the bone's pathological condition, and the non-pathological part of the first point cloud is a part of the first point cloud corresponding to the non-pathological part of the bone's pathological condition; the processing circuit is further configured to generate a second point cloud that includes points corresponding to the non-pathological part of the bone's pathological condition but does not include points corresponding to the pathological part of the bone's pathological condition; generate a third point cloud representing the pre-disease state of the bone based on the second point cloud; and output information indicating the third point cloud representing the pre-disease state of the bone.

[0012]

[0012] In one example, the present disclosure describes a system comprising a memory system configured to store a first point cloud representing a patient's bone having a prosthesis implanted during a first surgery, and a processing circuit configured to obtain a first point cloud representing a patient's bone having a prosthesis implanted during a first surgery; generate a second point cloud representing the patient's bone at the time of the first surgery based on the first point cloud; and output information indicating the second point cloud.

[0013]

[0013] In one or more examples, the present disclosure describes a system comprising means for implementing the methods of the present disclosure and a computer-readable storage medium storing instructions that, when executed, cause a computing system to implement the methods of the present disclosure.

[0014]

[0014] Details of various examples of the present disclosure are set forth in the accompanying drawings and the following description. Various features, objects, and advantages will become apparent from this description, the drawings, and the claims.

Brief Description of the Drawings

[0015]

Figure 1A

Figure 1B

[0016] Block diagram showing another exemplary system that can be used to implement the techniques of the present disclosure.

Figure 2

[0017] Block diagram showing exemplary components of a planned system according to one or more techniques of the present disclosure.

Figure 3

[0018] Conceptual diagram showing an exemplary point cloud neural network (PCNN) according to one or more techniques of the present disclosure.

Figure 4

[0019] Flowchart showing an exemplary architecture of a T-Net model according to one or more techniques of the present disclosure.

Figure 5

[0020] Flowchart showing an exemplary process for pre-disease characterization of a patient's anatomical structure according to one or more techniques of the present disclosure.

Figure 6

[0021] Flowchart showing an exemplary process for pre-operative characterization of a patient's anatomical structure for replacement surgery.

DETAILED DESCRIPTION OF THE INVENTION

[0016]

[0022] A patient may have a disease (e.g., illness) that causes damage to the patient's anatomical structure, or the patient may have suffered an injury that causes damage to the patient's anatomical structure. To address the disease or injury, a surgeon may perform a surgical procedure. It can be advantageous for a surgeon to determine, prior to surgery, the characteristics of the pre-injury patient anatomical structure, called pre-morbid characteristics (e.g., size, shape, and / or location). For example, determining the pre-morbid characteristics of the patient anatomical structure can help in the selection, design and / or placement of a prosthesis, and in planning surgical procedures for preparing the surface of a bone damaged to receive or interact with the prosthesis. With prior planning, the surgeon can determine, before the surgery rather than during the surgery, procedures for preparing bone or tissue, tools that will be needed, the size and shape of the tools, the size and shape of one or more prostheses to be implanted, or other characteristics, etc.

[0017]

[0023] For example, with respect to bone as an example of a patient anatomical structure, reconstructing the pre-injury bone (e.g., the pre-morbid characteristics of the bone) can be useful to assist the surgeon in fixing the damaged bone. For example, digital reconstruction of the pre-morbid anatomical structure can help in verifying the possible surgeries required and in verifying the function of adjacent joints. As an example, an overlay of the damaged bone and the reconstructed bone (e.g., a digital representation of the pre-morbid bone) can help in identifying which tools are needed.

[0018]

[0024] As described above, pre-morbid characterization refers to characterizing the patient anatomical structure that existed before the patient incurred a disease or injury. However, since a patient may not consult a physician or surgeon until after incurring a disease or injury, pre-morbid characterization of the anatomical structure is generally not available.

[0019]

[0025] The pre-disease anatomical structure, also called the native anatomical structure, refers to the anatomical structure before the onset of a disease or injury. Even after a disease or injury, there may be parts of the healthy anatomical structure and parts of the non-healthy (e.g., diseased or damaged) anatomical structure. The diseased or damaged part of the anatomical structure is called the pathological anatomical structure, and the healthy part of the anatomical structure is called the non-pathological anatomical structure.

[0020]

[0026] The present disclosure describes exemplary techniques for determining a representation of the pre-disease state of an anatomical structure (e.g., a predictor of the pre-disease anatomical structure) using point cloud processing such as a point cloud neural network (PCNN). The PCNN is implemented using an architecture based on a point cloud learning model. An architecture based on a point cloud learning model (e.g., a point cloud learning model) is a neural network-based architecture that receives one or more point clouds as input and generates one or more point clouds as output. Exemplary point cloud learning models include PointNet, PointTransformer, etc.

[0021]

[0027] In one or more examples described in the present disclosure, the processing circuit may be configured to determine (e.g., obtain) a first point cloud representing the diseased state of a patient's anatomical structure (e.g., a damaged or diseased patient's anatomical structure). For example, the processing circuit may receive one or more images of the patient's anatomical structure in a diseased state and determine the first point cloud based on the received one or more images.

[0022]

[0028] The present disclosure describes a processing circuit for obtaining a first point cloud. As one example of obtaining the first point cloud, the processing circuit may receive one or more images of the patient's anatomical structure in a diseased state and determine the first point cloud based on the received one or more images. As another example, some other circuit may generate the first point cloud, and the processing circuit may receive the generated first point cloud in order to obtain the first point cloud.

[0023]

[0029] The processing circuit can generate information indicating at least one of a pathological part or a non-pathological part of the first point cloud. The pathological part of the first point cloud is a part of the first point cloud corresponding to the pathological part of the pathological state of the anatomical structure, and the non-pathological part of the first point cloud is a part of the first point cloud corresponding to the non-pathological part of the pathological state of the anatomical structure. As an example, the processing circuit can generate information indicating at least one of a pathological part or a non-pathological part of the first point cloud based on applying a point cloud neural network (PCNN) to the first point cloud. In this example, the PCNN can be trained to identify at least one of a pathological part or a non-pathological part.

[0024]

[0030] For example, during training, the PCNN can receive, as input, point clouds representing various pathological states of bones, and can also receive, as input, information identifying the pathological and non-pathological parts of the input point cloud. As an example, a surgeon may provide information about the pathological and non-pathological parts that form the ground truth of the input point cloud representing various pathological states of bones. The processing circuit for training the PCNN can be configured to determine weights and other factors that generate information indicating the pathological and non-pathological parts that match the decisions made by the surgeon when applied to the input point cloud. As a result of training, it can become a trained PCNN that can be applied by the processing circuit to the first point cloud to generate information indicating at least one of a pathological part or a non-pathological part of the first point cloud.

[0025]

[0031] The use of a PCNN to determine pathological and non-pathological parts is not necessary in all cases. In some cases, the processing circuit may use other techniques such as receiving input from a surgeon, comparing with similar patients having non-diseased anatomical structures, or utilizing a static shape model (SSM) to determine pathological and non-pathological parts.

[0026]

[0032] For example, regarding the SSM, the processing circuit can obtain the point cloud of the SSM, and this SSM is a representative model of the pre-disease state of the anatomical structure. The processing circuit can orient the point cloud of the SSM or the point cloud representing the diseased state of the anatomical structure so that the point cloud of the SSM and the point cloud representing the diseased state of the anatomical structure are in the same orientation. The processing circuit can determine the non-pathological points within the point cloud representing the diseased state of the anatomical structure. For example, as described above, the point cloud representing the diseased state of the anatomical structure may have a pathological part and a non-pathological part. The processing circuit can identify one or more points (referred to as non-pathological points) in the non-pathological part. The non-pathological points for identification in the point cloud representing the diseased state of the anatomical structure can be predefined based on the cause of the diseased state (for example, there may be a specific part of the anatomical structure that is known not to be affected by the disease). The processing circuit can deform the point cloud of the SSM until the points within the point cloud of the SSM overlap the identified non-pathological points. The processing circuit can determine the difference between the superimposed SSM and the point cloud representing the diseased state of the anatomical structure. The result of the difference can be the pathological part.

[0027]

[0033] The SSM can be used to determine the pre-disease state of the anatomical structure, but the use of the SSM may not be as accurate as desired. However, using the SSM to identify the pathological and non-pathological parts of the point cloud representing the diseased state of the anatomical structure can be sufficiently accurate.

[0028]

[0034] The processing circuit may be configured to generate a second point group that includes points corresponding to non-pathological parts of the pathological state of the anatomical structure but does not include points corresponding to the pathological parts of the pathological state of the anatomical structure. For example, the processing circuit may be configured to remove a portion of the first point group representing the deformed anatomical structure to generate the second point group. That is, the processing circuit may generate a second point group from which the pathological part has been removed such that the second point group includes points corresponding to non-pathological parts but does not include points corresponding to pathological parts. In such an example, the second point group may be the first point group from which a portion including the deformed anatomical structure (e.g., the pathological part) has been removed so that the non-deformed anatomical structure (e.g., the non-pathological part) remains.

[0029]

[0035] According to one or more examples described in the present disclosure, the processing circuit may be configured to generate a third point group representing a pre-disease state of a pathological anatomical structure (e.g., a pre-disease state of a bone) based on the second point group. There may be various exemplary methods for generating a third point group representing the pre-disease state of the anatomical structure based on the second point group. As one example, the processing circuit may generate the third point group by applying a PCNN to the second point group. For example, the PCNN may be trained to reconstruct the pre-disease anatomical structure from the point group of the non-pathological part of the anatomical structure.

[0030]

[0036] As an example, during training, the PCNN may receive as input a point group representing various non-pathological parts of a bone (e.g., an incomplete point group of a healthy bone), or may also receive as input a point group of a healthy bone. For example, in the point group of a healthy bone, during training, the processing circuit or the user may remove N% of the points from the point group and, in some cases, from different regions of the point group. The remaining part of the bone may be regarded as a non-pathological part. The processing circuit may receive both the point group of the non-pathological part and the point group of the healthy bone.

[0031]

[0037] To train the PCNN, the processing circuit may be configured to determine weights and other factors that generate a point cloud that aligns with a healthy bone point cloud when applied to an input point cloud having non-pathological parts. As a result of the training, a trained PCNN may be obtained where the processing circuit may be applied to the second point cloud to generate a third point cloud representing the pre-pathological state of the anatomical structure.

[0032]

[0038] As another example, the processing circuit may determine a non-pathological estimate of the pathological part of the diseased state of the anatomical structure. The non-pathological estimate may be considered an estimate of what the pathological part of the first point cloud was like before the injury. The processing circuit may combine the non-pathological estimate of the pathological part with the second point cloud to generate the third point cloud. For example, to combine the non-pathological estimate of the pathological part with the second point cloud, the processing circuit may fill the second point cloud with the non-pathological estimate of the pathological part.

[0033]

[0039] For example, the PCNN may be trained to determine a non-pathological estimate of the pathological part of the diseased state of the anatomical structure. For example, for training similar to the above, the PCNN may receive point clouds representing various bones and non-pathological parts of healthy bones. In such an example, the PCNN may be trained to use the points of the non-pathological part to generate an estimate of the non-pathological part of the bone (e.g., how it looked before the pathological part was a disease or trauma). The processing circuit may combine the second point cloud with the non-pathological estimate of the pathological part to generate a third point cloud representing the pre-pathological state of the bone. For example, the PCNN may be trained to fill the pathological part of the bone with the estimate of the non-pathological part of the bone to complete the pre-pathological representation of the bone.

[0034]

[0040] The non-pathological estimation of a pathological part refers to the estimation of a non-pathological anatomical structure (e.g., bone) that fills the removed pathological part in order to obtain the pre-disease state of the anatomical structure. For example, the non-pathological estimation of the pathological part removed from the first point cloud to generate the second point cloud completes the second point cloud so that there is no longer a gap within the second point cloud after the removal of the pathological part. In one or more examples, the non-pathological estimation is called "estimation" because the PCNN can be configured to fill the removed pathological part with a non-pathological anatomical structure (e.g., non-pathological bone) that the PCNN has determined to be representative of the pathological part.

[0035]

[0041] In the above example, the processing circuit may use a PCNN to determine the pathological and / or non-pathological parts of the first point cloud to generate a second point cloud that includes and does not include the pathological part, or use the PCNN on the second point cloud to generate a third point cloud representing the pre-disease state of the anatomical structure (e.g., bone). However, in some examples, the processing circuit may utilize two PCNNs, namely, one for determining the pathological and / or non-pathological parts of the first point cloud to generate a second point cloud that includes and does not include the pathological part, and another for generating a third point cloud representing the pre-disease state of the anatomical structure based on the second point cloud. For example, the processing circuit can generate information indicating at least one of the pathological or non-pathological parts of the first point cloud by applying a first PCNN to the first point cloud, and the first PCNN is trained to identify at least one of the pathological or non-pathological parts. The processing circuit can generate a third point cloud representing the pre-disease state of the anatomical structure by applying a second PCNN to the second point cloud based on the second point cloud.

[0036]

[0042] In the above example, the use of PCNN to generate a third point group representing the pre-disease state of the bone was described. However, the exemplary techniques are not so limited. In some examples, the processing circuit may utilize SSM or some other technique to generate a third point group representing the pre-disease state of the bone. For example, after identifying at least one of the pathological part or the non-pathological part, the processing circuit may utilize PCNN to remove the pathological part. The processing circuit may utilize the non-pathological part to drive the fitting of the SSM represented by the point group. For example, the processing circuit can deform the point group of the SSM (e.g., stretch, shrink, rotate, translate, etc.) until the points in the point group of the SSM overlap the identified non-pathological points. The deformed point group of the SSM that overlaps the identified non-pathological points can become a third point group representing the pre-disease state of the bone.

[0037]

[0043] The processing circuit may output information indicating a point group of the pre-disease state of an anatomical structure (e.g., bone). In some examples, the processing circuit may further process the third point group of the pre-disease anatomical structure to generate other information such as a graphic representation of the pre-disease anatomical structure or dimensions that a surgeon can use for preoperative planning or during surgery. For example, a surgeon may use the graphic representation to plan before surgery which tool to use, where to cut, etc. The graphic representation of the pre-disease anatomical structure may enable the surgeon to determine which prosthesis to use and how to perform the implantation surgery such that the outcome of the surgery, the patient's experience (e.g., regarding motor ability), is approximately the same as before the patient experienced the injury or disease. In some examples, during surgery, to assist the surgeon in ensuring that the prosthesis approximates the pre-disease anatomical structure, the surgeon may wear augmented reality (AR) goggles that present an overlay of the graphic representation of the pre-disease anatomical structure over the diseased anatomical structure.

[0038]

[0044] The above describes exemplary techniques for pre - morbidity characterization of pre - morbidity anatomical structures. However, the exemplary techniques are not so limited. In some examples, the processing circuit may be configured to perform the exemplary techniques for replacement surgery described in this disclosure. A surgeon may perform an initial surgery to implant a prosthesis. Over time, the effectiveness of the prosthesis may decline. For example, as the disease progresses, as the prosthesis may move, or as the useful life of the prosthesis nears its end, the effectiveness of the prosthesis may decline.

[0039]

[0045] In such a case, the surgeon may determine that replacement surgery is appropriate. In replacement surgery, the surgeon removes the current prosthesis and implants another prosthesis that may better fit the patient's current condition. When planning and performing replacement surgery, the surgeon may consider it desirable to determine the size and shape of the anatomical structure at the time the initial surgery was performed. That is, the surgeon may wish to determine what the anatomical structure was that caused the initial surgery. Although not necessary in all examples, in some examples, the surgeon may be interested in characterizing (e.g., size and shape) the anatomical structure at the time of the initial surgery, but may not be so interested in the pre - morbidity shape.

[0040]

[0046] There may be various reasons why a surgeon may consider it desirable to determine the characterization of the anatomical structure at the time of the initial surgery. As one example, after surgery, ligaments and other joint structures may have changed. For example, the ligaments may have become stiffer. Attempting to reconstruct the damaged anatomical structure to its original pre - morbidity state in replacement surgery may have an adverse effect on the changed ligaments and other joints. Therefore, it may be desirable to determine the size and shape of the anatomical structure at the time of the initial surgery and for the surgeon to plan the surgery accordingly.

[0041]

[0047] In one or more examples, with respect to preoperative characterization of a patient's anatomical structure for a replacement surgery (e.g., preoperative characterization prior to the first surgery), the processing circuit may obtain a first point cloud representing the anatomical structure of a patient having a prosthesis implanted during the first surgery. The processing circuit may generate a second point cloud representing the anatomical structure of the patient at the time of the first surgery. For example, the processing circuit may generate the second point cloud by applying the PCNN to the first point cloud. The processing circuit may output information indicative of the second point cloud.

[0042]

[0048] For example, in training, the processing circuit for training the PCNN may receive, as inputs, various bone point clouds having currently implanted prostheses and the same bone point cloud at the time the prosthesis was implanted. The processing circuit may determine weights and other factors that generate a point cloud that matches the same bone point cloud at the time the prosthesis was implanted when applied to the input point clouds of various bones having the prosthesis. As a result, a trained PCNN may be obtained that outputs a second point cloud representing the bone at the time the prosthesis was implanted based on the input first point cloud of the bone having the implant.

[0043]

[0049] Utilizing the PCNN for replacement surgery can be beneficial for various reasons. As one example, at the time of the first surgery, the surgeon may not have requested a representation of the anatomical structure in its diseased state (e.g., the diseased or damaged state that caused the first surgery). In some cases, even if the surgeon requested a representation of the anatomical structure in its diseased state, that representation may be lost. By the exemplary techniques described in this disclosure, the processing circuit may be configured to determine the preoperative characterization of the anatomical structure even when such preoperative characterization information is not available.

[0044]

[0050] FIG. 1A is a block diagram showing an exemplary system 100A that can be used to implement the techniques of the present disclosure. FIG. 1A shows a computing system 102, which is an example of one or more computing devices configured to implement one or more exemplary techniques described in the present disclosure. The computing system 102 can include various types of computing devices such as server computers, personal computers, smartphones, laptop computers, and other types of computing devices. In some examples, the computing system 102 includes a plurality of computing devices that communicate with each other. In other examples, the computing system 102 includes only a single computing device. The computing system 102 includes a processing circuit 104, a memory system 106, a display 108, and a communication interface 110. The display 108 is optional, for example, in cases where the computing system 102 is a server computer.

[0045]

[0051] Examples of the processing circuit 104 include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. In general, the processing circuit 104 can be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. A fixed function circuit refers to a circuit that provides a specific function and is preset to the operations that can be performed. A programmable circuit refers to a circuit that can be programmed to perform various tasks and realizes an adaptable function in the operations that can be performed. For example, a programmable circuit can execute software or firmware that operates the programmable circuit as defined by instructions of the software or firmware. A fixed function circuit can execute software instructions (e.g., for receiving or outputting parameters), but the type of operations performed by the fixed function circuit is generally invariant. In some examples, one or more of the units can be individual circuit blocks (fixed function or programmable), and in some examples, one or more of the units can be integrated circuits. In some examples, the processing circuit 104 is distributed among multiple computing devices of the computing system 102 and the visualization device 114. In some examples, the processing circuit 104 is housed within a single computing device of the computing system 102.

[0046]

[0052] The processing circuit 104 may include a programmable core formed from an arithmetic logic unit (ALU), an elementary function unit (EFU), a digital circuit, an analog circuit, and / or a programmable circuit. In an example where the operation of the processing circuit 104 is implemented using software executed by the programmable circuit, the memory system 106 may store the object code of the software that the processing circuit 104 receives and executes, or another memory (not shown) within the processing circuit 104 may store such instructions. Examples of software include software designed for surgical planning.

[0047]

[0053] The memory system 106 may be formed by any of various memory devices, such as a dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), a magnetoresistive RAM (MRAM), a resistive RAM (RRAM (registered trademark)), or other types of memory devices. Examples of the display 108 include a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices. In some examples, the memory system 106 may include multiple separate memory devices, such as multiple disk drives, memory modules, etc., that may be distributed among multiple computing devices or may be housed within the same computing device.

[0048]

[0054] The communication interface 110 enables the computing system 102 to communicate with other devices via the network 112. For example, the computing system 102 may output medical images, images of segmentation masks, and other information for display. The communication interface 110 may include hardware circuitry that enables the computing system 102 to communicate (e.g., wirelessly or using wires) with other computing systems and devices such as the visualization device 114 and the imaging system 116. The network 112 may include various types of communication networks, including one or more wide area networks such as the Internet, local area networks, etc. In some examples, the network 112 may include wired and / or wireless communication links.

[0049]

[0055] The visualization device 114 may utilize various visualization techniques to display image content to the surgeon. In some examples, the visualization device 114 is a computer monitor or display screen. In some examples, the visualization device 114 may be a mixed reality (MR) visualization device, a virtual reality (VR) visualization device, a holographic projector, or other device for presenting augmented reality (XR) visualization. For example, in some examples, the visualization device 114 may be the Microsoft HOLOLENS (registered trademark) headset commercially available from Microsoft Corporation of Redmond, Washington, USA, or a similar device such as, for example, a similar MR visualization device including a waveguide. The HOLOLENS (registered trademark) device can be used to present 3D virtual objects through a holographic lens or a waveguide while enabling the user to view actual objects in the real-world scene, i.e., in the real-world environment, through the holographic lens. In some examples, there may be multiple visualization devices for multiple users.

[0050]

[0056] The visualization device 114 can utilize visualization tools that are available for using patient image data to generate a three-dimensional model of the bone contour, a segmentation mask, or other data to facilitate preoperative planning. These tools can enable a surgeon to design and / or select surgical guides and implant components that closely match the patient's anatomical structure. These tools can improve surgical outcomes by customizing the surgical plan for each patient. One example of such a visualization tool is the BLUEPRINT® system commercially available from Stryker Corp. A surgeon can use the BLUEPRINT® system to determine how to optimally position and orient an implant component, how to shape the bone surface to receive the component, for selecting, designing, or modifying an appropriate implant component, and for designing, selecting, or modifying a surgical guide tool or instrument for performing the surgical plan. The information generated by the BLUEPRINT® system is compiled as the patient's preoperative surgical plan and stored in a database in a suitable location such as the memory system 106, and this preoperative surgical plan can be accessed by the surgeon or other healthcare provider, including before and during the actual surgery.

[0051]

[0057] The imaging system 116 can include one or more devices configured to generate medical image data. For example, the imaging system 116 can include a device that generates CT images. In some examples, the imaging system 116 can include a device that generates MRI images. Further, in some examples, the imaging system 116 can include one or more computing devices configured to process data from an imaging device to generate medical image data. For example, the medical image data can include 3D images of one or more bones of a patient. In this example, the imaging system 116 can include one or more computing devices configured to generate 3D images based on CT or MRI images.

[0052]

[0058] Computing system 102 can obtain a point cloud representing one or more patient anatomical structures (e.g., bones) of a patient. The point cloud can be generated based on medical image data generated by imaging system 116. In some examples, imaging system 116 can include one or more computing devices configured to generate the point cloud. Imaging system 116 or computing system 102 can generate the point cloud by identifying the surface of one or more bones in the image and sampling points on the identified surface. Each point within the point cloud can correspond to a set of 3D coordinates of a point on the surface of the patient's bone. In other examples, computing system 102 can include one or more computing devices configured to generate medical image data based on data from devices within imaging system 116.

[0053]

[0059] The memory system 106 of computing system 102 can store instructions that, when executed by processing circuit 104, cause computing system 102 to perform various operations. For example, in the example of FIG. 1A, memory system 106 can store instructions that, when executed by processing circuit 104, cause computing system 102 to perform operations related to planning system 118. For ease of explanation, rather than discussing computing system 102 performing operations when processing circuit 104 executes instructions, in this disclosure, there may be a brief reference to planning system 118 or its components when performing those operations, or there may be a direct description of computing system 102 when performing those operations.

[0054]

[0060] In the example of FIG. 1A, the memory system 106 stores a surgical plan 120A. The surgical plan 120A can correspond to an individual patient. A surgical plan corresponding to a patient can include data related to orthopedic surgeries planned or completed for the corresponding patient. A surgical plan corresponding to a patient can include the patient's medical image data 126, a point cloud 128, the point cloud 128, an intermediate point cloud 130, a pre-onset point cloud 132, and in some examples, the patient's tool data (e.g., the type of tools required for the surgery). The medical image data 126 can include computerized tomography (CT) images of the patient's anatomical structures such as bones, or 3D images of the patient's anatomical structures based on the CT images.

[0055]

[0061] The exemplary techniques of the present disclosure are described with respect to a bone, which is an example of a patient anatomical structure. However, the exemplary techniques should not be considered limited to bones. In the present disclosure, the term "bone" may refer to an entire bone or a bone fragment. Examples of bones include the tibia, fibula, scapula and humeral head (e.g., shoulder), femur, patella (e.g., knee), vertebrae, or iliac crest, ilium, ischial spine, coccyx (e.g., hip joint), etc.

[0056]

[0062] In some examples, the medical image data 126 can include magnetic resonance imaging (MRI) images of one or more of the patient's bones, or 3D images based on the MRI images of one or more of the patient's bones. In some examples, the medical image data 126 can include ultrasound images of one or more of the patient's bones. The point cloud 128 can include a point cloud representing the patient's bone. In some examples, the tool alignment data related to the surgical plan 120A can include data representing one or more tool alignments for use in the surgery.

[0057]

[0063] The planning system 118 can be configured to generate a pre-onset characterization of a damaged or diseased patient anatomical structure (e.g., a bone). For example, a patient may be suffering from a disease such as osteoarthritis that has the potential to damage bones due to wear of the joints between the bones. As another example, a patient may have suffered an injury such as a fracture.

[0058]

[0064] Bones affected by a disease or trauma are referred to as diseased bones (e.g., damaged bones). A surgeon performs a surgery to correct the damaged bone, such as implanting a prosthesis or performing other surgeries that can assist the patient in returning to the pre-injury state of the bone. To perform the surgery, the surgeon may prepare one of the surgical plans 120A. One component of the surgical plan can be information indicating the characteristics of the diseased bone before the injury. The surgeon can use such information about the diseased bone before the injury for the surgical plan (e.g., tool selection, prosthesis selection, how to perform the surgery, etc.) or as part of the surgery (e.g., by using an AR headset such as the visualization device 114 to view an overlay of the bone before the injury over the damaged bone).

[0059]

[0065] For example, the point cloud 128 can be a point cloud representing the diseased state of the patient's bone. The point cloud 128 may be referred to as the diseased point cloud 128, or the first point cloud 128, to indicate that the point cloud 128 represents the diseased state of the patient's bone. According to one or more techniques of the present disclosure, the planning system 118 can be configured to generate information indicating at least one of the pathological or non-pathological portions of the first point cloud, as will be described in more detail. Although the point cloud 128 represents the diseased state of the bone, not all of the bone may be damaged. For example, the bone can have non-deformed (e.g., non-pathological) bone portions and deformed (e.g., pathological) bone portions.

[0060]

[0066] In some examples, the point cloud 128 need not include the entire bone. As one example, in an ankle fracture, the distal end of the tibia near the ankle is often prone to injury. In one or more examples, since the planning system 118 assumes that the distal end of the tibia is damaged and that the distal end of the tibia is a pathological part, the distal end of the tibia may not be included as a part for determining the pathological part and the non-pathological part. Thus, in some examples, the point cloud 128 representing the diseased state of the patient's bone may be a point cloud of the diseased tibia, with the distal end of the tibia near the ankle removed from the point cloud of the tibia. In such examples, to generate information indicating at least one of the pathological part or the non-pathological part of the point cloud 128, the planning system 118 may generate information indicating at least one of the pathological part or the non-pathological part of the point cloud 128 of the tibia with the distal end removed.

[0061]

[0067] The planning system 118 may be configured to identify within the point cloud 128 which parts of the bone are pathological and which parts are non-pathological. The pathological part of the first point cloud may be the part of the first point cloud corresponding to the pathological part of the diseased state of the bone, and the non-pathological part of the first point cloud may be the part of the first point cloud corresponding to the non-pathological part of the diseased state of the bone.

[0062]

[0068] That is, if the planning system 118 identifies only the pathological part of the bone, the other parts may be considered non-pathological, and vice versa. The planning system 118 may be capable of identifying both the pathological part and the non-pathological part of the bone.

[0063]

[0069] The planning system may generate an intermediate point group 130, also referred to as a second point group 130, that includes points corresponding to non-pathological portions of the bone's diseased state but does not include points corresponding to pathological portions of the bone's diseased state. As an example, the planning system 118 may remove portions identified as pathological portions of the first point group 128. As a result, an intermediate point group 130 may be obtained. That is, the planning system 118 may generate an intermediate point group 130 (e.g., a second point group) that includes points corresponding to non-pathological portions of the bone's diseased state but does not include points corresponding to pathological portions of the bone's diseased state (e.g., by removing the pathological portions of the first point group 128).

[0064]

[0070] In one or more examples, the planning system 118 may generate a third point group representing a pre-disease state of the bone based on the intermediate point group 130 (e.g., as a second point group). For example, the third point group may represent pre-disease bone prior to a disease or trauma. The third point group representing the pre-disease state of the bone is the pre-disease point group 132.

[0065]

[0071] In one or more techniques of the present disclosure, the planning system 118 may apply a point cloud neural network (PCNN) to generate the intermediate point group 130 and / or the pre-disease point group 132. For example, the point group 128 (e.g., the first point group) may be an input point group. The input point group represents a diseased state of one or more bones of a patient, and the one or more bones include pathological and non-pathological portions. The intermediate point group 130 (e.g., the second point group) may be a first output point group. In some examples, the planning system 118 applies the PCNN to the point group 128 to determine information indicating at least one of the pathological and non-pathological portions, and may generate an intermediate point group 130 that includes points corresponding to non-pathological portions of the bone's diseased state but does not include points corresponding to pathological portions of the bone's diseased state. For example, the planning system 118 may remove the pathological portions of the point group 128 to generate the intermediate point group 130.

[0066]

[0072] As an example, the output from the PCNN applied by the planning system 118 to the point cloud 128 (e.g., the diseased point cloud 128 or the first point cloud 128) can be a label for each point within the point cloud 128. For example, the diseased point cloud 128 can have a pathological portion and a non-pathological portion. The pathological portion of the diseased point cloud 128 can be the portion of the diseased point cloud 128 that corresponds to the pathological portion of the bone's diseased state, and the non-pathological portion of the diseased point cloud 128 can be the portion of the diseased point cloud 128 that corresponds to the non-pathological portion of the bone's diseased state.

[0067]

[0073] The label can classify each point within the point cloud 128 as either a pathological point or a non-pathological point. A pathological point indicates that it is in the pathological portion (e.g., of the point cloud 128), and a non-pathological point indicates that it is in the non-pathological portion (e.g., of the point cloud 128). For example, the planning system 118 can generate labels indicating the pathological and non-pathological portions of the point cloud 128 by applying the PCNN.

[0068]

[0074] Next, the planning system 118 can utilize the labels to determine which points to remove from the point cloud 128. For example, for each point labeled as a pathological point, the planning system 118 can remove that point from the point cloud 128. For each point labeled as a non-pathological point, the planning system 118 can leave that point in the point cloud 128. As a result of removing points from the point cloud 128, an intermediate point cloud 130 is formed. For example, the intermediate point cloud 130 can include points corresponding to the non-pathological portion of the bone's diseased state but not points corresponding to the pathological portion of the bone's diseased state.

[0069]

[0075] In some examples, the planning system 118 can apply the PCNN to the intermediate point cloud 130 to generate a pre-onset point cloud 132. In this example, the pre-onset point cloud 132 can be an output point cloud that includes points indicating the characteristics (e.g., size, shape, etc.) of the diseased bone before the disease or injury. That is, the planning system 118 can generate a pre-onset point cloud 132 (e.g., a third point cloud) representing the pre-onset state of the bone based on the intermediate point cloud 130.

[0070]

[0076] In the example of FIG. 1A, the fabrication system 140 is included in the system 100A. The fabrication system 100A can fabricate a patient-specific tool alignment guide configured to guide a tool along a tool alignment to a target bone of a patient. Including the fabrication system 140 is merely one example and should not be considered limiting. In some examples, the fabrication system 140 can fabricate a tool alignment guide based on a representation of a pre-morbid anatomical structure. For example, the planning system 118 can utilize the pre-morbid point cloud 132 to generate a graphic representation of the pre-morbid bone or to generate information indicating the size and dimensions of the pre-morbid bone. The fabrication system 140 can utilize such information to fabricate a desired tool or tool alignment guide.

[0071]

[0077] The fabrication system 140 can include an additional fabrication device (e.g., a 3D printer) configured to generate a patient-specific tool alignment guide. In an example where the tool alignment corresponds to the cutting plane of a reciprocating saw, the patient-specific tool alignment guide can define a slot for the reciprocating saw. When the patient-specific tool alignment guide is correctly positioned on the patient's bone, the slot is aligned with the determined tool alignment. Thus, the surgeon can use the reciprocating saw with the determined tool alignment by inserting the reciprocating saw into the slot of the patient-specific tool alignment guide. In an example where the tool alignment corresponds to a drill axis or a pin insertion axis, the patient-specific tool alignment guide can define a channel for a drill bit or a pin. When the patient-specific tool alignment guide is correctly positioned on the patient's bone, the channel is aligned with the determined tool alignment. Thus, the surgeon can drill a hole or insert a pin by inserting the drill bit or the pin into the channel of the patient-specific tool alignment guide.

[0072]

[0078] FIG. 1B is a block diagram showing another exemplary system 100B that can be used to implement the techniques of the present disclosure. Various components of FIG. 1B having the same reference numerals as in FIG. 1A may be considered the same or substantially the same, and further description regarding FIG. 1B will not be provided.

[0073]

[0079] The system 100B of FIG. 1B includes a surgical plan 120B. The surgical plan 120B can be a surgical plan for a replacement surgery. As described above, a replacement surgery is a surgery when the current prosthesis is removed and replaced with another prosthesis. For replacement surgery, there can be various reasons, including changes in the disease state, movement of the prosthesis, reaching the practical lifespan of the prosthesis, etc. In some examples, the planning system 118 can be configured to generate a representation of the damaged or diseased bone at the time of the first surgery when the prosthesis was implanted. That is, rather than or in addition to generating a representation of the pre-disease bone, the planning system 118 can be configured to determine a representation of the diseased bone at the time of the first surgery when the prosthesis was implanted.

[0074]

[0080] As shown, the surgical plan 120B includes a point cloud 142. One example of the point cloud 142 can be a point cloud representing the anatomical structure of a patient (e.g., the patient's current anatomical structure) with a prosthesis implanted during the first surgery. The planning system 118 can obtain the point cloud 142 in the same manner as the planning system 118 obtained the point cloud 128.

[0075]

[0081] The planning system 118 can generate a preoperative point cloud 144 representing the anatomical structure of the patient at the time of the first surgery. For example, the planning system 118 can apply a PCNN to the point cloud 142 to generate the preoperative point cloud 144.

[0076]

[0082] In a replacement surgery, there is already a prosthesis implanted. Thus, in some examples, the point cloud 142 may include a representation of the prosthesis. In one or more examples, the planning system 118 may be configured to generate the preoperative point cloud 144 without using a portion within the point cloud 142 that includes the prosthesis.

[0077]

[0083] There may be various ways in which the planning system 118 can determine a portion within the point cloud 142 that includes the prosthesis. As one example, the prosthesis may appear in the medical image data 126 as relatively high-intensity image content. The planning system 118 may remove image content having an intensity higher than a threshold value and generate the point cloud 142 based on the resulting image data. As another example, it may be possible for the planning system 118 to utilize another PCNN that is trained to distinguish between the prosthesis and bone. The planning system 118 may remove a portion identified as the prosthesis by this PCNN in order to generate the point cloud 142.

[0078]

[0084] The replacement surgery can be performed on various bone portions where a prosthesis can be implanted. For example, the prosthesis can be for one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, ilium, etc.

[0079]

[0085] FIG. 2 is a block diagram showing exemplary components of the planning system 118 according to one or more techniques of the present disclosure. In the example of FIG. 2, the components of the planning system 118 include a PCNN 200, a prediction unit 202, a training unit 204, and a reconstruction unit 206. In other examples, the planning system 118 may be implemented using more, fewer, or different components. For example, the training unit 204 may be omitted if the PCNN 200 is already trained. In some examples, one or more of the components of the planning system 118 are implemented as software modules. Further, the components of FIG. 2 are presented as examples, and the planning system 118 may be implemented in other ways.

[0080]

[0086] There may be another example of the PCNN200. For example, as described above, in some examples, the planning system 118 may apply the PCNN to the point cloud 128 to generate information indicating at least one of the pathological or non-pathological portions of the point cloud 128. The first example of the PCNN200 (or simply referred to as the first PCNN200) may be trained to identify at least one of the pathological or non-pathological portions.

[0081]

[0087] In some examples, the planning system 118 may apply the PCNN to the intermediate point cloud 130 to generate a point cloud representing a pre-disease state of the bone (e.g., to generate the pre-disease point cloud 132). The second example of the PCNN200 (or simply referred to as the second PCNN200) may be trained to generate the pre-disease point cloud 132 based on the intermediate point cloud 130.

[0082]

[0088] For example, the second PCNN200 may be configured to determine a non-pathological estimate of the pathological portion of the diseased state of the bone and to combine the non-pathological estimate of the pathological portion with the intermediate point cloud 130 to generate the pre-disease point cloud 132. For example, to combine the non-pathological estimate of the pathological portion with the intermediate point cloud 130, the second PCNN may fill the intermediate point cloud 130 with the non-pathological estimate of the pathological portion.

[0083]

[0089] As described above, the non-pathological estimation of the pathological part refers to the estimation of non-pathological anatomical structures (e.g., bones) that fill in the pathological part removed from the pathological point cloud 128 to generate the intermediate point cloud 130. For example, the non-pathological estimation of the pathological part completes the intermediate point cloud 130 so that there are no longer gaps within the intermediate point cloud 130 due to the removal of the pathological part. In one or more examples, the non-pathological estimation is called "estimation" because the second PCNN 200 is configured to fill the pathological part with what the second PCNN 200 determines to be a representative of the pathological part, but with non-pathological anatomical structures (e.g., non-pathological bones). The estimation of non-pathological bones may be all that is available because the bones may be damaged and there may be no pre-injury bone image data.

[0084]

[0090] The second PCNN that determines the non-pathological estimation of the pathological part and filling the intermediate point cloud 130 are presented as exemplary techniques. In some examples, the second PCNN may be configured to directly determine the pre-onset point cloud 132 from the intermediate point cloud 130.

[0085]

[0091] In a replacement surgery, the planning system 118 may apply a PCNN to the point cloud 142 to generate a pre-operative point cloud 144. A third example of the PCNN 200 (or simply referred to as the third PCNN 200) may be trained to determine the characteristics of the pre-operative anatomical structure, and the input is a point cloud representing the anatomical structure of a patient with a prosthesis implanted during a first surgery.

[0086]

[0092] Exemplary techniques do not require the use of both the first PCNN 200 and the second PCNN 200. In some examples, the planning system 118 may utilize the first PCNN 200 instead of the second PCNN 200. In some examples, the planning system 118 may utilize the second PCNN 200 and not utilize the first PCNN 200. In some examples, the planning system 118 may utilize both the first PCNN 200 and the second PCNN 200. In some examples, the use of a third PCNN is not necessary (e.g., for revision surgery), and the planning system 118 may utilize some other technique to generate the preoperative point cloud 144 (e.g., a point cloud representing the patient's anatomical structure at the time of the first surgery).

[0087]

[0093] The prediction unit 202 may apply the PCNN 200 to generate an output point cloud based on the input point cloud. In the first PCNN 200, the input point cloud (e.g., point cloud 128) represents one or more bones of a patient (e.g., a diseased bone), and the output point cloud (e.g., intermediate point cloud 130) includes the points of the point cloud 128 but the pathological portions are removed. That is, the intermediate point cloud 130 includes points corresponding to the non-pathological portions of the bone's diseased state but does not include points corresponding to the pathological portions of the bone's diseased state. In some examples, the output point cloud of the first PCNN 200 may be regarded as a point cloud 128 having a label for each point indicating whether the point is a pathological point or a non-pathological point. In this case, the planning system 118 removes the pathological points from this output point cloud to generate the intermediate point cloud 130. In some examples, for simplicity of explanation, in some examples there may be a previous output point cloud with labels indicating whether the points in the point cloud 128 are pathological points or non-pathological points. In this case, it is understood that the pathological points are removed to generate the intermediate point cloud 130, and the output point cloud of the first PCNN 200 is described as being the intermediate point cloud 130.

[0088]

[0094] The prediction unit 202 can obtain the point cloud 128 in one of various ways. For example, the prediction unit 202 can generate the point cloud 128 based on the medical image data 126. The medical image data of a patient can include a plurality of input images (e.g., CT images or MRI images, etc.). In this example, each of the input images can have a width dimension and a height dimension, and each of the input images can correspond to a different depth dimension layer among a plurality of depth dimension layers. In other words, the plurality of input images can be conceptualized as a stack of 2D images, and the positions of the individual 2D images within the stack correspond to the depth dimension. As part of generating the point cloud 128, the prediction unit 202 can perform an edge detection algorithm (e.g., Canny edge detection, Phase Stretch Transform (PST), etc.) on the 2D image (or a 3D image based on the 2D image).

[0089]

[0095] The prediction unit 202 can select the points on the detected edges as the points within the point cloud. In other examples, the prediction unit 202 can obtain the point cloud 128 from one or more devices external to the computing system 102. In some examples, for use such as that of the second PCNN 200, the prediction unit 202 can receive an intermediate point cloud 130 from components within the planning system 118.

[0090]

[0096] As shown above, the output point cloud (e.g., the intermediate point cloud 130 for the first PCNN 200) can include the non-pathological points of the point cloud 128 and exclude the pathological points of the point cloud 128. As one example, the planning system 118 can utilize the information indicating the pathological and non-pathological portions generated by the first PCNN 200 to remove the pathological portion in order to generate the intermediate point cloud 130.

[0091]

[0097] The output point cloud (e.g., the pre-onset point cloud 132 for the second PCNN 200) may represent pre-onset bone. The representation of pre-onset bone can be the bone that appeared before the disease or trauma, and can be considered as a combination of the non-pathological part of the bone included in the intermediate point cloud 130 and the non-pathological estimation of the pathological part (e.g., how the pathological part looked before the disease or trauma). For example, the output point cloud from the second PCNN 200 can be the intermediate point cloud 130, and the removed pathological part is filled with the non-pathological estimation of the removed pathological part. However, filling in this way with non-pathological estimation should not be considered limited, and it may be possible to directly generate the pre-onset point cloud 132 from the intermediate point cloud 130.

[0092]

[0098] In the replacement surgery, the prediction unit 202 can obtain a point cloud 142 similar to the above description for the point cloud 128. The output point cloud (e.g., the pre-operative point cloud 144 for the third PCNN 200) may represent pre-operative bone (e.g., the bone of a patient with a prosthesis implanted during the first surgery at the time of the first surgery).

[0093]

[0099] The first PCNN 200, the second PCNN 200, and the third PCNN are implemented using a point cloud learning model-based architecture. The point cloud learning model-based architecture (e.g., the point cloud learning model) is a neural network-based architecture that receives one or more point clouds as inputs and generates one or more point clouds as outputs. Exemplary point cloud learning models include PointNet, PointTransformer, etc. An exemplary point cloud learning model-based architecture based on PointNet will be described below with respect to FIG. 3.

[0094]

[0100] As described above, among some examples, there may be a first PCNN200, a second PCNN200, and a third PCNN200. In some examples, there may be different sets of the first PCNN200 and the second PCNN200 based on, for example, which bone pre-onset characterization is required. Similarly, there may be different sets of the third PCNN200 based on which bone replacement surgery there is.

[0095]

[0101] For example, the set of PCNNs for an ankle total replacement surgery may include a first PCNN for ankle surgery and a second PCNN200 that generate an output point group including points indicating the pre-onset ankle. Similarly, there may be a first PCNN and a second PCNN200 for surgeries of the tibia, fibula, scapula, and humeral head (e.g., shoulder), vertebra, patella (e.g., knee), or iliac crest, ilium, ischial spine, coccyx (e.g., hip joint). For replacement surgery, there may be a set of the third PCNN200 for replacing the prosthesis of the ankle and another set of the third PCNN200 for replacing the prosthesis of the shoulder. Generally, the prosthesis may be for the tibia, fibula, scapula, humeral head, femur, patella, vertebra, or ilium, but this technique is not so limited.

[0096]

[0102] The training unit 204 may train the PCNN200. For example, the training unit 204 may generate a group of training data sets. The first group of the training sets may be for training the first PCNN200. The second group of the training sets may be for training the second PCNN200. The third group of the training sets may be for training the third PCNN200.

[0097]

[0103] Each of the training data sets may correspond to an individual past patient among a plurality of past patients. The past patients may include patients with diseased bones and may also include patients with non-diseased bones.

[0098]

[0104] The training dataset for past patients may include training input data and predicted output data. For the first PCNN200 (e.g., for generating information indicating at least one of the pathological or non-pathological parts in the point cloud 128), the training input data may include a point cloud representing the pathological state of the bone, and the predicted output data may include a label for each point in the point cloud 128 indicating whether the point belongs to the pathological points belonging to the pathological part or the non-pathological points belonging to the non-pathological part.

[0099]

[0105] For the second PCNN200 for generating the pre-onset point cloud 132 (e.g., by combining the non-pathological estimation of the pathological part with the intermediate point cloud 130, etc.), the training input data may include a point cloud representing a subset of points of the non-diseased bone, and the predicted output data may be the non-diseased bone. For the third PCNN200 (e.g., for replacement surgery), the training input data may include the point cloud of the patient determined to undergo replacement surgery, and the predicted output data may be the bone at the time of the first surgery where the prosthesis was implanted.

[0100]

[0106] In some examples, the training unit 204 may generate the training input data based on the medical image data stored in the surgical plan of past patients. In some examples, the training unit 204 may generate the predicted output data for the first PCNN200 based on the information from past patients for those determined to be the pathological part and those determined to be the non-pathological part. For example, a surgeon may indicate the parts to be regarded as pathological and non-pathological within the point cloud of past patients. There may be other methods for training the first PCNN200 that generates information indicating the pathological and non-pathological parts of the first point cloud 128.

[0101]

[0107] In some examples, the training unit 204 for the second PCNN can generate predicted output data based on information from past patients with non-diseased bone. For example, the input can be non-diseased bone that has been removed from past patients or other possible non-patient individuals who volunteer for the study. The predicted output data can be a point cloud of the non-diseased bone of these individuals.

[0102]

[0108] In some examples, the training unit 204 for the third PCNN can generate predicted output data based on information from past patients. For example, the training unit 204 can receive a point cloud of a past patient at the time of the first surgery and use this past patient data as the predicted output data of the patient. In this case, the input is the point cloud of these past patients at the time when these patients were about to undergo a replacement surgery. As another example, the training unit 204 can receive information from the surgeon who determined how the pre-operative bone would look when the image data of the patient undergoing the replacement surgery was given.

[0103]

[0109] The training unit 204 can train the first PCNN 200, the second PCNN 200, and the third PCNN 200 based on each group of the training data set. Since the training unit 204 generates the training data set based on how an actual surgeon actually planned and / or performed the surgery on past patients or based on the point cloud of non-diseased bone, the surgeon who ultimately uses the recommendations generated by the planning system 118 can be confident that the recommendations are based on how other actual surgeons determined the pathological and non-pathological parts, or how actual non-diseased bone looks, or how the actual pre-operative anatomical structure looks.

[0104]

[0110] In some examples, as part of training the first PCNN 200, the second PCNN 200, and the third PCNN, the training unit 204 may use the input point cloud of the training dataset as the input to the PCNN 200 to perform a forward pass on the PCNN 200. Next, the training unit 204 may perform a process of comparing the output point cloud as the result generated by the PCNN 200 with the corresponding predicted output point cloud. In other words, the training unit 204 may use a loss function to calculate a loss value based on the output point cloud generated by the PCNN 200 and the corresponding predicted output point cloud. In some examples, the loss function is aimed at minimizing the difference between the output point cloud generated by the PCNN 200 and the corresponding predicted output point cloud. Examples of the loss function may include the Chamfer Distance (CD) and the Earth Mover's Distance (EMD). CD may be given by the average of the first average and the second average. The first average is the average of the distances between each point in the output point cloud generated by the PCNN 200 and its nearest point in the predicted output point cloud. The second average is the average of the distances between each point in the predicted output point cloud and its nearest point in the output point cloud generated by the PCNN 200. CD may be defined as follows.

[0105]

Number

[0106] In the above formula, S1 is the output point cloud generated by the PCNN 200, S2 is the predicted output point cloud, |..| is the element indicating the number of elements, and ||..|| indicates the absolute value.

[0107]

[0111] For example, in the first PCNN200, S1 may be the intermediate point cloud 130, or may be a point cloud with labeled points indicating whether the points are pathological or non-pathological. S2 may be a predicted point cloud determined by a surgeon or other method. In the second PCNN200, S1 may be the pre-onset point cloud 132. S2 may be a predicted point cloud of the pre-onset bone. In the third PCNN200, S1 may be the pre-operative point cloud 144. S2 may be a predicted point cloud regarding the representation of the bone at the time of the first surgery.

[0108]

[0112] Next, the training unit 204 may perform backpropagation processing based on the loss value to adjust the parameters of the PCNN200 (e.g., the weights of the neurons in the PCNN200). In some examples, the training unit 204 may determine an average loss value based on loss values calculated from output point clouds generated by performing a plurality of forward passes through the PCNN200 using different input point clouds of the training data. In such examples, the training unit 204 may perform backpropagation processing using the average loss value to adjust the parameters of the PCNN200. The training unit 204 may repeat this process during a plurality of training epochs.

[0109]

[0113] During the use of the PCNN200 (e.g., after training of the PCNN200), the prediction unit 202 of the planning system 118 may apply the PCNN200 to generate an output point cloud of the patient based on an input point cloud representing one or more bones of the patient. In some examples, the reconstruction unit 206 may be configured to generate one or more images of the pre-onset bone (e.g., a reconstructed version of the diseased bone) using the point cloud 132, or to generate information such as the size and dimensions of the pre-onset bone. In some examples, the reconstruction unit 206 may be configured to generate one or more images of the bone at the time of the first surgery using the pre-operative point cloud 144.

[0110]

[0114] In some examples, the reconstruction unit 206 may output one or more images (e.g., one or more 2D or 3D images) or a model showing the pre - onset bone for display. As an example, the reconstruction unit 206 may use the points within the pre - onset point cloud 132 or the pre - operative point cloud 144 as vertices of a polygon that forms the outer skin of the pre - onset bone. The reconstruction unit 206 may output for display an image showing the pre - onset bone relative to a model of one or more diseased bones of the patient, or an image showing the bone at the time of the first surgery when the prosthesis was implanted. In some such examples, the output point cloud (e.g., the pre - onset point cloud 132 or the pre - operative point cloud 144) generated by the PCNN 200 and the input point cloud (e.g., the point cloud 128 or the point cloud 142) are in the same coordinate system.

[0111]

[0115] In some examples, the reconstruction unit 206 may generate an MR visualization showing the pre - onset bone based on the pre - onset point cloud 132. In a replacement surgery, the reconstruction unit 206 may generate an MR visualization showing the bone at the time of the first surgery based on the pre - operative point cloud 144. In an example where the visualization device 114 (FIGS. 1A and 1B) is an MR visualization device, the visualization device 114 may display the MR visualization. In some examples, the visualization device 114 may display the MR visualization during the surgical planning phase. In such examples, the reconstruction unit 206 may generate the MR visualization in space as a 3D image. The reconstruction unit 206 may generate the 3D image in the same manner as described above to generate the 3D image.

[0112]

[0116] In some examples, the MR visualization is an intra - operative MR visualization. In other words, the visualization device 114 may display the MR visualization during the surgery. In some examples, the visualization device 114 may perform an overlay process of overlaying the MR visualization on the patient's physical bone. Thus, in such examples, a surgeon wearing the visualization device 114 can view the pre - onset bone in comparison to the patient's diseased bone, or view the bone at the time of the first surgery in comparison to the patient's current bone for a replacement surgery.

[0113]

[0117] Figure 3 is a conceptual diagram showing an exemplary point cloud learning model 300 according to one or more techniques of the present disclosure. The point cloud learning model 300 can receive an input point cloud. An input point cloud is a collection of points. The points within the collection of points are not necessarily arranged in any particular order. Thus, the input point cloud can have an unstructured representation.

[0114]

[0118] In the example of Figure 3, the point cloud learning model 300 includes an encoder network 301 and a decoder network 302. The encoder network 301 receives an array 303 of n points. The points of the array 303 can be the input point cloud of the point cloud learning model 300. In the example of Figure 3, each of the points of the array 303 has a dimensionality of 3. For example, in a Cartesian coordinate system, each of the points can have an x coordinate, a y coordinate, and a z coordinate.

[0115]

[0119] The encoder network 301 may apply an input transformation 304 to the points of the array 303 to generate an array 305. Next, the encoder network 301 may use a first shared multi-layer perceptron (MLP) 306 to map each of the n points of the array 305 from 3 dimensions to a larger number of dimensions a (for example, a = 64 in the example of FIG. 3), thereby generating an array 307 of n×a (for example, n×64 values). For ease of explanation, in the following description of FIG. 3, it is assumed that a is equal to 64, but in other examples, other values of a may be used. Next, the encoder network 301 may apply a feature transformation 308 to the values of the array 307 to generate an array 309 of n×64 values. For each of the n points of the array 309, the encoder network 301 may use a second shared MLP 310 to map the n points from a dimensions to b dimensions (for example, b = 1024 in the example of FIG. 3), thereby generating an array 311 of n×b (for example, n×1024 values). For ease of explanation, in the following description of FIG. 3, it is assumed that b is equal to 1024, but in other examples, other values of b may be used. The encoder network 301 applies a max pooling layer 312 to generate a global feature vector 313. In the example of FIG. 3, each of the n points of the global feature vector 313 has 1024 dimensions.

[0116]

[0120] Thus, as part of applying the PCNN 200, the computing system 102 may apply an input transformation (e.g., input transformation 304) to a first array (e.g., array 303) comprising a point cloud to generate a second array (e.g., array 305), where the input transformation is implemented using a first T-Net model (e.g., T-Net model 326), may apply a first MLP (e.g., MLP 306) to the second array to generate a third array (e.g., array 307), may apply a feature transformation (e.g., feature transformation 308) to the third array to generate a fourth array (e.g., array 309), where the input transformation is implemented using a second T-Net model (e.g., T-Net model 330), may apply a second MLP (e.g., MLP 310) to the fourth array to generate a fifth array (e.g., array 311), and may apply a max pooling layer (e.g., max pooling layer 312) to the fifth array to generate a global feature vector (e.g., global feature vector 313).

[0117]

[0121] The fully connected network 314 may map the global feature vector 313 to k output classification scores. The value k is an integer indicating the number of classes. Each of the output classification scores corresponds to a separate class. The output classification score corresponding to a class may indicate the confidence that the input point cloud as a whole corresponds to that class. The fully connected network 314 includes a neural network having two or more neuron layers in which each neuron in one layer is connected to each neuron in a subsequent layer. In the example of FIG. 3, the fully connected network 314 includes an input layer having 512 neurons, an intermediate layer having 256 neurons, and an output layer having k neurons. In some examples, the fully connected network 314 may be omitted from the encoder network 301.

[0118]

[0122] In some examples, the input 316 to the decoder network 302 can be formed by concatenating the n 64-dimensional points of the array 309 with the global feature vector 313. In other words, for each of the n points of the array 309, the corresponding 64 dimensions of that point are concatenated with the 1024 features of the global feature vector 313. In some examples, the array 309 is not concatenated with the global feature vector 313.

[0119]

[0123] The decoder network 302 can sample N points within a two-dimensional unit square. Thus, the decoder network 302 can randomly determine N points having an x - coordinate in the range of [0, 1] and a y - coordinate in the range of [0, 1]. For each of the N points, the decoder network 302 can obtain each input vector by concatenating each point with the global feature vector 313. Thus, in an example where the array 309 is not concatenated with the global feature vector 313, each of the input vectors can have 1026 features. For each of the respective input vectors, the decoder network 302 can apply each of the K MLP318 (where K is an integer greater than or equal to 1) to each input vector. Each of the MLP318 can correspond to a separate patch (e.g., region) of the output point cloud. When the decoder 302 applies the MLP to the input vector, the MLP can generate 3D points in the patch (e.g., region) corresponding to the MLP. Thus, each of the MLP318 can reduce the number of features from 1026 to 3. The three features can correspond to the three coordinates of one point of the output point cloud. For example, for each sampled point n out of N, the MLP318 can reduce the features from 1026 to 512, to 256, to 128, to 64, to 3. Thus, the decoder network 302 can generate a K×N×3 vector including the output point cloud 320. In some examples, K = 16 and N = 512, and as a result, a second point cloud including 8192 3D points is obtained. In other examples, other values of K and N can be used. In some examples, as part of training the MLP of the decoder network 302, the decoder network 302 can calculate the chamfer loss of the output point cloud with respect to the ground truth point cloud. The decoder network 302 can use the chamfer loss in the backpropagation process to adjust the parameters of the MLP. In this way, the planning system 118 can apply a decoder (e.g., the decoder network 302) to generate the intermediate point cloud 130 and the pre - onset point cloud 132 for the example of FIG. 1A, or the pre - operative point cloud 144 for the example of FIG. 1B.

[0120]

[0124] In some examples, MLP 318 may include a series of four fully-connected neuron layers. For each of the MLP 318, the decoder network 302 may pass an input vector of 1026 features to the input layer of the MLP. The fully-connected layers may reduce the number of features from 1026 to 512, to 256, and then to 3.

[0121]

[0125] The input transformation 304 and the feature transformation 308 of the encoder network 301 may provide transformation invariance. In other words, the point cloud learning model 300 may be able to generate the output point cloud (e.g., the intermediate point cloud 130, the pre-onset point cloud 132, and / or the pre-operative point cloud 144) in the same way regardless of how the input point cloud (e.g., the input bone model) is rotated, scaled, or translated. The fact that the point cloud learning model 300 provides transformation invariance may be advantageous because it may reduce the susceptibility of the generator ML model to errors based on the placement / scaling in the diseased bone model. As shown in the example of FIG. 3, the input transformation 304 may be implemented using the T-Net model 326 and the matrix multiplication operation 328. The T-Net model 326 generates a 3×3 transformation matrix based on the array 303. The matrix multiplication operation 328 multiplies the array 303 by the 3×3 transformation matrix. Similarly, the feature transformation 308 may be implemented using the T-Net model 330 and the matrix multiplication operation 332. The T-Net model 330 may generate a 64×64 transformation matrix based on the array 307. The matrix multiplication operation 328 multiplies the array 307 by the 64×64 transformation matrix.

[0122]

[0126] FIG. 4 is a block diagram showing an exemplary architecture of a T-Net model 400 according to one or more techniques of the present disclosure. The T-Net model 400 may implement the T-Net model 326 used in the input transformation 304. In the example of FIG. 4, the T-Net model 400 receives an array 402 as input. The array 402 contains n points. Each of these points has a dimensionality of 3. The first shared MLP maps each of the n points of the array 402 from 3 dimensions to 64 dimensions, thereby generating an array 404. The second shared MLP maps each of the n points of the array 404 from 64 dimensions to 128 dimensions, thereby generating an array 406. The third shared MLP maps each of the n points of the array 406 from 128 dimensions to 1024 dimensions, thereby generating an array 408. The T-Net model 400 then applies a max pooling operation to the array 408, resulting in an array 810 of 1024 values. The first fully connected neural network maps the array 410 to an array 812 of 512 values. The second fully connected neural network maps the array 412 to an array 414 of 256 values. The T-Net model 400 applies a matrix multiplication operation 416 to a matrix 418 of trainable weights. The matrix 418 of trainable weights has a dimension of 256×9. Thus, multiplying the array 414 by the matrix 418 of trainable weights results in an array 820 of size 1×9. The T-Net model 400 can then add a trainable bias 422 to the values of the array 420. A reconstruction operation 424 may remap the values obtained by adding the trainable bias 422 to a 3×3 transformation matrix. In other examples, the sizes of the matrices and arrays may be different.

[0123]

[0127] The T-Net model 330 (FIG. 3) may be implemented in the same manner as the T-Net model 400 to perform the feature transformation 308. However, in this example, the matrix 418 of trainable weights is 256×4096, and the trainable bias 422 has a bias value of size 1×4096 instead of 9. Thus, the T-Net model for performing the feature transformation 308 may generate a transformation matrix of size 64×64. In other examples, the sizes of the matrices and arrays may be different.

[0124]

[0128] Figure 5 is a flowchart showing an exemplary process for pre - disease characterization of a patient's anatomical structure by one or more techniques of the present disclosure. Computing system 102 (e.g., processing circuit 104 implementing planning system 118) may obtain (500) a first point cloud representing a diseased state of a patient's bone. For example, the first point cloud may be point cloud 128. As described above, computing system 102 may utilize medical image data 126 to generate point cloud 128. Point cloud 128 may include a diseased anatomical structure such as one or more bones. Examples of bones include at least one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, iliac crest, ilium, ischial spine, or coccyx.

[0125]

[0129] Computing system 102 may generate (502) information indicating at least one of a pathological portion of the first point cloud or a non - pathological portion of the first point cloud. The pathological portion of the first point cloud (e.g., point cloud 128) may be the portion of the first point cloud corresponding to the pathological portion of the diseased - state bone, and the non - pathological portion of the first point cloud may be the portion of the first point cloud corresponding to the non - pathological portion of the diseased - state bone. For example, computing system 102 may generate information indicating at least one of a pathological portion or a non - pathological portion of the first point cloud by applying a PCNN (e.g., first PCNN 200) to the first point cloud. For example, the first PCNN may be trained to identify at least one of a pathological portion or a non - pathological portion.

[0126]

[0130] In some examples, to generate information indicating at least one of a pathological portion or a non - pathological portion of the first point cloud, computing system 102 may label each point in the first point cloud as either a pathological point or a non - pathological point based on applying a first PCNN to the first point cloud. In one or more examples, a pathological point indicates being in the pathological portion, and a non - pathological point indicates being in the non - pathological portion.

[0127]

[0131] Also, in some examples, the first point cloud (e.g., point cloud 128) does not necessarily have to include all of the diseased bone. For example, the first point cloud may represent the point cloud of a diseased tibia, and the distal end of the tibia near the ankle is removed from the point cloud 128 of the diseased tibia. In such an example, to generate information indicating at least one of the pathological part or the non-pathological part of the first point cloud, the computing system 102 may be configured to generate information indicating at least one of the pathological part or the non-pathological part of the point cloud of the diseased tibia from which the distal end has been removed.

[0128]

[0132] The use of the first PCNN may not be necessary in all examples. There may be other methods for generating information indicating at least one of the pathological part and the non-pathological part. For example, a surgeon may be able to distinguish between the pathological part and the non-pathological part.

[0129]

[0133] As another example, the computing system 102 may compare the first point cloud with a statistical shape model (SSM) to generate information indicating the pathological part and the non-pathological part. For example, for the SSM, the computing system 102 can obtain the point cloud of the SSM, which is a representative model of the pre-diseased state of the anatomical structure. The processing circuit may orient the point cloud of the SSM or the point cloud representing the diseased state of the anatomical structure so that the point cloud of the SSM and the point cloud representing the diseased state of the anatomical structure are in the same orientation. The computing system 102 may determine the non-pathological points within the point cloud representing the diseased state of the anatomical structure. For example, as described above, the point cloud representing the diseased state of the anatomical structure may have a pathological part and a non-pathological part. The computing system 102 may deform the point cloud of the SSM until the points within the SSM point cloud overlap the identified non-pathological points. The computing system 102 may determine the difference between the superimposed SSM and the point cloud representing the diseased state of the anatomical structure. The result of the difference may be the pathological part.

[0130]

[0134] Computing system 102 can generate a second point group that includes points corresponding to non-pathological portions of the diseased bone but does not include points corresponding to the pathological portions of the diseased bone (504). For example, computing system 102 can generate a second point group from which the pathological portions (e.g., of the first point group) have been removed. One example of the second point group is the intermediate point group 130. As one example, computing system 102 can utilize labels indicating pathological and non-pathological points within point group 128. Computing system 102 can remove the points within point group 128 that are labeled as pathological points and maintain the points within point group 128 that are labeled as non-pathological points as they are. The result of removing the pathological points can be the intermediate point group 130, which includes points corresponding to non-pathological portions of the diseased bone but does not include points corresponding to the pathological portions of the diseased bone.

[0131]

[0135] Based on the second point group, computing system 102 can generate a third point group representing the pre-diseased state of the bone (506). As one example, computing system 102 can apply a second PCNN configured to directly generate a third point group based on the second point group.

[0132]

[0136] As another example, to generate a third point group based on the second point group, computing system 102 can use the second PCNN to determine a non-pathological estimate of the pathological portion of the diseased bone and can combine the non-pathological estimate of the pathological portion with the second point group to generate the third point group. For example, to combine the non-pathological estimate of the pathological portion with the second point group using the second PCNN, computing system 102 may fill the second point group with the non-pathological estimate of the pathological portion.

[0133]

[0137] As described above, the non-pathological estimation of the pathological part refers to the estimation of non-pathological anatomical structures (e.g., bones) that fill the pathological part removed from the first point cloud to generate the second point cloud. In other words, the non-pathological part estimation of the pathological part completes the point cloud so that there are no longer gaps in the point cloud due to the removal of the pathological part. As a result of completing the point cloud by estimating the non-pathological anatomical structure, the remodeling of the diseased bone, which is the pre-onset characterization of the diseased bone, is obtained.

[0134]

[0138] As another example, to generate a third point cloud based on the second point cloud, computing system 102 may utilize the SSM. For example, as described above, after identifying at least one of the pathological part or the non-pathological part, computing system 102 may utilize the PCNN to remove the pathological part. Computing system 102 may utilize the non-pathological part to drive the fitting of the SSM represented by the point cloud. For example, computing system 102 may orient the non-pathological part to have the same orientation as the SSM. Computing system 102 can deform (e.g., stretch, shrink, rotate, translate, etc.) the point cloud of the SSM until the points in the point cloud of the SSM overlap with the identified non-pathological points. That is, computing system 102 can determine a first deformed SSM (e.g., by stretching, shrinking, rotating, translating, etc.) and determine the distance between the corresponding points in the first deformed SSM and the points of the non-pathological part. Computing system 102 can repeat such operations for different deformed SSMs. Computing system 102 can identify the version of the SSM that overlaps with the identified non-pathological points (e.g., the version of the SSM having the points that best match the corresponding points of the non-pathological points). The resulting version of the SSM can be the third point cloud representing the pre-onset state of the bone.

[0135]

[0139] Computing system 102 may output information indicating a third point cloud representing a pre-disease state of the bone (508). As one example, reconstruction unit 206 may generate a graphical representation of the pre-disease point cloud 132 that a surgeon can view using visualization device 114 as one example.

[0136]

[0140] FIG. 6 is a flowchart illustrating an exemplary process for preoperative characterization of a patient's anatomical structure for a replacement surgery. Computing system 102 (e.g., processing circuit 104 implementing planning system 118) may obtain a first point cloud representing the bone of a patient having a prosthesis implanted during a first surgery (600). For example, in a replacement surgery, the current prosthesis may be replaced, and thus, computing system 102 may obtain point cloud 142 representing the anatomical structure of the patient having the prosthesis. In some examples, the prosthesis is for one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, or ilium.

[0137]

[0141] Based on the first point cloud, computing system 102 may generate a second point cloud representing the patient's anatomical structure at the time of the first surgery (602). For example, computing system 102 may generate preoperative point cloud 144 by applying a third PCNN (e.g., as described above with respect to FIG. 2) to point cloud 142. In some examples, computing system 102 may generate preoperative point cloud 144 without using a portion of point cloud 142 that includes the prosthesis.

[0138]

[0142] Computing system 102 may output information indicating the second point cloud (604). As one example, reconstruction unit 206 may generate a graphical representation of preoperative point cloud 144 that a surgeon can view using visualization device 114 as one example.

[0139]

[0143] Although the techniques have been disclosed with respect to a limited number of examples, those of ordinary skill in the art obtaining the benefits of this disclosure will appreciate a number of modifications and variations thereof. For example, any reasonable combination of the described examples is contemplated to be practicable. The appended claims are intended to embrace modifications and variations that fall within the true spirit and scope of the invention.

[0140]

[0144] In some examples, it should be recognized that some of the operations or events of any of the techniques described herein can be performed in a different order, can be added, combined, or entirely excluded (e.g., not all of the described operations or events are necessarily required for the practice of the technique). Further, in some examples, the operations or events can be performed concurrently rather than sequentially, for example, by multithreading, interrupt processing, or by a plurality of processors.

[0141]

[0145] In one or more examples, the described functionality can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functionality can be stored on or transmitted across a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium can include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, the computer-readable medium generally can correspond to (1) a tangible non-transitory computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[0142]

[0146] By way of example and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM (registered trademark), CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that is accessible by a computer. Also, any connection can be properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather non-transitory tangible storage media are intended. As used herein, disk and disc include compact disc (CD), laser disc (registered trademark), optical disc, digital versatile disc (DVD), floppy (registered trademark) disk, and Blu-ray disc, where disk typically magnetically reproduces data, while disc optically reproduces data with a laser. The above combinations should also be included within the scope of computer-readable media.

[0143]

[0147] The operations described in this disclosure may be performed by one or more processors, which may be implemented as fixed-function processing circuits, programmable circuits, or combinations thereof, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated logic circuits or discrete logic circuits. Fixed-function circuits refer to circuits that provide a specific function and are preset to the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide adaptable functions in the operations that can be performed. For example, a programmable circuit can execute instructions specified by software or firmware that cause the programmable circuit to operate as defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the types of operations performed by fixed-function circuits are generally invariant. Thus, the terms "processor" and "processing circuit" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein.

Claims

1. A method for pre-disease characterization of a patient's anatomical structure, the method comprising: obtaining, by a computing system, a first point cloud representing a diseased state of a bone of a patient; generating, by the computing system, information indicating at least one of a pathological portion of the first point cloud or a non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to the pathological portion of the diseased state of the bone, and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to the non-pathological portion of the diseased state of the bone; generating, by the computing system, a second point cloud that includes points corresponding to the non-pathological portion of the diseased state of the bone but does not include points corresponding to the pathological portion of the diseased state of the bone; generating, by the computing system, a third point cloud representing a pre-disease state of the bone based on the second point cloud; outputting, by the computing system, information indicating the third point cloud representing the pre-disease state of the bone A method comprising the above steps.

2. Generating the third point cloud comprises: determining a non-pathological estimate of the pathological portion of the diseased state of the bone; combining the non-pathological estimate of the pathological portion and the second point cloud to generate the third point cloud The method according to claim 1, comprising the above steps.

3. Combining the non-pathological estimate of the pathological portion and the second point cloud comprises filling the second point cloud with the non-pathological estimate of the pathological portion. The method according to claim 2, comprising the above step.

4. Generating information indicating at least one of the pathological portion of the first point cloud or the non-pathological portion of the first point cloud is based on applying a point cloud neural network (PCNN) to the first point cloud to generate information indicating at least one of the pathological portion of the first point cloud or the non-pathological portion of the first point cloud, wherein the PCNN is trained to identify at least one of the pathological portion or the non-pathological portion. The method according to any one of claims 1 to 3, comprising the above step.

5. Generating the third point cloud representing the pre-disease state of the bone comprises generating the third point cloud based on applying a point cloud neural network (PCNN) to the second point cloud, according to the method of any one of claims 1 to 3.

6. Generating information indicating at least one of the pathological part or the non-pathological part of the first point cloud comprises generating information indicating at least one of the pathological part or the non-pathological part of the first point cloud based on applying a first point cloud neural network (PCNN) to the first point cloud, wherein the first PCNN is trained to identify at least one of the pathological part or the non-pathological part. Generating the third point cloud representing the pre-disease state of the bone comprises generating the third point cloud based on applying a second PCNN to the second point cloud, according to the method of any one of claims 1 to 3.

7. The method according to any one of claims 1 to 6, wherein the bone comprises at least one of a tibia, a fibula, a scapula, a humeral head, a femur, a patella, a vertebra, an iliac crest, an ilium, an ischial spine, or a coccyx.

8. Generating information indicating at least one of the pathological part or the non-pathological part of the first point cloud comprises, based on applying a point cloud neural network (PCNN) to the first point cloud, labeling each point in the first point cloud as either a pathological point or a non-pathological point, wherein the pathological point indicates being in the pathological part and the non-pathological point indicates being in the non-pathological part, according to the method of any one of claims 1 to 7.

9. The first point cloud representing the diseased state of the bone of the patient comprises a point cloud of a diseased tibia, points in the point cloud representing the distal end of the tibia close to the ankle are removed from the point cloud of the diseased tibia, and generating information indicating at least one of the pathological part or the non-pathological part of the first point cloud comprises generating information indicating at least one of the pathological part of the point cloud or the non-pathological part of the point cloud of the diseased tibia from which the distal end has been removed, according to the method of any one of claims 1 to 8.

10. A method for preoperative characterization of a patient's anatomical structure for replacement surgery, the method comprising Obtaining, by a computing system, a first point cloud representing the bone of a patient having a prosthesis implanted during an initial surgery; Generating, by the computing system, a second point cloud representing the bone of the patient at the time of the initial surgery based on the first point cloud; Outputting, by the computing system, information indicating the second point cloud; A method comprising: **Claim 11** The method according to claim 10, wherein generating the second point cloud comprises generating the second point cloud based on applying a point cloud neural network (PCNN) to the first point cloud. **Claim 12** The method according to claim 10 or 11, wherein generating the second point cloud comprises generating the second point cloud without using a portion of the first point cloud that includes the prosthesis. **Claim 13** The method according to any one of claims 10 to 12, wherein the prosthesis is for one of a tibia, a fibula, a scapula, a humeral head, a femur, a patella, a vertebra, or an ilium. **Claim 14** A memory system configured to store a first point cloud representing a diseased state of a patient's bone, and a processing circuit, wherein the processing circuit obtains the first point cloud representing the diseased state of the patient's bone, generates information indicating at least one of a pathological portion of the first point cloud or a non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to the pathological portion of the diseased state of the bone, and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to the non-pathological portion of the diseased state of the bone, generates a second point cloud that includes points corresponding to the non-pathological portion of the diseased state of the bone but does not include points corresponding to the pathological portion of the diseased state of the bone, generates a third point cloud representing a pre-disease state of the bone based on the second point cloud, and outputs information indicating the third point cloud representing the pre-disease state of the bone. A system configured to perform the above. **Claim 15** For generating the third point cloud, the processing circuit is configured to determine a non-pathological estimation of the pathological portion of the diseased state of the bone, and combine the non-pathological estimation of the pathological portion with the second point cloud for generating the third point cloud. The system according to claim 14, configured to perform the above. ​

16. The method according to claim 15, wherein, in order to combine the non-pathological estimation of the pathological part and the second point cloud, the processing circuit is configured to fill the second point cloud with the non-pathological estimation of the pathological part.

17. The system according to any one of claims 14 to 16, wherein, in order to generate information indicating at least one of the pathological part of the first point cloud or the non-pathological part of the first point cloud, the processing circuit is configured to generate information indicating at least one of the pathological part of the first point cloud or the non-pathological part of the first point cloud based on applying a point cloud neural network (PCNN) to the first point cloud, wherein the PCNN is trained to identify at least one of the pathological part or the non-pathological part.

18. The system according to any one of claims 14 to 16, wherein, in order to generate the third point cloud representing the pre-disease state of the bone, the processing circuit is configured to generate the third point cloud based on applying a point cloud neural network (PCNN) to the second point cloud.

19. In order to generate information indicating at least one of the pathological part of the first point cloud or the non-pathological part of the first point cloud, the processing circuit is configured to generate information indicating at least one of the pathological part of the first point cloud or the non-pathological part of the first point cloud based on applying a first point cloud neural network (PCNN) to the first point cloud, wherein the first PCNN is trained to identify at least one of the pathological part or the non-pathological part. The system according to any one of claims 14 to 16, wherein, in order to generate the third point cloud representing the pre-disease state of the bone, the processing circuit is configured to generate the third point cloud based on applying a second PCNN to the second point cloud.

20. The system according to any one of claims 14 to 19, wherein the bone comprises at least one of a tibia, a fibula, a scapula, a humeral head, a femur, a patella, a vertebra, an iliac crest, an ilium, an ischial spine, or a coccyx.

21. To generate information indicating at least one of the pathological part of the first point cloud or the non-pathological part of the first point cloud, the processing circuit is configured to label each point in the first point cloud as either a pathological point or a non-pathological point based on applying a point cloud neural network (PCNN) to the first point cloud, wherein the pathological points indicate being in the pathological part and the non-pathological points indicate being in the non-pathological part. The system according to any one of claims 14 to 20.

22. The first point cloud representing the diseased state of the bone of the patient comprises a point cloud of a diseased tibia, and points in the point cloud representing the distal end of the tibia close to the ankle are removed from the point cloud of the diseased tibia. To generate information indicating at least one of the pathological part of the first point cloud or the non-pathological part of the first point cloud, the processing circuit is configured to generate information indicating at least one of the pathological part of the point cloud or the non-pathological part of the point cloud of the diseased tibia from which the distal end has been removed. The system according to any one of claims 14 to 21.

23. A storage system configured to store a first point cloud representing the bone of a patient having a prosthesis implanted during a first surgery, and a processing circuit, wherein the processing circuit obtains the first point cloud representing the bone of the patient having the prosthesis implanted during the first surgery, generates a second point cloud representing the bone of the patient at the time of the first surgery based on the first point cloud, and outputs information indicating the second point cloud is configured to perform. A system.

24. To generate the second point cloud, the processing circuit is configured to generate the second point cloud based on applying a point cloud neural network (PCNN) to the first point cloud. The system according to claim 23.

25. To generate the second point cloud, the processing circuit is configured to generate the second point cloud without using the part of the first point cloud that includes the prosthesis. The system according to claim 23 or 24.

26. The prosthesis is for one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, or ilium. The system according to any one of claims 23 to 25.

27. A system comprising means for implementing the method according to any one of claims 1 to 9 or 10 to 13.

28. A computer-readable storage medium storing instructions that, when executed, cause one or more processors to implement the method according to any one of claims 1 to 9 or 10 to 13.

Images