Automated pre-symptomatic characterization of patient anatomical structures using point clouds.

Abstract

A method for pre-disease characterization of a patient's anatomical structure includes obtaining a first point cloud representing a diseased state of the patient's bone, generating information indicating at least one of a pathological part of the first point cloud or a non-pathological part of the first point cloud, wherein the pathological part of the first point cloud corresponds to the pathological part of the diseased state of the bone and the non-pathological part of the first point cloud corresponds to the non-pathological part of the diseased state of the bone, generating a second point cloud that includes points corresponding to the non-pathological part but does not include points corresponding to the pathological part, generating a third point cloud representing a pre-disease state of the bone based on the second point cloud, and outputting information indicating the third point cloud representing the pre-disease state of the bone.

Description

Technical Field

[0001]

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 350,732, filed on June 9, 2022, the entire content of which is incorporated herein by reference.

Background Art

[0002]

[0002] Surgical joint repair procedures include the repair and / or replacement of damaged or diseased joints. A surgical joint repair procedure, such as arthroplasty as an example, may include replacing a damaged joint with a prosthesis implanted into the patient's bone. The appropriate selection or design of a properly sized and formed prosthesis and the proper placement of that prosthesis are important to ensure an optimal surgical outcome. A surgeon may analyze the damaged bone to assist in selecting, designing, and / or placing the prosthesis and further in performing surgical procedures to prepare the bone or tissue to receive or interact with the prosthesis.

Summary of the Invention

[0003]

[0003] This disclosure describes exemplary techniques for performing pre-morbid characterization of patient anatomical structures, such as one or more anatomical objects. Pre-morbid characterization refers to determining a predictive model that predicts the characteristics (e.g., size, shape, location) of a patient anatomical structure as an anatomical structure that existed prior to injury to the patient anatomical structure or disease progression of the anatomical structure. In an example described in this disclosure, the predictive model can be a point cloud of the pre-morbid anatomical structure of a pathological anatomical structure (e.g., the pre-morbid state of a bone). A processing circuit can be configured to further process the point cloud of the pre-morbid anatomical structure. For example, the processing circuit can generate a graphic shape model of the pre-morbid anatomical structure that a surgeon can view to assist in planning a surgical procedure (e.g., for repairing or replacing an orthopedic joint).

[0004]

[0004] In one or more examples, the processing circuit may be configured to utilize a point cloud neural network (PCNN) to generate a point cloud of anatomical structures before the onset of disease. For example, the processing circuit may apply a first PCNN to the point cloud representation of the pathological state of an anatomical structure in order to distinguish between pathological (e.g., deformed) and non-pathological (e.g., undeformed) parts of the pathological state of the anatomical structure. The pathological parts of the point cloud are the parts of the point cloud corresponding to the pathological parts of the pathological state of the anatomical structure, and the non-pathological parts of the point cloud are the parts of the point cloud corresponding to the non-pathological parts of the pathological state of the anatomical structure.

[0005]

[0005] The processing circuit may remove pathological portions from the point cloud and then apply a second PCNN to the non-pathological portions to generate a point cloud representing the pre-pathological state of the anatomical structures (e.g., pre-pathological characterization of the patient's anatomical structures). In this way, the exemplary technique utilizes point cloud processing with a neural network that can improve the accuracy of determining the pre-pathological characterization of the patient's anatomical structures.

[0006]

[0006] In one or more examples, it may not be necessary to utilize both the first and second PCNNs. For example, the processing circuit may determine the pathological and non-pathological parts of an anatomical structure in a pathological condition using techniques that do not necessarily rely on PCNNs, such as surgical input, comparison with point clouds of other similar patients with non-pathological anatomical structures, and statistical shape models (SSM). In such examples, after removing the pathological parts, the processing circuit may utilize PCNNs to generate a point cloud representing the pre-pathological state of the anatomical structure. In another example, the processing circuit may apply PCNNs to the first point cloud to distinguish between the pathological and non-pathological parts of an anatomical structure in a pathological condition. After removing the pathological parts, the processing circuit may generate a point cloud representing the pre-pathological state of the anatomical structure based on surgical input, comparison with point clouds of other similar patients with non-pathological anatomical structures, SSM, etc., without necessarily using PCNNs.

[0007]

[0007] The above example describes pre-disease characterization techniques to aid in surgical planning. The exemplary techniques described herein are not limiting. In some examples, the processing circuit may be configured to perform the exemplary techniques described herein for revision surgery. A patient may have had a prosthesis implanted. However, at some point in the future, surgery may be necessary to address disease progression, prosthesis migration, or because the prosthesis has reached the end of its service life. Surgery to replace the current prosthesis with another prosthesis is called revision surgery.

[0008]

[0008] In one or more examples, the processing circuit may acquire a first point cloud representing the patient's anatomical structure with the prosthesis implanted during the first surgery. Based on the first point cloud, the processing circuit may generate a second point cloud representing the patient's anatomical structure before the first surgery. For example, the processing circuit may generate the second point cloud by applying a point cloud neural network (PCNN) to the first point cloud. In this example, the PCNN may be trained to generate the patient's anatomical structure before the first surgery (e.g., in the affected or injured state that led to the surgery). The processing circuit may output information indicating the second point cloud (e.g., representing the patient's anatomical structure before the first surgery).

[0009]

[0009] In one example, the present disclosure describes a method for pre-pathological characterization of a patient's anatomical structure, the method comprising: acquiring a first point cloud representing a pathological state of the patient's bones by a computing system; generating information by the computing system indicating at least one of the pathological portion or the non-pathological portion of the first point cloud; the pathological portion of the first point cloud being the portion of the first point cloud corresponding to the pathological portion of the bone's pathological state, and the non-pathological portion of the first point cloud being the portion of the first point cloud corresponding to the non-pathological portion of the bone's pathological state; generating a second point cloud by the computing system that includes points corresponding to the non-pathological portion of the bone's pathological state but does not include points corresponding to the pathological portion of the bone's pathological state; generating a third point cloud representing the pre-pathological state of the bone by the computing system based on the second point cloud; and outputting information by the computing system indicating the third point cloud representing the pre-pathological state of the bone.

[0010]

[0010] In one example, the present disclosure describes a method for preoperative characterization of a patient's anatomical structure for revision surgery, the method comprising: acquiring a first point cloud representing the bone of a patient having a prosthesis implanted during the first surgery using a computing system; generating a second point cloud representing the bone of the patient at the time of the first surgery using the computing system based on the first point cloud; and outputting information indicating the second point cloud using the computing system.

[0011]

[0011] In one example, the present disclosure describes a system comprising: a storage system configured to store a first point cloud representing a pathological state of the bone of a patient; and a processing circuit, the processing circuit being configured to: acquire a first point cloud representing a pathological state of the bone of a patient; generate information indicating at least one of a pathological portion of the first point cloud or a non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to a pathological portion of the bone pathology, and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to a non-pathological portion of the bone pathology; the processing circuit being configured to further generate a second point cloud including points corresponding to the non-pathological portion of the bone pathology but not points corresponding to the pathological portion of the bone pathology; generate a third point cloud representing a pre-pathological state of the bone based on the second point cloud; and output information indicating the third point cloud representing the pre-pathological state of the bone.

[0012]

[0012] In one example, the present disclosure describes a system comprising: a storage system configured to store a first point cloud representing the bone of a patient having a prosthesis implanted during an initial surgery; a processing circuit configured to acquire the first point cloud representing the bone of a patient having a prosthesis implanted during an initial surgery; a second point cloud representing the bone of the patient at the time of the initial surgery based on the first point cloud; and an output of information indicating the second point cloud.

[0013]

[0013] In one or more examples, the disclosure describes a system comprising means for carrying out the method of the disclosure and a computer-readable storage medium storing instructions that cause a computing system to carry out the method of the disclosure when executed.

[0014]

[0014] Details of various examples of the present disclosure are shown in the accompanying drawings and the following description. Various features, purposes, and advantages will become apparent from this description, drawings, and claims. [Brief explanation of the drawing]

[0015] [Figure 1A]

[0015] A block diagram showing an exemplary system that may be used to implement the techniques of the present disclosure. [Figure 1B]

[0016] A block diagram illustrating another exemplary system that may be used to implement the techniques of this disclosure. [Figure 2]

[0017] A block diagram illustrating exemplary components of a planning system using one or more techniques of the present disclosure. [Figure 3]

[0018] A conceptual diagram illustrating an exemplary point cloud neural network (PCNN) using one or more of the techniques of this disclosure. [Figure 4]

[0019] A flowchart illustrating an exemplary architecture of a T-Net model using one or more of the techniques of this disclosure. [Figure 5]

[0020] A flowchart illustrating exemplary processes for pre-pathogenic characterization of patient anatomical structures using one or more techniques of the present disclosure. [Figure 6]

[0021] A flowchart illustrating an exemplary process for preoperative characterization of patient anatomical structures for revision surgery. [Modes for carrying out the invention]

[0016]

[0022] A patient may have a disease (e.g., illness) that causes damage to the patient's anatomical structures, or a patient may have an injury that causes damage to the patient's anatomical structures. To address the disease or injury, a surgeon may perform a surgical procedure. It may be beneficial for a surgeon to determine, before surgery, the characteristics of the patient's anatomical structures before the injury, known as pre-disease characteristics (e.g., size, shape, and / or location). For example, determining the pre-disease characteristics of the patient's anatomical structures can help in planning the surgical procedures for selecting, designing, and / or placing prostheses, as well as for preparing the damaged bone surface to accept or interact with the prosthesis. Prior planning allows a surgeon to determine, before surgery rather than during surgery, the procedures for preparing the bone or tissue, the tools that will be needed, the size and shape of the tools, the size and shape of one or more prostheses to be implanted, or other characteristics, etc.

[0017]

[0023] For example, regarding bone as one example of a patient's anatomical structure, reconstruction of the bone before injury (e.g., pre-disease characteristics of the bone) may be useful in helping surgeons fix the injured bone. For example, digital reconstruction of the pre-disease anatomical structure may help verify the possible surgeries required and the function of adjacent joints. For example, an overlay of the injured bone and the reconstructed bone (e.g., a digital representation of the pre-disease bone) may help identify which tools are needed.

[0018]

[0024] As mentioned above, pre-symptomatic characterization refers to characterizing the anatomical structures of a patient that existed before the patient developed a disease or injury. However, since patients may not consult a doctor or surgeon until they develop a disease or injury, pre-symptomatic characterization of anatomical structures is not generally available.

[0019]

[0025] The pre-disease anatomical structure, also called the native anatomical structure, refers to the anatomical structure before the onset of a disease or injury. Even after a disease or injury, there may be parts of the healthy anatomical structure and parts of the non-healthy (e.g., diseased or damaged) anatomical structure. The diseased or damaged part of the anatomical structure is called the pathological anatomical structure, and the healthy part of the anatomical structure is called the non-pathological anatomical structure.

[0020]

[0026] The present disclosure describes exemplary techniques for determining a representation of the pre-disease state of an anatomical structure (e.g., a predictor of the pre-disease anatomical structure) using point cloud processing such as a Point Cloud Neural Network (PCNN). The PCNN is implemented using an architecture based on a point cloud learning model. An architecture based on a point cloud learning model (e.g., a point cloud learning model) is a neural network-based architecture that receives one or more point clouds as input and generates one or more point clouds as output. Exemplary point cloud learning models include PointNet, PointTransformer, etc.

[0021]

[0027] In one or more examples described in the present disclosure, the processing circuit may be configured to determine (e.g., obtain) a first point cloud representing the diseased state of a patient's anatomical structure (e.g., a damaged or diseased patient's anatomical structure). For example, the processing circuit may receive one or more images of the patient's anatomical structure in a diseased state and determine the first point cloud based on the received one or more images.

[0022]

[0028] The present disclosure describes a processing circuit for obtaining a first point cloud. As one example of obtaining the first point cloud, the processing circuit may receive one or more images of the patient's anatomical structure in a diseased state and determine the first point cloud based on the received one or more images. As another example, some other circuit may generate the first point cloud, and the processing circuit may receive the generated first point cloud in order to obtain the first point cloud.

[0023]

[0029] The processing circuit can generate information indicating at least one of the pathological or non-pathological parts of a first point cloud. The pathological part of the first point cloud is the part of the first point cloud corresponding to the pathological part of a pathological condition of an anatomical structure, and the non-pathological part of the first point cloud is the part of the first point cloud corresponding to the non-pathological part of a pathological condition of an anatomical structure. As one example, the processing circuit can generate information indicating at least one of the pathological or non-pathological parts of the first point cloud based on applying a point cloud neural network (PCNN) to the first point cloud. In this example, the PCNN can be trained to identify at least one of the pathological or non-pathological parts.

[0024]

[0030] For example, during training, a PCNN can receive point clouds representing various bone pathological conditions as input, or it can receive information identifying pathological and non-pathological parts of an input point cloud as input. As an example, a surgeon might provide information about the pathological and non-pathological parts that form the ground truth of an input point cloud representing various bone pathological conditions. A processing circuit for training the PCNN may be configured to determine weights and other factors that, when applied to the input point cloud, generate information indicating pathological and non-pathological parts consistent with the decisions made by the surgeon. As a result of training, the PCNN may be trained such that the processing circuit can be applied to the first point cloud to generate information indicating at least one of the pathological or non-pathological parts of the first point cloud.

[0025]

[0031] The use of PCNNs to determine pathological and non-pathological areas is not necessary in all cases. In some cases, the processing circuit may utilize other techniques to determine pathological and non-pathological areas, such as taking input from a surgeon, comparing the patient with similar anatomical structures, or using a static shape model (SSM).

[0026]

[0032] For example, with respect to SSM, the processing circuit can acquire a point cloud of SSM, which is a representative model of the pre-pathological state of the anatomical structure. The processing circuit can orient the point cloud of SSM, or the point cloud representing the pathological state of the anatomical structure, so that the point cloud of SSM and the point cloud representing the pathological state of the anatomical structure are oriented in the same direction. The processing circuit can determine non-pathological points within the point cloud representing the pathological state of the anatomical structure. For example, as mentioned above, the point cloud representing the pathological state of the anatomical structure may contain pathological and non-pathological parts. The processing circuit can identify one or more points (called non-pathological points) within the non-pathological parts. Non-pathological points for identification in the point cloud representing the pathological state of the anatomical structure may be predefined based on the cause of the pathological state (for example, there may be certain parts of the anatomical structure that are known not to be affected by the disease). The processing circuit can deform the point cloud of SSM until the points in the point cloud of SSM overlap with the identified non-pathological points. The processing circuit can determine the differences between the superimposed SSM and the point cloud representing the pathological state of the anatomical structure. The resulting differences may represent pathological areas.

[0027]

[0033] While SSMs can be used to determine the pre-pathological state of anatomical structures, their use may not always be as accurate as desired. However, using SSMs to distinguish between pathological and non-pathological portions of a point cloud representing the pathological state of an anatomical structure may be sufficiently accurate.

[0028]

[0034] The processing circuit may be configured to generate a second point group that includes points corresponding to the non-pathological portion of the pathological state of the anatomical structure, but does not include points corresponding to the pathological portion of the pathological state of the anatomical structure. For example, the processing circuit may be configured to remove portions of the first point group that represent the deformed anatomical structure in order to generate the second point group. That is, the processing circuit may generate a second point group from which the pathological portion has been removed, such that the second point group includes points corresponding to the non-pathological portion, but does not include points corresponding to the pathological portion. In such an example, the second point group may be the first point group from which portions containing the deformed anatomical structure (e.g., the pathological portion) have been removed, leaving the non-deformed anatomical structure (e.g., the non-pathological portion).

[0029]

[0035] According to one or more examples described herein, a processing circuit may be configured to generate a third point cloud representing the pre-pathological state of a pathological anatomical structure (e.g., the pre-pathological state of bone) based on a second point cloud. There may be various exemplary methods for generating the third point cloud representing the pre-pathological state of an anatomical structure based on the second point cloud. As one example, the processing circuit may generate the third point cloud by applying a PCNN to the second point cloud. For example, the PCNN may be trained to reconstruct the pre-pathological anatomical structure from a point cloud of non-pathological parts of the anatomical structure.

[0030]

[0036] For example, during training, a PCNN may receive point clouds representing non-pathological parts of various bones (e.g., an incomplete point cloud of healthy bone) as input, or it may receive point clouds of healthy bone as input. For example, with a point cloud of healthy bone, during training, the processing circuit or the user may remove N% of the points from the point cloud, and possibly from different regions of the point cloud. The remaining part of the bone may be considered non-pathological. The processing circuit may receive point clouds of both non-pathological and healthy bone.

[0031]

[0037] To train a PCNN, the processing circuit may be configured to determine weights and other factors that, when applied to an input point cloud containing non-pathological portions, generate a point cloud consistent with a point cloud of healthy bone. As a result of training, a trained PCNN may be obtained, to which the processing circuit may be applied to the second point cloud to generate a third point cloud representing the pre-pathological state of anatomical structures.

[0032]

[0038] As another example, a processing circuit may determine a non-pathological estimate of the pathological portion of an anatomical structure. The non-pathological estimate may be considered an estimate of what the pathological portion of the first point cloud looked like before the injury. The processing circuit may combine the non-pathological estimate of the pathological portion with the second point cloud to generate a third point cloud. For example, to combine the non-pathological estimate of the pathological portion with the second point cloud, the processing circuit may fill the second point cloud with the non-pathological estimate of the pathological portion.

[0033]

[0039] For example, a PCNN can be trained to determine non-pathological estimates of pathological parts of an anatomical structure in a pathological state. For example, for training similar to the above, a PCNN may receive a point cloud representing the non-pathological parts of various bones and healthy bones. In such an example, the PCNN can be trained to use the points of the non-pathological parts to generate an estimate of the non-pathological parts of the bone (e.g., how the pathological parts looked before disease or trauma). The processing circuit may combine the second point cloud with the non-pathological estimates of the pathological parts to generate a third point cloud representing the pre-pathological state of the bone. For example, a PCNN can be trained to fill in the pathological parts of the bone with estimates of the non-pathological parts of the bone to complete the pre-pathological representation of the bone.

[0034]

[0040] Non-pathological estimation of pathological parts refers to the estimation of non-pathological anatomical structures (e.g., bone) that fill in the removed pathological parts in order to obtain the pre-pathological state of the anatomical structure as a result. For example, non-pathological estimation of pathological parts removed from a first point cloud to generate a second point cloud completes the second point cloud so that there are no longer any gaps in the second point cloud after the removal of the pathological parts. In one or more examples, non-pathological estimation is called "estimation" because the PCNN can be configured to fill in the removed pathological parts with non-pathological anatomical structures (e.g., non-pathological bone) that the PCNN has determined to be representative of the pathological parts.

[0035]

[0041] In the example above, the processing circuit may use a PCNN to determine the pathological and / or non-pathological parts of the first point cloud to generate a second point cloud containing and not containing pathological parts, or it may use a PCNN on the second point cloud to generate a third point cloud representing the pre-pathological state of an anatomical structure (e.g., bone). However, in some examples, the processing circuit may utilize two PCNNs: one for determining the pathological and / or non-pathological parts of the first point cloud to generate a second point cloud containing and not containing pathological parts, and another for generating a third point cloud representing the pre-pathological state of an anatomical structure based on the second point cloud. For example, the processing circuit may generate information indicating at least one of the pathological or non-pathological parts of the first point cloud by applying the first PCNN to the first point cloud, and the first PCNN is trained to identify at least one of the pathological or non-pathological parts. The processing circuit can generate a third point cloud representing the pre-disease state of anatomical structures by applying a second PCNN to the second point cloud based on the second point cloud.

[0036]

[0042] The above example described using a PCNN to generate a third point cloud representing the pre-pathological state of the bone. However, the illustrative techniques are not so limited. In some examples, the processing circuit may utilize an SSM or some other technique to generate a third point cloud representing the pre-pathological state of the bone. For example, the processing circuit may use a PCNN to remove the pathological portion after identifying at least one of the pathological or non-pathological portion. The processing circuit may utilize the non-pathological portion to drive the fitting of the SSM represented in the point cloud. For example, the processing circuit may deform the SSM point cloud (e.g., stretch, shrink, rotate, translate, etc.) until points in the SSM point cloud overlap with identified non-pathological points. The deformed point cloud of the SSM that overlaps with identified non-pathological points can become a third point cloud representing the pre-pathological state of the bone.

[0037]

[0043] The processing circuit may output information representing a point cloud of the pre-disease state of an anatomical structure (e.g., bone). In some cases, the processing circuit may further process a third point cloud of the pre-disease anatomical structure to generate a graphic representation of the pre-disease anatomical structure, or other information such as dimensions that the surgeon can use for preoperative planning or during surgery. For example, a surgeon may use the graphic representation to plan before surgery which tools to use, where to cut, etc. The graphic representation of the pre-disease anatomical structure may enable the surgeon to determine which prosthesis to use and how to perform the implantation surgery so that the surgical outcome, and the patient's experience (e.g., regarding motor function), is approximately the same as before the patient experienced the injury or disease. In some cases, during surgery, the surgeon may wear augmented reality (AR) goggles that display an overlay of the graphic representation of the pre-disease anatomical structure on top of the pathological anatomical structure to help the surgeon ensure during surgery that the prosthesis approximates the pre-disease anatomical structure.

[0038]

[0044] The above describes exemplary techniques for pre-symptomatic characterization of anatomical structures before the onset of disease. However, exemplary techniques are not so limited. In some examples, the processing circuit may be configured to perform exemplary techniques for revision surgery as described in this disclosure. A surgeon may perform the initial surgery to implant a prosthesis. Over time, the effectiveness of the prosthesis may decrease. For example, the effectiveness of the prosthesis may decrease as the disease progresses, as the prosthesis may migrate, or as the practical lifespan of the prosthesis nears its end.

[0039]

[0045] In such cases, the surgeon may determine that revision surgery is appropriate. In revision surgery, the surgeon removes the current prosthesis and implants another prosthesis that is a better fit for the patient's current condition. When planning and performing revision surgery, the surgeon may find it desirable to determine the size and shape of the anatomical structure at the time the initial surgery was performed. That is, the surgeon may want to determine what the anatomical structure was that caused the initial surgery. In some cases, though not all, the surgeon may be interested in the characterization of the anatomical structure at the time of the initial surgery (e.g., size and shape), but less interested in its pre-symptomatic shape.

[0040]

[0046] There may be several reasons why it is desirable for surgeons to determine the characterization of anatomical structures at the time of the initial surgery. One example is that ligaments and other joint structures may change after surgery. For instance, ligaments may become stiff. In revision surgery, attempting to reconstruct the damaged anatomical structure to its original pre-disease state can have adverse effects on those altered ligaments and other joints. Therefore, it may be desirable for surgeons to determine the size and shape of anatomical structures at the time of the initial surgery and to plan the surgery accordingly.

[0041]

[0047] In one or more examples, with respect to the preoperative characterization of the patient's anatomical structure for revision surgery (e.g., characterization before the initial surgery), the processing circuit may obtain a first point cloud representing the anatomical structure of the patient with the implanted prosthesis during the initial surgery. The processing circuit may generate a second point cloud representing the patient's anatomical structure at the time of the initial surgery. For example, the processing circuit may generate the second point cloud by applying a PCNN to the first point cloud. The processing circuit may output information indicating the second point cloud.

[0042]

[0048] For example, during training, a processing circuit for training a PCNN might accept as input a point cloud of various bones with currently implanted prostheses and a point cloud of the same bones at the time the prostheses were implanted. The processing circuit could determine weights and other factors that, when applied to the input point clouds of various bones with prostheses, generate a point cloud consistent with the point cloud of the same bones at the time the prostheses were implanted. As a result, a trained PCNN could output a second point cloud representing the bones at the time the prostheses were implanted, based on the first input point cloud of the bones with implants.

[0043]

[0049] The use of PCNNs for revision surgery can be beneficial for a variety of reasons. For example, at the time of the initial surgery, the surgeon may not have requested a representation of the anatomical structure in its pathological state (e.g., the affected or injured condition that led to the initial surgery). In some cases, even if the surgeon requested a representation of the anatomical structure in its pathological state, that representation may be lost. The exemplary techniques described in this disclosure may allow the processing circuit to be configured to determine the preoperative characterization of the anatomical structure even when such preoperative characterization information is unavailable.

[0044]

[0050] Figure 1A is a block diagram showing an exemplary system 100A that may be used to implement the techniques of the present disclosure. Figure 1A shows a computing system 102, which is an example of one or more computing devices configured to implement one or more exemplary techniques described in the present disclosure. The computing system 102 may include various types of computing devices, such as a server computer, a personal computer, a smartphone, a laptop computer, and other types of computing devices. In some examples, the computing system 102 includes multiple computing devices that communicate with each other. In other examples, the computing system 102 includes only a single computing device. The computing system 102 includes a processing circuit 104, a storage system 106, a display 108, and a communication interface 110. The display 108 is optional, for example, in an example where the computing system 102 is a server computer.

[0045]

[0051] Examples of the processing circuit 104 include one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Generally, the processing circuit 104 can be implemented as a fixed-function circuit, a programmable circuit, or a combination thereof. A fixed-function circuit is a circuit that provides a specific function and is preset to an operation that can be performed. A programmable circuit is a circuit that can be programmed to perform a variety of tasks and implements adaptive functions in an operation that can be performed. For example, a programmable circuit may execute software or firmware that operates the programmable circuit as defined by software or firmware instructions. A fixed-function circuit may execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more of the units may be discrete circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits. In some examples, the processing circuit 104 is distributed across multiple computing devices of the computing system 102 and the visualization device 114. In some examples, the processing circuit 104 is housed within a single computing device of the computing system 102.

[0046]

[0052] The processing circuit 104 may include a programmable core formed from an arithmetic logic unit (ALU), an elementary function unit (EFU), digital circuits, analog circuits, and / or programmable circuits. In an example where the operation of the processing circuit 104 is carried out using software executed by the programmable circuits, the memory system 106 may store object code of the software that the processing circuit 104 receives and executes, or another memory (not shown) within the processing circuit 104 may store such instructions. An example of software includes software designed for surgical planning.

[0047]

[0053] The storage system 106 may be formed by any of various memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM®), or other types of memory devices. Examples of displays 108 include liquid crystal displays (LCDs), plasma displays, organic light-emitting diode (OLED) displays, or other types of display devices. In some examples, the storage system 106 may include multiple separate memory devices, such as multiple disk drives and memory modules, which may be distributed across multiple computing devices or housed within the same computing device.

[0048]

[0054] The communication interface 110 enables the computing system 102 to communicate with other devices via the network 112. For example, the computing system 102 may output medical images, segmentation mask images, and other information for display. The communication interface 110 may include hardware circuitry that enables the computing system 102 to communicate (for example, wirelessly or using wires) with other computing systems and devices such as the visualization device 114 and the imaging system 116. The network 112 may include various types of communication networks, including one or more wide area networks such as the Internet, local area networks, etc. In some examples, the network 112 may include wired and / or wireless communication links.

[0049]

[0055] The visualization device 114 may utilize various visualization techniques to display image content to the surgeon. In some examples, the visualization device 114 is a computer monitor or display screen. In some examples, the visualization device 114 may be a mixed reality (MR) visualization device, a virtual reality (VR) visualization device, a holographic projector, or other device for presenting augmented reality (XR) visualization. For example, in some examples, the visualization device 114 may be a similar device, such as the Microsoft HOLOLENS® headset, commercially available from Microsoft in Redmond, Washington, USA, or a similar MR visualization device including a waveguide. The HOLOLENS® device can be used to present 3D virtual objects through a holographic lens or waveguide while allowing the user to see real objects in a real-world scene, i.e., in a real-world environment, through the holographic lens. In some examples, there may be multiple visualization devices for multiple users.

[0050]

[0056] The visualization device 114 may utilize visualization tools that are available to use patient image data to generate a three-dimensional model of bone contour, segmentation masks, or other data to facilitate preoperative planning. These tools may enable surgeons to design and / or select surgical guides and implant components that closely match the patient's anatomical structure. These tools can improve surgical outcomes by customizing the surgical plan for each patient. One example of such visualization tools is the BLUEPRINT® system, commercially available from Stryker Corp. Surgeons can use the BLUEPRINT® system to select, design, or modify appropriate implant components, to determine how to optimally position and orient implant components, how to shape the bone surface to receive the components, and to design, select, or modify surgical guide tools or instruments for performing the surgical plan. The information generated by the BLUEPRINT® system is compiled as a patient's preoperative surgical plan, which is stored in a database in an appropriate location, such as the memory system 106. This preoperative surgical plan can be accessed by surgeons or other healthcare providers, including before and during the actual surgery.

[0051]

[0057] The imaging system 116 may include one or more devices configured to generate medical image data. For example, the imaging system 116 may include a device that generates CT images. In some examples, the imaging system 116 may include a device that generates MRI images. Furthermore, in some examples, the imaging system 116 may include one or more computing devices configured to process data from the imaging devices to generate medical image data. For example, the medical image data may include 3D images of one or more bones of a patient. In this example, the imaging system 116 may include one or more computing devices configured to generate 3D images based on CT or MRI images.

[0052]

[0058] The computing system 102 may acquire a point cloud representing one or more patient anatomical structures (e.g., bones) of the patient. The point cloud may be generated based on medical image data generated by the imaging system 116. In some examples, the imaging system 116 may include one or more computing devices configured to generate the point cloud. The imaging system 116 or the computing system 102 may generate the point cloud by identifying the surface of one or more bones in the image and sampling points on the identified surface. Each point in the point cloud may correspond to a set of 3D coordinates of a point on the surface of the patient's bone. In other examples, the computing system 102 may include one or more computing devices configured to generate medical image data based on data from devices in the imaging system 116.

[0053]

[0059] The memory system 106 of the computing system 102 can store instructions that cause the computing system 102 to perform various operations when executed by the processing circuit 104. For example, in the example of Figure 1A, the memory system 106 can store instructions that cause the computing system 102 to perform operations related to the planning system 118 when executed by the processing circuit 104. For ease of explanation, rather than discussing the computing system 102 performing operations when the processing circuit 104 executes instructions, this disclosure may briefly refer to the planning system 118 or its components when performing such operations, or it may directly describe the computing system 102 when performing such operations.

[0054]

[0060] In the example in Figure 1A, the memory system 106 stores the surgical plan 120A. The surgical plan 120A may correspond to an individual patient. A patient-specific surgical plan may include data related to orthopedic surgeries planned or completed for the corresponding patient. A patient-specific surgical plan may include the patient's medical image data 126, point clouds 128, intermediate point clouds 130, pre-disease point clouds 132, and, in some examples, the patient's tool data (e.g., the type of tool required for the surgery). The medical image data 126 may include computed tomography (CT) images of the patient's anatomical structures, such as the patient's bones, or 3D images of the patient's anatomical structures based on CT images.

[0055]

[0061] The exemplary techniques described herein are explained in relation to bone, which is an example of a patient's anatomical structure. However, the exemplary techniques should not be considered to be limited to bone. In this disclosure, the term “bone” may refer to the whole bone or a bone fragment. Examples of bone include the tibia, fibula, scapula and humeral heads (e.g., shoulder), femur, patella (e.g., knee), vertebrae, or iliac crest, ilium, ischial spine, coccyx (e.g., hip).

[0056]

[0062] In some examples, medical image data 126 may include magnetic resonance imaging (MRI) images of one or more bones of the patient, or 3D images based on MRI images of one or more bones of the patient. In some examples, medical image data 126 may include ultrasound images of one or more bones of the patient. Point cloud 128 may include a point cloud representing the bones of the patient. In some examples, tool alignment data associated with the surgical plan 120A may include data representing one or more tool alignments for use in surgery.

[0057]

[0063] The planning system 118 may be configured to generate pre-symptomatic characterizations of damaged or affected patient anatomical structures (e.g., bones). For example, a patient may have a disease such as osteoarthritis, which can damage bones due to wear and tear on the joints between bones. Another example is a patient who has suffered trauma such as a fracture.

[0058]

[0064] A bone affected by disease or trauma is called pathological bone (e.g., damaged bone). A surgeon performs surgery to correct the damaged bone, such as by implanting a prosthesis or other surgery that can help the patient return the bone to its pre-damaged state. To perform the surgery, the surgeon may prepare one of the surgical plans 120A. One component of the surgical plan may be information describing the characteristics of the pathological bone before the injury. The surgeon may utilize such information about the pathological bone before the injury for the surgical plan (e.g., tool selection, prosthesis selection, how to perform the surgery, etc.) or as part of the surgery (e.g., by using an AR headset such as a visualization device 114 to view an overlay of the pre-damaged bone on top of the damaged bone).

[0059]

[0065] For example, point cloud 128 may represent a pathological condition of a patient's bone. Point cloud 128 may be referred to as pathological point cloud 128 or first point cloud 128 to indicate that point cloud 128 represents a pathological condition of a patient's bone. According to one or more techniques of this disclosure, the planning system 118 may be configured to generate information indicating at least one of the pathological or non-pathological portions of the first point cloud, as will be described in more detail. Although point cloud 128 represents a pathological condition of bone, not all bone can be damaged. For example, bone may have undeformed (e.g., non-pathological) bone portions and deformed (e.g., pathological) bone portions.

[0060]

[0066] In some cases, the point group 128 does not need to include the entire bone. For example, in ankle fractures, the distal end of the tibia near the ankle is almost always injured. In one or more cases, the planning system 118 assumes that the distal end of the tibia is injured and that this distal end is a pathological portion, and therefore may not include the distal end of the tibia as the portion that determines the pathological and non-pathological portions. Thus, in some cases, the point group 128 representing the pathological state of the patient's bone may be the point group of a pathological tibia, with the distal end of the tibia near the ankle removed from the tibia point group. In such cases, to generate information indicating at least one of the pathological or non-pathological portions of the point group 128, the planning system 118 may generate information indicating at least one of the pathological or non-pathological portions of the tibia point group 128 from which the distal end has been removed.

[0061]

[0067] The planning system 118 may be configured to identify which parts of the bone are pathological and which parts are non-pathological within the point cloud 128. The pathological parts of the first point cloud may be parts of the first point cloud corresponding to pathological conditions of the bone, and the non-pathological parts of the first point cloud may be parts of the first point cloud corresponding to non-pathological conditions of the bone.

[0062]

[0068] In other words, if the planning system 118 identifies only the pathological parts of the bone, the other parts may be considered non-pathological, and vice versa. The planning system 118 may be capable of identifying both the pathological and non-pathological parts of the bone.

[0063]

[0069] The planning system may generate an intermediate point group 130, also called a second point group 130, which includes points corresponding to the non-pathological parts of the bone pathology but does not include points corresponding to the pathological parts of the bone pathology. For example, the planning system 118 may remove parts of the first point group 128 that have been identified as pathological. This may result in an intermediate point group 130. That is, the planning system 118 may generate an intermediate point group 130 (for example, a second point group) which includes points corresponding to the non-pathological parts of the bone pathology but does not include points corresponding to the pathological parts of the bone pathology (for example, by removing the pathological parts of the first point group 128).

[0064]

[0070] In one or more examples, the planning system 118 may generate a third point cloud representing the pre-disease state of the bone based on the intermediate point cloud 130 (for example, as a second point cloud). For example, the third point cloud may represent the pre-disease state of the bone before disease or injury. The third point cloud representing the pre-disease state of the bone is the pre-disease point cloud 132.

[0065]

[0071] In one or more techniques of this disclosure, the planning system 118 may apply a point cloud neural network (PCNN) to generate an intermediate point cloud 130 and / or a pre-pathological point cloud 132. For example, point cloud 128 (e.g., a first point cloud) may be an input point cloud. The input point cloud represents pathological conditions of one or more bones of a patient, which include pathological and non-pathological portions. The intermediate point cloud 130 (e.g., a second point cloud) may be a first output point cloud. In some examples, the planning system 118 may apply a PCNN to point cloud 128 to determine information indicating at least one of the pathological and non-pathological portions, and generate an intermediate point cloud 130 that includes points corresponding to the non-pathological portions of the bone pathological condition but does not include points corresponding to the pathological portions of the bone pathological condition. For example, the planning system 118 may remove the pathological portions of point cloud 128 to generate the intermediate point cloud 130.

[0066]

[0072] As one example, the output from the PCNN that the planning system 118 applies to the point cloud 128 (e.g., pathological point cloud 128 or first point cloud 128) could be a label for each point in the point cloud 128. For example, the pathological point cloud 128 may have pathological and non-pathological parts. The pathological part of the pathological point cloud 128 may be the part of the pathological point cloud 128 corresponding to the pathological part of the pathological condition of the bone, and the non-pathological part of the pathological point cloud 128 may be the part of the pathological point cloud 128 corresponding to the non-pathological part of the pathological condition of the bone.

[0067]

[0073] Labels can classify each point in the point cloud 128 as either a pathological point or a non-pathological point. Pathological points indicate that they are located in a pathological region (e.g., of the point cloud 128), and non-pathological points indicate that they are located in a non-pathological region (e.g., of the point cloud 128). For example, the planning system 118 may generate labels indicating the pathological and non-pathological regions of the point cloud 128 by applying a PCNN.

[0068]

[0074] Next, the planning system 118 may use labels to determine which points to remove from the point cloud 128. For example, for each point labeled as a pathological point, the planning system 118 may remove that point from the point cloud 128. For each point labeled as a non-pathological point, the planning system 118 may leave that point in the point cloud 128. As a result of removing points from the point cloud 128, an intermediate point cloud 130 is obtained. For example, the intermediate point cloud 130 may include points corresponding to the non-pathological portion of the pathological state of the bone, but not points corresponding to the pathological portion of the pathological state of the bone.

[0069]

[0075] In some examples, the planning system 118 may apply a PCNN to the intermediate point cloud 130 to generate a pre-disease point cloud 132. In this example, the pre-disease point cloud 132 may be an output point cloud containing points that represent the characteristics of the pathological bone before the disease or injury (e.g., size, shape, etc.). That is, the planning system 118 may generate a pre-disease point cloud 132 (e.g., a third point cloud) representing the pre-disease state of the bone based on the intermediate point cloud 130.

[0070]

[0076] In the example shown in Figure 1A, system 100A includes a fabrication system 140. Fabrication system 100A may fabricate a patient-specific tool alignment guide configured to guide the tool to the patient's target bone along the tool alignment. The inclusion of fabrication system 140 is merely one example and should not be considered limiting. In some examples, fabrication system 140 may fabricate a tool alignment guide based on a representation of pre-disease anatomical structures. For example, planning system 118 may utilize a pre-disease point cluster 132 to generate a graphic representation of the pre-disease bone or to generate information indicating the size and dimensions of the pre-disease bone. Fabrication system 140 may utilize such information to fabricate a desired tool or tool alignment guide.

[0071]

[0077] The manufacturing system 140 may include an additional manufacturing device (e.g., a 3D printer) configured to generate a patient-specific tool alignment guide. In an example where the tool alignment corresponds to the cutting surface of a vibratory saw, the patient-specific tool alignment guide may define a slot for the vibratory saw. When the patient-specific tool alignment guide is correctly positioned in the patient's bone, the slot aligns with the determined tool alignment. Thus, the surgeon can use the vibratory saw with the determined tool alignment by inserting the vibratory saw into the slot of the patient-specific tool alignment guide. In an example where the tool alignment corresponds to a drill axis or pin insertion axis, the patient-specific tool alignment guide may define a channel for a drill bit or pin. When the patient-specific tool alignment guide is correctly positioned in the patient's bone, the channel aligns with the determined tool alignment. Thus, the surgeon can drill a hole or insert a pin by inserting the drill bit or pin into the channel of the patient-specific tool alignment guide.

[0072]

[0078] Figure 1B is a block diagram of another exemplary system 100B that may be used to implement the techniques of the present disclosure. Various components of Figure 1B having the same reference numerals as Figure 1A may be considered to be the same or substantially the same, and no further description of Figure 1B is given.

[0073]

[0079] The system 100B in Figure 1B includes a surgical plan 120B. The surgical plan 120B may be a surgical plan for revision surgery. As mentioned above, revision surgery is a surgery in which the current prosthesis is removed and replaced with another prosthesis. There can be various reasons for revision surgery, including changes in the disease state, migration of the prosthesis, or reaching the practical lifespan of the prosthesis. In some examples, the planning system 118 may be configured to generate a representation of the damaged or affected bone at the time of the initial surgery in which the prosthesis was implanted. That is, rather than generating a representation of the pre-disease bone, or in addition to generating it, the planning system 118 may be configured to determine a representation of the diseased bone at the time of the initial surgery in which the prosthesis was implanted.

[0074]

[0080] As shown in the figure, the surgical plan 120B includes a point cloud 142. One example of a point cloud 142 may be a point cloud representing the anatomical structure of a patient with a prosthesis implanted during the initial surgery (e.g., the patient's current anatomical structure). The planning system 118 may acquire the point cloud 142 in the same way that the planning system 118 acquired the point cloud 128.

[0075]

[0081] The planning system 118 may generate a preoperative point cloud 144 representing the patient's anatomical structure at the time of the first surgery. For example, the planning system 118 may apply a PCNN to the point cloud 142 to generate the preoperative point cloud 144.

[0076]

[0082] In revision surgery, the implanted prosthesis is already present. Therefore, in some cases, the point group 142 may include a representation of the prosthesis. In one or more cases, the planning system 118 may be configured to generate the preoperative point group 144 without utilizing the portion of the point group 142 that contains the prosthesis.

[0077]

[0083] There are various ways in which the planning system 118 can determine the portion of the point cloud 142 that contains the prosthesis. For example, the prosthesis may appear in the medical image data 126 as relatively high-luminance image content. The planning system 118 may remove the image content with luminance higher than a threshold and generate the point cloud 142 based on the resulting image data. As another example, the planning system 118 may utilize another PCNN trained to distinguish between the prosthesis and bone. The planning system 118 may remove the portion identified as the prosthesis by this PCNN in order to generate the point cloud 142.

[0078]

[0084] Revision surgery can be performed on various bone regions to which a prosthesis can be implanted. For example, the prosthesis may be placed in one of the following: the tibia, fibula, scapula, humeral head, femur, patella, vertebra, or ilium.

[0079]

[0085] Figure 2 is a block diagram showing exemplary components of a planning system 118 according to one or more techniques of the present disclosure. In the example of Figure 2, the components of the planning system 118 include a PCNN 200, a prediction unit 202, a training unit 204, and a reconstruction unit 206. In other examples, the planning system 118 may be implemented using more, fewer, or different components. For example, the training unit 204 may be omitted if the PCNN 200 has already been trained. In some examples, one or more components of the planning system 118 are implemented as software modules. Furthermore, the components in Figure 2 are presented as examples, and the planning system 118 may be implemented in other ways.

[0080]

[0086] There may be other examples of PCNN200. For example, as mentioned above, in some examples, the planning system 118 may apply a PCNN to the point cloud 128 to generate information indicating at least one of the pathological or non-pathological parts of the point cloud 128. The first example of PCNN200 (or simply called the first PCNN200) may be trained to identify at least one of the pathological or non-pathological parts.

[0081]

[0087] In some examples, the planning system 118 may apply a PCNN to an intermediate point cloud 130 to generate a point cloud representing the pre-disease state of the bone (for example, to generate a pre-disease point cloud 132). A second example of PCNN200 (or simply called the second PCNN200) may be trained to generate a pre-disease point cloud 132 based on the intermediate point cloud 130.

[0082]

[0088] For example, the second PCNN200 may be configured to determine a non-pathological estimate of the pathological portion of the bone pathology and to combine the non-pathological estimate of the pathological portion with the intermediate point group 130 to generate a pre-disease point group 132. For example, to combine the non-pathological estimate of the pathological portion with the intermediate point group 130, the second PCNN may fill the intermediate point group 130 with the non-pathological estimate of the pathological portion.

[0083]

[0089] As described above, non-pathological estimation of pathological portions refers to the estimation of non-pathological anatomical structures (e.g., bone) that fill in the pathological portions removed from the pathological point cluster 128 in order to generate the intermediate point cluster 130. For example, non-pathological estimation of pathological portions completes the intermediate point cluster 130 so that there are no longer any gaps in the intermediate point cluster 130 due to the removal of the pathological portions. In one or more examples, non-pathological estimation is called "estimation" because the second PCNN 200 may be configured to fill in the pathological portions, which the second PCNN 200 has determined to be representative of the pathological portions, but with non-pathological anatomical structures (e.g., non-pathological bone). Estimation of non-pathological bone may be all that is available, as the bone may be damaged and there may be no image data of the bone before the damage.

[0084]

[0090] A second PCNN for determining the non-pathological estimation of pathological parts and filling in the intermediate point cluster 130 is presented as an exemplary technique. In some examples, the second PCNN may be configured to directly determine the pre-disease point cluster 132 from the intermediate point cluster 130.

[0085]

[0091] In revision surgery, the planning system 118 may apply a PCNN to the point cloud 142 to generate a preoperative point cloud 144. A third example of the PCNN 200 (or simply called the third PCNN 200) may be trained to determine the characteristics of preoperative anatomical structures, the input being a point cloud representing the anatomical structures of a patient with a prosthesis implanted during the initial surgery.

[0086]

[0092] The exemplary techniques do not require the use of both the first and second PCNN200. In some examples, the planning system 118 may use only the first PCNN200 instead of the second PCNN200. In some examples, the planning system 118 may use only the second PCNN200 and not the first PCNN200. In some examples, the planning system 118 may use both the first and second PCNN200. In some examples, the use of a third PCNN is not required (e.g., revision surgery), and the planning system 118 may use some other technique to generate the preoperative point cloud 144 (e.g., a point cloud representing the patient's anatomical structure at the time of the initial surgery).

[0087]

[0093] The prediction unit 202 may apply a PCNN 200 to generate an output point cloud based on an input point cloud. In a first PCNN 200, the input point cloud (e.g., point cloud 128) represents one or more bones of a patient (e.g., pathological bones), and the output point cloud (e.g., intermediate point cloud 130) includes the points from point cloud 128 but with the pathological parts removed. That is, the intermediate point cloud 130 includes points corresponding to the non-pathological parts of the pathological state of the bone, but does not include points corresponding to the pathological parts of the pathological state of the bone. In some examples, the output point cloud of the first PCNN 200 may be considered as point cloud 128 with a label indicating whether each point is a pathological or non-pathological point, in which case the planning system 118 removes the pathological points from this output point cloud to generate the intermediate point cloud 130. In some examples, simply for the sake of explanation, there may be an earlier output point cloud labeled indicating whether the points in point cloud 128 are pathological or non-pathological, in which case the output point cloud of the first PCNN200 is described as the intermediate point cloud 130, by understanding that the pathological points are removed to generate the intermediate point cloud 130.

[0088]

[0094] The prediction unit 202 may acquire the point cloud 128 by one of several methods. For example, the prediction unit 202 may generate the point cloud 128 based on medical image data 126. The patient's medical image data may include multiple input images (e.g., CT images or MRI images). In this example, each input image may have width and height dimensions, and each input image may correspond to a different depth dimension layer within multiple depth dimension layers. In other words, the multiple input images may be conceptualized as a stack of 2D images, and the position of each individual 2D image in the stack corresponds to the depth dimension. As part of generating the point cloud 128, the prediction unit 202 may perform an edge detection algorithm (e.g., Canny edge detection, phase stretch transform (PST), etc.) on the 2D images (or 3D images based on 2D images).

[0089]

[0095] The prediction unit 202 may select points on detected edges as points in the point cloud. In other examples, the prediction unit 202 may obtain the point cloud 128 from one or more devices outside the computing system 102. In some examples, such as the use of a second PCNN 200, the prediction unit 202 may receive an intermediate point cloud 130 from components within the planning system 118.

[0090]

[0096] As shown above, the output point cloud (for example, the intermediate point cloud 130 for the first PCNN200) may include non-pathological points from point cloud 128 and exclude pathological points from point cloud 128. As one example, the planning system 118 may use information indicating the pathological and non-pathological parts generated by the first PCNN200 to generate the intermediate point cloud 130 and remove the pathological parts.

[0091]

[0097] The output point cloud (for example, the pre-disease point cloud 132 for the second PCNN200) may represent the pre-disease bone. The representation of the pre-disease bone may be the bone as it appeared before the disease or trauma, and can be considered a combination of the non-pathological portion of the bone included in the intermediate point cloud 130 and a non-pathological estimation of the pathological portion (for example, how the pathological portion looked before the disease or trauma). For example, the output point cloud from the second PCNN200 may be the intermediate point cloud 130, and the removed pathological portion is filled in with a non-pathological estimation of the removed pathological portion. However, this filling with non-pathological estimation should not be considered limiting, and it may be possible to generate the pre-disease point cloud 132 directly from the intermediate point cloud 130.

[0092]

[0098] In revision surgery, the prediction unit 202 may acquire a point cloud 142 similar to the point cloud 128 described above. The output point cloud (e.g., preoperative point cloud 144 for the third PCNN 200) may represent the preoperative bone (e.g., the bone of a patient with a prosthesis implanted during the initial surgery at the time of the initial surgery).

[0093]

[0099] The first PCNN200, the second PCNN200, and the third PCNN are implemented using a point cloud learning model-based architecture. A point cloud learning model-based architecture (e.g., a point cloud learning model) is a neural network-based architecture that takes one or more point clouds as input and produces one or more point clouds as output. Exemplary point cloud learning models include PointNet and PointTransformer. An exemplary point cloud learning model-based architecture based on PointNet is described below with reference to Figure 3.

[0094]

[0100] As mentioned above, in some examples, there may be a first PCNN200, a second PCNN200, and a third PCNN200. In some examples, there may be different sets of the first and second PCNN200s, based on which bones require pre-disease characterization, for example. Similarly, there may be different sets of the third PCNN200, based on which bones require revision surgery.

[0095]

[0101] For example, a set of PCNNs for total ankle replacement surgery might include a first PCNN and a second PCNN200 for ankle surgery, which generate an output point set containing points representing the ankle before the onset of the disease. Similarly, there might be a first PCNN and a second PCNN200 for surgery on the tibia, fibula, scapula and humeral heads (e.g., shoulder), vertebrae, patella (e.g., knee), or iliac crest, ilium, ischial spine, coccyx (e.g., hip). For revision surgery, there might be a third PCNN200 set for replacing an ankle prosthesis and another third PCNN200 set for replacing a shoulder prosthesis. Generally, the prosthesis could be for the tibia, fibula, scapula, humeral head, femur, patella, vertebrae, or ilium, but this technique is not so limited.

[0096]

[0102] The training unit 204 can train a PCNN200. For example, the training unit 204 can generate groups of training datasets. A first group of training sets may be for training a first PCNN200. A second group of training sets may be for training a second PCNN200. A third group of training sets may be for training a third PCNN200.

[0097]

[0103] Each training dataset may correspond to an individual past patient among several past patients. Past patients may include patients with pathological bones and patients with non-pathological bones.

[0098]

[0104] A training dataset for past patients may include training input data and predicted output data. For a first PCNN200 (for example, one for generating information indicating at least one of pathological or non-pathological parts in a point cloud 128), the training input data may include a point cloud representing pathological conditions of bone, and the predicted output data may include labels for each point in the point cloud 128 indicating whether the point cloud is a pathological point belonging to a pathological part or a non-pathological point belonging to a non-pathological part.

[0099]

[0105] For a second PCNN200 for generating a pre-disease point cluster 132 (for example, by combining a non-pathological estimate of the pathological portion with an intermediate point cluster 130), the training input data may include a point cluster representing a subset of points of non-pathological bone, and the predicted output data may be non-pathological bone. For a third PCNN200 (for example, for revision surgery), the training input data may include a point cluster of patients determined to undergo revision surgery, and the predicted output data may be bone at the time of the initial surgery in which the prosthesis was implanted.

[0100]

[0106] In some examples, the training unit 204 may generate training input data based on medical imaging data stored in surgical plans for past patients. In some examples, the training unit 204 may generate predicted output data for the first PCNN 200 based on information from past patients about what was determined to be pathological and what was determined to be non-pathological. For example, a surgeon may indicate in a point cloud of a past patient which parts should be considered pathological and which should be considered non-pathological. Other methods may exist for training the first PCNN 200 to generate information indicating pathological and non-pathological parts in the first point cloud 128.

[0101]

[0107] In some examples, the training unit 204 for the second PCNN may generate predicted output data based on information from past patients with non-pathological bone. For example, the input could be non-pathological bone, including portions removed from past patients or other possible non-patient individuals who volunteer for the study. The predicted output data could be a point cloud of the non-pathological bone of these individuals.

[0102]

[0108] In some examples, the training unit 204 for the third PCNN may generate predicted output data based on information from past patients. For example, the training unit 204 could receive a point cloud of past patients at the time of their initial surgery and use this past patient data as predicted output data for patients, in which case the input would be the point cloud of these past patients at the time they were about to undergo revision surgery. In another example, the training unit 204 could receive information from the surgeon who made the decision about what the preoperative bone would look like given image data of a patient undergoing revision surgery.

[0103]

[0109] The training unit 204 may train the first PCNN200, the second PCNN200, and the third PCNN200 based on each group of training datasets. Since the training unit 204 generates training datasets based on how real surgeons actually planned and / or performed past patient surgeries, or based on point clouds of non-pathological bone, a surgeon who ultimately uses the recommendations generated by the planning system 118 can be confident that those recommendations are based on how other real surgeons determined pathological and non-pathological parts, or what real non-pathological bone looks like, or what real preoperative anatomical structures look like.

[0104]

[0110] In some examples, as part of training the first, second, and third PCNN200s, training unit 204 may perform a forward pass to PCNN200 by using the input point cloud of the training dataset as input to PCNN200. Next, training unit 204 may perform a process that compares the resulting output point cloud generated by PCNN200 with the corresponding expected output point cloud. In other words, training unit 204 may use a loss function to calculate a loss value based on the output point cloud generated by PCNN200 and the corresponding expected output point cloud. In some examples, the loss function aims to minimize the difference between the output point cloud generated by PCNN200 and the corresponding expected output point cloud. Examples of loss functions may include Chamfer Distance (CD) and Earth Mover's Distance (EMD). CD may be given by the average of the first mean and the second mean. The first mean is the average distance between each point in the output point cloud generated by PCNN200 and its nearest point in the predicted output point cloud. The second mean is the average distance between each point in the predicted output point cloud and its nearest point in the output point cloud generated by PCNN200. CD can be defined as follows:

[0105]

number

[0106] In the above equation, S1 is the output point set generated by PCNN200, S2 is the predicted output point set, |..| is an element indicating the number of elements, and ||..|| indicates the absolute value.

[0107]

[0111] For example, in the first PCNN200, S1 may be the intermediate point group 130, or it may be the labeled point group indicating whether a point is pathological or non-pathological. S2 may be the predicted point group determined by the surgeon or other means. In the second PCNN200, S1 may be the pre-pathological point group 132. S2 may be the predicted point group of the pre-pathological bone. In the third PCNN200, S1 may be the pre-operative point group 144. S2 may be the predicted point group of the bone representation at the time of the first surgery.

[0108]

[0112] Next, the training unit 204 may perform backpropagation based on the loss value to tune the parameters of the PCNN200 (for example, the neuronal weights of the PCNN200). In some examples, the training unit 204 may determine the average loss value based on the loss value calculated from the output point cloud generated by performing multiple forward passes through the PCNN200 using different input point clouds from the training data. In such examples, the training unit 204 may perform backpropagation using the average loss value to tune the parameters of the PCNN200. The training unit 204 may repeat this process over multiple training epochs.

[0109]

[0113] During the use of PCNN200 (for example, after training PCNN200), the prediction unit 202 of the planning system 118 may apply PCNN200 to generate an output point cloud of the patient based on an input point cloud representing one or more bones of the patient. In some examples, the reconstruction unit 206 may be configured to use the point cloud 132 to generate one or more images of the pre-pathogenic bone (for example, a reconstructed version of the pathological bone), or to generate information such as the size and dimensions of the pre-pathogenic bone. In some examples, the reconstruction unit 206 may be configured to use the preoperative point cloud 144 to generate one or more images of the bone at the time of the first surgery.

[0110]

[0114] In some examples, the reconstruction unit 206 may output one or more images (e.g., one or more 2D or 3D images) or a model showing the pre-disease bone for display. For example, the reconstruction unit 206 may use points in the pre-disease point cloud 132 or the pre-operative point cloud 144 as vertices of a polygon, which forms the outer layer of the pre-disease bone. The reconstruction unit 206 may output an image showing the pre-disease bone for display against a model of one or more diseased bones of the patient, or display an image showing the bone at the time of the first surgery in which the prosthesis was implanted. In some such examples, the output point cloud generated by the PCNN200 (e.g., the pre-disease point cloud 132 or the pre-operative point cloud 144) and the input point cloud (e.g., point cloud 128 or point cloud 142) are in the same coordinate system.

[0111]

[0115] In some examples, the reconstruction unit 206 may generate an MR visualization showing the pre-disease bone based on the pre-disease point group 132. In revision surgery, the reconstruction unit 206 may generate an MR visualization showing the bone at the time of the initial surgery based on the pre-operative point group 144. In examples where the visualization device 114 (Figures 1A and 1B) is an MR visualization device, the visualization device 114 may display an MR visualization. In some examples, the visualization device 114 may display an MR visualization during the planning phase of the surgery. In such examples, the reconstruction unit 206 may generate the MR visualization in space as a 3D image. The reconstruction unit 206 may generate a 3D image in the same manner as described above in order to generate a 3D image.

[0112]

[0116] In some cases, the MR visualization is intraoperative MR visualization. In other words, the visualization device 114 can display MR visualizations during surgery. In some cases, the visualization device 114 can perform an overlay process, superimposing the MR visualization onto the patient's physical bone. Thus, in such cases, the surgeon wearing the visualization device 114 can view the pre-pathogenic bone in comparison to the patient's pathological bone, or, for revision surgery, view the bone at the time of the initial surgery in comparison to the patient's current bone.

[0113]

[0117] Figure 3 is a conceptual diagram showing an exemplary point cloud learning model 300 using one or more techniques of the present disclosure. The point cloud learning model 300 may accept an input point cloud. The input point cloud is a collection of points. The points in the collection are not necessarily arranged in any particular order. Therefore, the input point cloud may have an unstructured representation.

[0114]

[0118] In the example in Figure 3, the point cloud learning model 300 includes an encoder network 301 and a decoder network 302. The encoder network 301 receives an array 303 of n points. The points in array 303 can be the input point cloud of the point cloud learning model 300. In the example in Figure 3, each point in array 303 has a dimension of 3. For example, in a Cartesian coordinate system, each point may have an x-coordinate, a y-coordinate, and a z-coordinate.

[0115]

[0119] The encoder network 301 may apply an input transformation 304 to the points of array 303 to generate array 305. Next, the encoder network 301 may use a first shared multilayer perceptron (MLP) 306 to map each of the n points of array 305 from 3 dimensions to a larger dimension a (for example, a=64 in the example of Figure 3), thereby generating an array 307 of n × a (for example, n × 64 values). For ease of explanation, the following explanation for Figure 3 assumes that a is equal to 64, but other values ​​of a may be used in other examples. Next, the encoder network 301 may apply a feature transformation 308 to the values ​​of array 307 to generate an array 309 of n × 64 values. For each of the n points in array 309, the encoder network 301 uses a second shared MLP 310 to map the n points from dimension a to dimension b (for example, b = 1024 in the example in Figure 3), thereby generating an array 311 of n × b (for example, n × 1024 values). For ease of explanation, the following description of Figure 3 assumes that b is equal to 1024, but other values ​​of b may be used in other examples. The encoder network 301 applies a max pooling layer 312 to generate a global feature vector 313. In the example in Figure 3, each of the n points in the global feature vector 313 has 1024 dimensions.

[0116]

[0120] Therefore, as part of applying PCNN200, the computing system 102 may apply an input transformation (e.g., input transformation 304) to the first array (e.g., array 303) comprising a point cloud to generate a second array (e.g., array 305), where the input transformation is implemented using a first T-Net model (e.g., T-Net model 326), and may apply a first MLP (e.g., MLP306) to the second array to generate a third array (e.g., array 307), a fourth array (... For example, a feature transformation (e.g., feature transformation 308) may be applied to a third array to generate array 309), where the input transformation is implemented using a second T-Net model (e.g., T-Net model 330); a second MLP (e.g., MLP 310) may be applied to a fourth array to generate a fifth array (e.g., array 311); and a max pooling layer (e.g., max pooling layer 312) may be applied to a fifth array to generate a global feature vector (e.g., global feature vector 313).

[0117]

[0121] The fully connected network 314 can map the global feature vector 313 to k output classification scores, where k is an integer representing the number of classes. Each output classification score corresponds to a separate class. An output classification score corresponding to a class can indicate the confidence that the input point cloud as a whole corresponds to that class. The fully connected network 314 includes a neural network having two or more neuron layers, where each neuron in one layer is connected to each neuron in a subsequent layer. In the example in Figure 3, the fully connected network 314 includes an input layer with 512 neurons, a hidden layer with 256 neurons, and an output layer with k neurons. In some examples, the fully connected network 314 may be omitted from the encoder network 301.

[0118]

[0122] In some examples, the input 316 to the decoder network 302 may be formed by concatenating n 64-dimensional points of array 309 with the global feature vector 313. In other words, for each of the n points in array 309, the corresponding 64 dimensions of that point are concatenated with 1024 features of the global feature vector 313. In some examples, array 309 is not concatenated with the global feature vector 313.

[0119]

[0123] The decoder network 302 can sample N points within a 2D unit square. Thus, the decoder network 302 can randomly determine N points having x-coordinates in the range [0,1] and y-coordinates in the range [0,1]. For each of the N points, the decoder network 302 can obtain its respective input vector by concatenating each point with the global feature vector 313. Thus, in the example where array 309 is not concatenated with the global feature vector 313, each input vector may have 1026 features. For each respective input vector, the decoder network 302 can apply each of the K MLPs 318 (where K is an integer greater than or equal to 1) to the respective input vector. Each MLP 318 may correspond to a separate patch (e.g., a region) of the output point cloud. When the decoder 302 applies an MLP to an input vector, the MLP can generate a 3D point in the patch (e.g., a region) corresponding to the MLP. Thus, each MLP 318 can reduce the number of features from 1026 to 3. The three features can correspond to the three coordinates of a single point in the output point cloud. For example, for every n sampled points out of N, the MLP 318 can reduce the number of features from 1026 to 512, 256, 128, 64, and 3. Thus, the decoder network 302 can generate a K×N×3 vector containing the output point cloud 320. In some examples, K=16 and N=512, resulting in a second point cloud containing 8192 3D points. In other examples, other values ​​for K and N may be used. In some examples, as part of training the MLP in the decoder network 302, the decoder network 302 may calculate the chamfer loss of the output point cloud relative to the ground truth point cloud. The decoder network 302 may use the chamfer loss in the backpropagation process to tune the parameters of the MLP. In this way, the planning system 118 may apply a decoder (for example, a decoder network 302) to generate an intermediate point cloud 130 and a pre-disease point cloud 132 for the example in Figure 1A, or a pre-operative point cloud 144 for the example in Figure 1B.

[0120]

[0124] In some examples, the MLP318 may include a series of four fully connected neuron layers. For each of the MLP318, the decoder network 302 may pass an input vector of 1026 features to the input layer of the MLP. The fully connected layers can reduce the number of features from 1026 to 512, 256, and 3.

[0121]

[0125] The input transformation 304 and feature transformation 308 of the encoder network 301 may provide transformation invariance. In other words, the point cloud learning model 300 may be able to generate the output point cloud (e.g., intermediate point cloud 130, pre-disease point cloud 132, and / or pre-operative point cloud 144) in the same way regardless of how the input point cloud (e.g., input bone model) is rotated, scaled, or translated. The fact that the point cloud learning model 300 provides transformation invariance may be advantageous because it may reduce the susceptibility of the generator ML model to errors based on placement / scale in the pathological bone model. As shown in the example in Figure 3, the input transformation 304 may be implemented using a T-Net model 326 and a matrix multiplication operation 328. The T-Net model 326 generates a 3x3 transformation matrix based on array 303. The matrix multiplication operation 328 multiplies array 303 by the 3x3 transformation matrix. Similarly, the feature transformation 308 can be implemented using the T-Net model 330 and the matrix multiplication operation 332. The T-Net model 330 can generate a 64x64 transformation matrix based on the array 307. The matrix multiplication operation 328 multiplies the array 307 by the 64x64 transformation matrix.

[0122]

[0126] Figure 4 is a block diagram showing an exemplary architecture of T-Net Model 400 using one or more techniques of the present disclosure. T-Net Model 400 may implement T-Net Model 326 used in input transformation 304. In the example of Figure 4, T-Net Model 400 receives array 402 as input. Array 402 contains n points. Each of these points has a dimension of 3. A first shared MLP maps each of the n points in array 402 from 3 dimensions to 64 dimensions, thereby generating array 404. A second shared MLP maps each of the n points in array 404 from 64 dimensions to 128 dimensions, thereby generating array 406. A third shared MLP maps each of the n points in array 406 from 128 dimensions to 1024 dimensions, thereby generating array 408. T-Net Model 400 then applies a max pooling operation to array 408, resulting in an array 810 with 1024 values. A first fully connected neural network maps array 410 to array 812 with 512 values. A second fully connected neural network maps array 412 to array 414 with 256 values. The T-Net model 400 applies a matrix multiplication operation 416 to the trainable weight matrix 418. The trainable weight matrix 418 has dimensions of 256 × 9. Therefore, multiplying array 414 by the trainable weight matrix 418 yields array 820 of size 1 × 9. The T-Net model 400 can then add a trainable bias 422 to the values ​​in array 420. A reconstruction operation 424 can remap the values ​​obtained by adding the trainable bias 422 to a 3 × 3 transformation matrix. In other examples, the size of the matrix and the size of the array may differ.

[0123]

[0127] The T-Net model 330 (Figure 3) can be implemented similarly to the T-Net model 400 to perform feature transformation 308. However, in this example, the trainable weight matrix 418 is 256 × 4096, and the trainable bias 422 has bias values ​​of 1 × 4096 instead of 9. Therefore, the T-Net model for performing feature transformation 308 may generate a transformation matrix of size 64 × 64. In other examples, the size of the matrix and the size of the array may differ.

[0124]

[0128] Figure 5 is a flowchart illustrating an exemplary process for pre-pathological characterization of patient anatomical structures using one or more techniques of the present disclosure. A computing system 102 (for example, a processing circuit 104 implementing a planning system 118) may acquire a first point cloud representing the pathological condition of the patient's bones (500). For example, the first point cloud may be point cloud 128. As described above, the computing system 102 may utilize medical image data 126 to generate point cloud 128. Point cloud 128 may include pathological anatomical structures, such as one or more bones. Examples of bones include at least one of the following: tibia, fibula, scapula, humeral head, femur, patella, vertebra, iliac crest, ilium, ischial spine, or coccyx.

[0125]

[0129] The computing system 102 can generate information indicating at least one of the pathological or non-pathological portions of the first point cloud (502). The pathological portion of the first point cloud (e.g., point cloud 128) may be a portion of the first point cloud corresponding to a pathological portion of bone in a pathological state, and the non-pathological portion of the first point cloud may be a portion of the first point cloud corresponding to a non-pathological portion of bone in a pathological state. For example, the computing system 102 can generate information indicating at least one of the pathological or non-pathological portions of the first point cloud by applying a PCNN (e.g., a first PCNN 200) to the first point cloud. For example, the first PCNN may be trained to identify at least one of the pathological or non-pathological portions.

[0126]

[0130] In some examples, in order to generate information indicating at least one of the pathological or non-pathological parts of a first point cloud, the computing system 102 may label each point in the first point cloud as either a pathological or non-pathological point, based on applying a first PCNN to the first point cloud. In one or more examples, pathological points indicate that they are in a pathological part, and non-pathological points indicate that they are in a non-pathological part.

[0127]

[0131] Furthermore, in some examples, the first point group (e.g., point group 128) does not necessarily have to include all of the pathological bone. For example, the first point group may represent the point group of a pathological tibia, with the distal end of the tibia near the ankle removed from the point group 128 of the pathological tibia. In such examples, to generate information indicating at least one of the pathological portion or the non-pathological portion of the first point group, the computing system 102 may be configured to generate information indicating at least one of the pathological portion or the non-pathological portion of the point group of the pathological tibia with the distal end removed.

[0128]

[0132] The use of the first PCNN may not be necessary in all cases. Other methods may exist to generate information indicating at least one of the pathological and non-pathological parts. For example, a surgeon may be able to distinguish between pathological and non-pathological parts.

[0129]

[0133] As another example, the computing system 102 may compare a first point cloud with a statistical shape model (SSM) to generate information indicating pathological and non-pathological parts. For example, for an SSM, the computing system 102 may obtain a point cloud of the SSM, which is a representative model of the pre-pathological state of an anatomical structure. The processing circuit may orient the point cloud of the SSM, or the point cloud representing the pathological state of the anatomical structure, so that the SSM point cloud and the point cloud representing the pathological state of the anatomical structure are oriented in the same direction. The computing system 102 may determine non-pathological points within the point cloud representing the pathological state of the anatomical structure. For example, as mentioned above, the point cloud representing the pathological state of the anatomical structure may contain pathological and non-pathological parts. The computing system 102 may deform the point cloud of the SSM until the points within the SSM point cloud overlap with identified non-pathological points. The computing system 102 can determine the differences between the superimposed SSM and the point cloud representing the pathological state of the anatomical structure. The resulting differences may represent pathological areas.

[0130]

[0134] The computing system 102 may generate a second point cloud that includes points corresponding to the non-pathological parts of the pathological bone but does not include points corresponding to the pathological parts of the pathological bone (504). For example, the computing system 102 may generate a second point cloud from which the pathological parts (e.g., of the first point cloud) have been removed. One example of the second point cloud is the intermediate point cloud 130. As an example, the computing system 102 may utilize labels indicating pathological and non-pathological points in the point cloud 128. The computing system 102 may remove points in the point cloud 128 that are labeled as pathological points and leave the points in the point cloud 128 that are labeled as non-pathological points. The result of removing the pathological points may be the intermediate point cloud 130, which includes points corresponding to the non-pathological parts of the pathological bone but does not include points corresponding to the pathological parts of the pathological bone.

[0131]

[0135] The computing system 102 may generate a third point cloud representing the pre-disease state of the bone based on the second point cloud (506). As one example, the computing system 102 may apply a second PCNN configured to directly generate the third point cloud based on the second point cloud.

[0132]

[0136] As another example, in order to generate a third point cloud based on a second point cloud, the computing system 102 may use a second PCNN to determine non-pathological estimates of pathological parts of bone in a pathological state, and then combine the non-pathological estimates of pathological parts with the second point cloud to generate a third point cloud. For example, in order to combine the non-pathological estimates of pathological parts with the second point cloud using a second PCNN, the computing system 102 may fill the second point cloud with non-pathological estimates of pathological parts.

[0133]

[0137] As described above, non-pathological estimation of pathological parts refers to the estimation of non-pathological anatomical structures (e.g., bone) that fill in the pathological parts that have been removed from the first point cloud in order to generate the second point cloud. In other words, non-pathological estimation of pathological parts completes the point cloud so that there are no longer any gaps in the point cloud due to the removal of pathological parts. As a result of completing the point cloud by estimating non-pathological anatomical structures, a reconstruction of pathological bone is obtained, which is the pre-disease characterization of pathological bone.

[0134]

[0138] As another example, computing system 102 may utilize SSM to generate a third point cloud based on a second point cloud. For example, computing system 102 may utilize PCNN to remove the pathological portion after identifying at least one of the pathological or non-pathological portion, as described above. Computing system 102 may utilize the non-pathological portion to drive the fitting of the SSM represented in the point cloud. For example, computing system 102 may orient the non-pathological portion to have the same orientation as the SSM. Computing system 102 may deform the point cloud of the SSM (e.g., stretch, shrink, rotate, translate, etc.) until the points in the point cloud of the SSM overlap with the identified non-pathological points. That is, computing system 102 may determine a first deformed SSM (e.g., by stretching, shrinking, rotating, translating, etc.) and determine the distance between the corresponding points in the first deformed SSM and the points of the non-pathological portion. The computing system 102 can repeat this operation for different deformed SSMs. The computing system 102 can identify versions of the SSM that overlap with identified non-pathological points (for example, versions of the SSM that have points that best match the corresponding points of the non-pathological points). The resulting versions of the SSM may be a third point group representing the pre-pathological state of the bone.

[0135]

[0139] The computing system 102 may output information showing a third point cloud representing the pre-disease state of the bone (508). As one example, the reconstruction unit 206 may generate a graphic representation of the pre-disease point cloud 132, which a surgeon may view using a visualization device 114, as one example.

[0136]

[0140] Figure 6 is a flowchart illustrating an exemplary process for preoperative characterization of a patient's anatomical structure for revision surgery. A computing system 102 (for example, a processing circuit 104 implementing a planning system 118) may acquire a first point cloud (600) representing the bones of a patient with a prosthesis implanted during the initial surgery. For example, in revision surgery, the current prosthesis may be replaced, and therefore the computing system 102 may acquire a point cloud 142 representing the anatomical structure of a patient with a prosthesis. In some examples, the prosthesis is for one of the following: tibia, fibula, scapula, humeral head, femur, patella, vertebra, or ilium.

[0137]

[0141] The computing system 102 may generate a second point cloud representing the patient's anatomical structure at the time of the first surgery, based on the first point cloud (602). For example, the computing system 102 may generate a preoperative point cloud 144 by applying a third PCNN (as described above, for example with respect to Figure 2) to the point cloud 142. In some examples, the computing system 102 may generate the preoperative point cloud 144 without using the portion of the point cloud 142 that contains the prosthesis.

[0138]

[0142] The computing system 102 may output information indicating a second point cloud (604). As one example, the reconstruction unit 206 may generate a graphic representation of the preoperative point cloud 144, which a surgeon may view using a visualization device 114, as one example.

[0139]

[0143] Although the techniques have been disclosed in terms of a limited number of examples, a number of modifications and variations will be understood by those skilled in the art who benefit from this disclosure. For example, it is intended that any reasonable combination of the examples described may be implemented. The appended claims are intended to encompass modifications and variations that fall within the true spirit and scope of the invention.

[0140]

[0144] It should be noted that, in some cases, some of the operations or events of the techniques described herein may be performed in a different order, and may be added, merged, or completely excluded (for example, not all operations or events described may be necessary for the practice of the techniques). Furthermore, in some cases, the operations or events may be performed not sequentially, but simultaneously, for example, through multithreading, interrupt handling, or by multiple processors.

[0141]

[0145] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes in or transmitted through a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any media that facilitates the transfer of computer programs from one location to another, for example, by a communication protocol. Thus, the computer-readable medium may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes and / or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

[0142]

[0146] Such computer-readable storage media may include, but are not limited to, computer-readable storage media, RAM, ROM, EEPROM®, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is appropriately called computer-readable media. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology such as infrared, radio, or microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary, tangible storage media. As used herein, the terms "disk" and "disc" include compact discs (CDs), laserdiscs (registered trademark), optical discs, digital multipurpose discs (DVDs), floppy disks (registered trademark), and Blu-ray discs, where a "disk" typically reproduces data magnetically, while a "disc" reproduces data optically using a laser. Any combination of the above should also be included in the scope of computer-readable media.

[0143]

[0147] The operations described herein may be performed by one or more processors, which may be implemented as fixed-function processing circuits, programmable circuits, or combinations thereof, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. A fixed-function circuit is a circuit that provides a specific function and is preset to an operation that can be performed. A programmable circuit is a circuit that can be programmed to perform a variety of tasks and to provide adaptive functions in an operation that can be performed. For example, a programmable circuit may execute instructions specified by software or firmware to operate the programmable circuit as defined by software or firmware instructions. A fixed-function circuit may execute software instructions (for example, to receive or output parameters), but the types of operations performed by a fixed-function circuit are generally immutable. Accordingly, the terms “processor” and “processing circuit” as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. The following is a direct reproduction of the claims as originally filed. [C1] A method for pre-symptomatic characterization of patient anatomical structures, wherein the method is The first point cloud representing the pathological condition of the patient's bones is obtained by a computing system, The computing system generates information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to the pathological portion of the bone in the pathological state, and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to the non-pathological portion of the bone in the pathological state. The computing system generates a second point cloud which includes points corresponding to the non-pathological portion of the pathological state of the bone, but does not include points corresponding to the pathological portion of the pathological state of the bone. The computing system generates a third point cloud representing the pre-disease state of the bone based on the second point cloud, The computing system outputs information representing the third point cloud that indicates the pre-disease state of the bone. A method that includes [a certain feature]. [C2] The generation of the third point cloud is To determine the non-pathological estimation of the pathological portion of the said pathological state of the bone, To generate the third point cloud, the non-pathological estimation of the pathological portion and the second point cloud are combined. A method of C1 comprising: [C3] The method of C2, wherein combining the non-pathological estimation of the pathological portion with the second point cloud comprises filling the second point cloud with the non-pathological estimation of the pathological portion. [C4] The method according to any one of C1 to C3, wherein generating information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud is based on applying a point cloud neural network (PCNN) to the first point cloud, wherein the PCNN is trained to identify at least one of the pathological portion or the non-pathological portion. [C5] The method according to any one of C1 to C3, wherein generating the third point cloud representing the pre-disease state of the bone is performed by generating the third point cloud based on applying a point cloud neural network (PCNN) to the second point cloud. [C6] Generating information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud comprises generating information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud based on applying a first point cloud neural network (PCNN) to the first point cloud, wherein the first PCNN is trained to identify at least one of the pathological portion or the non-pathological portion. The method according to any one of C1 to C3, wherein generating the third point cloud representing the pre-disease state of the bone is performed by generating the third point cloud based on applying a second PCNN to the second point cloud. [C7] The method according to any one of C1 to C6, wherein the bone comprises at least one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, iliac crest, ilium, ischial spine, or coccyx. [C8] The method according to any one of C1 to C7, comprising: generating information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud, based on applying a point cloud neural network (PCNN) to the first point cloud, and labeling each point in the first point cloud as either a pathological point or a non-pathological point, wherein the pathological point indicates that it is in a pathological portion and the non-pathological point indicates that it is in a non-pathological portion. [C9] The method according to any one of C1 to C8, wherein the first point group representing the pathological condition of the bone of the patient comprises a point group of pathological tibia, and a point in the point group representing the distal end of the tibia near the ankle is removed from the point group of pathological tibia, and generating information indicating at least one of the pathological portion of the first point group or the non-pathological portion of the first point group generates information indicating at least one of the pathological portion of the point group or the non-pathological portion of the point group of pathological tibia from which the distal end has been removed. [C10] A method for preoperatively characterizing a patient's anatomical structure for revision surgery, wherein the method is The first point cloud representing the bone of a patient with a prosthesis implanted during the initial surgery is obtained by a computing system, Based on the first point cloud, the computing system generates a second point cloud representing the patient's bone at the time of the first surgery, The computing system outputs information representing the second point cloud. A method that includes [a certain feature]. [C11] The method according to C10, wherein generating the second point cloud is performed by applying a point cloud neural network (PCNN) to the first point cloud. [C12] The method according to C10 or 11, wherein generating the second point cloud is performed without using the portion of the first point cloud that includes the prosthesis. [C13] The method according to any of C10-12, wherein the prosthesis is for one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, or ilium. [C14] A memory system configured to store a first point cloud representing the pathological state of a patient's bones, The system comprises a processing circuit, and the processing circuit Obtaining the first point cloud representing the pathological condition of the bone of the patient, The method involves generating information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to the pathological portion of the bone in the pathological state, and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to the non-pathological portion of the bone in the pathological state. To generate a second point group that includes points corresponding to the non-pathological portion of the pathological state of the bone, but does not include points corresponding to the pathological portion of the pathological state of the bone, Based on the second point cloud, a third point cloud representing the pre-disease state of the bone is generated, Outputting information representing the third point cloud that shows the pre-disease state of the bone. A system configured to perform the following actions. [C15] In order to generate the third point cloud, the processing circuit, To determine the non-pathological estimation of the pathological portion of the said pathological state of the bone, To generate the third point cloud, the non-pathological estimation of the pathological portion and the second point cloud are combined. The system described in C14 is configured to perform the following actions. [C16] The method according to C15, wherein the processing circuit is configured to fill the second point cloud with the non-pathological estimation of the pathological portion in order to combine the non-pathological estimation of the pathological portion with the second point cloud. [C17] The system according to any one of C14 to C16, wherein the processing circuit is configured to generate information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud, based on applying a point cloud neural network (PCNN) to the first point cloud, wherein the PCNN is trained to identify at least one of the pathological portion or the non-pathological portion. [C18] The system according to any one of C14 to C16, wherein the processing circuit is configured to generate the third point cloud representing the pre-disease state of the bone by applying a point cloud neural network (PCNN) to the second point cloud. [C19] To generate information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud, the processing circuit is configured to generate information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud based on applying a first point cloud neural network (PCNN) to the first point cloud, wherein the first PCNN is trained to identify at least one of the pathological portion or the non-pathological portion. The system according to any one of C14 to C16, wherein the processing circuit is configured to generate the third point cloud representing the pre-disease state of the bone by applying a second PCNN to the second point cloud. [C20] The system according to any one of C14 to C19, wherein the bone comprises at least one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, iliac crest, ilium, ischial spine, or coccyx. [C21] The system according to any one of C14 to 20, wherein the processing circuit is configured to label each point in the first point cloud as either a pathological point or a non-pathological point, based on applying a point cloud neural network (PCNN) to the first point cloud, in order to generate information indicating at least one of the pathological portion or the non-pathological portion of the first point cloud, the processing circuit is configured to apply a point cloud neural network (PCNN) to the first point cloud, the processing circuit is configured to label each point in the first point cloud as either a pathological point or a non-pathological point, the processing circuit is configured to apply a point cloud neural network (PCNN) to apply a point cloud neural network (PCNN) [C22] The system according to any one of C14 to 21, wherein the first point group representing the pathological condition of the bone of the patient comprises a point group of pathological tibia, and points in the point group representing the distal end of the tibia near the ankle are removed from the point group of the pathological tibia, and the processing circuit is configured to generate information indicating at least one of the pathological portion of the point group or the non-pathological portion of the point group of the pathological tibia from which the distal end has been removed, in order to generate information indicating at least one of the pathological portion of the first point group or the non-pathological portion of the point group of the pathological tibia from which the distal end has been removed. [C23] A memory system configured to store a first point cloud representing the bone of a patient with a prosthesis implanted during the initial surgery, It comprises a processing circuit, and the processing circuit is Obtaining the first point cloud representing the bone of the patient having the prosthesis implanted during the first surgery, Based on the first point cloud, a second point cloud is generated representing the patient's bone at the time of the first surgery, Outputting information representing the second point cloud A system configured to perform the following actions. [C24] The system according to C23, wherein, in order to generate the second point cloud, the processing circuit is configured to generate the second point cloud based on applying a point cloud neural network (PCNN) to the first point cloud. [C25] The system according to C23 or 24, wherein the processing circuit is configured to generate the second point cloud without using the portion of the first point cloud that includes the prothesis. [C26] The system according to any one of C23-25, wherein the prosthesis is for one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, or ilium. [C27] A system comprising means for carrying out any of the methods described in C1-9 or C10-13. [C28] A computer-readable storage medium that stores instructions that, when executed, cause one or more processors to perform any of the methods described in C1-9 or C10-13.

Claims

1. A method for pre-symptomatic characterization of patient anatomical structures, wherein the method is The first point cloud representing the pathological state of the patient's bones is obtained by a computing system, The computing system generates information indicating the pathological portion and the non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to the pathological portion of the bone in the pathological state, and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to the non-pathological portion of the bone in the pathological state. The computing system generates a second point cloud that includes points corresponding to the non-pathological portion of the pathological state of the bone, but does not include points corresponding to the pathological portion of the pathological state of the bone. The computing system generates a third point cloud representing the pre-disease state of the bone based on the second point cloud; the generation of the third point cloud determines a non-pathological estimation of the pathological portion of the pathological state of the bone; and the non-pathological estimation of the pathological portion and the second point cloud are combined in order to generate the third point cloud. The computing system outputs information representing the third point cloud that indicates the pre-disease state of the bone. A method that includes [a certain feature].

2. The method according to claim 1, wherein combining the non-pathological estimation of the pathological portion with the second point cloud comprises filling the second point cloud with the non-pathological estimation of the pathological portion.

3. The method according to any one of claims 1 to 2, wherein generating information indicating the pathological portion and the non-pathological portion of the first point cloud is based on applying a point cloud neural network (PCNN) to the first point cloud, the PCNN being trained to identify at least one of the pathological portion or the non-pathological portion.

4. The method according to any one of claims 1 to 2, wherein generating the third point cloud representing the pre-disease state of the bone is performed by generating the third point cloud based on applying a point cloud neural network (PCNN) to the second point cloud.

5. Generating information indicating the pathological portion and the non-pathological portion of the first point cloud comprises generating information indicating the pathological portion and the non-pathological portion of the first point cloud based on applying a first point cloud neural network (PCNN) to the first point cloud, wherein the first PCNN is trained to identify at least one of the pathological portion or the non-pathological portion. The method according to any one of claims 1 to 2, wherein generating the third point cloud representing the pre-disease state of the bone is performed by generating the third point cloud based on applying a second PCNN to the second point cloud.

6. The method according to any one of claims 1 to 2, wherein the bone comprises at least one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, iliac crest, ilium, ischial spine, or coccyx.

7. The method according to any one of claims 1 to 2, comprising: generating information indicating the pathological portion and the non-pathological portion of the first point cloud, based on applying a point cloud neural network (PCNN) to the first point cloud, labeling each point in the first point cloud as either a pathological point or a non-pathological point, wherein the pathological point indicates that it is in a pathological portion and the non-pathological point indicates that it is in a non-pathological portion.

8. The method according to any one of claims 1 to 2, wherein the first point group representing the pathological condition of the bone of the patient comprises a point group of pathological tibia, and a point in the point group representing the distal end of the tibia near the ankle is removed from the point group of pathological tibia, and generating information indicating the pathological portion of the first point group and the non-pathological portion of the first point group comprises generating information indicating at least one of the pathological portion of the point group or the non-pathological portion of the point group of pathological tibia from which the distal end has been removed.

9. A memory system configured to store a first point cloud representing the pathological state of a patient's bones, The system comprises a processing circuit, and the processing circuit To obtain the first point cloud representing the pathological condition of the bone of the patient, The method involves generating information indicating the pathological portion and the non-pathological portion of the first point cloud, wherein the pathological portion of the first point cloud is the portion of the first point cloud corresponding to the pathological portion of the bone in the pathological state, and the non-pathological portion of the first point cloud is the portion of the first point cloud corresponding to the non-pathological portion of the bone in the pathological state. To generate a second point group that includes points corresponding to the non-pathological portion of the pathological state of the bone, but does not include points corresponding to the pathological portion of the pathological state of the bone, Based on the second point cloud, a third point cloud representing the pre-disease state of the bone is generated; and in order to generate the third point cloud, the processing circuit is configured to determine a non-pathological estimation of the pathological portion of the pathological state of the bone, and to combine the non-pathological estimation of the pathological portion with the second point cloud in order to generate the third point cloud. Outputting information representing the third point cloud that shows the pre-disease state of the bone. A system configured to perform the following actions.

10. The system according to claim 9, wherein the processing circuit is configured to fill the second point cloud with the non-pathological estimation of the pathological portion in order to combine the non-pathological estimation of the pathological portion with the second point cloud.

11. The system according to any one of claims 9 to 10, wherein the processing circuit is configured to generate information indicating the pathological portion and the non-pathological portion of the first point cloud by applying a point cloud neural network (PCNN) to the first point cloud, wherein the PCNN is trained to identify at least one of the pathological portion or the non-pathological portion.

12. The system according to any one of claims 9 to 10, wherein the processing circuit is configured to generate the third point cloud representing the pre-disease state of the bone by applying a point cloud neural network (PCNN) to the second point cloud.

13. In order to generate information indicating the pathological portion and the non-pathological portion of the first point cloud, the processing circuit is configured to generate information indicating the pathological portion and the non-pathological portion of the first point cloud by applying a first point cloud neural network (PCNN) to the first point cloud, wherein the first PCNN is trained to identify at least one of the pathological portion or the non-pathological portion. The system according to any one of claims 9 to 10, wherein the processing circuit is configured to generate the third point cloud representing the pre-disease state of the bone by applying a second PCNN to the second point cloud.

14. The system according to any one of claims 9 to 10, wherein the bone comprises at least one of the tibia, fibula, scapula, humeral head, femur, patella, vertebra, iliac crest, ilium, ischial spine, or coccyx.

15. The system according to any one of claims 9 to 10, wherein, in order to generate information indicating the pathological portion and the non-pathological portion of the first point cloud, the processing circuit is configured to label each point in the first point cloud as either a pathological point or a non-pathological point, based on applying a point cloud neural network (PCNN) to the first point cloud, such that the pathological points indicate that they are in a pathological portion and the non-pathological points indicate that they are in a non-pathological portion.

16. The system according to any one of claims 9 to 10, wherein the first point group representing the pathological condition of the bone of the patient comprises a point group of pathological tibia, and points in the point group representing the distal end of the tibia near the ankle are removed from the point group of pathological tibia, and the processing circuit is configured to generate information indicating at least one of the pathological portion of the point group or the non-pathological portion of the point group of pathological tibia from which the distal end has been removed, in order to generate information indicating the pathological portion of the first point group and the non-pathological portion of the first point group.

17. A computer-readable storage medium that stores instructions causing one or more processors to perform the method according to any one of claims 1 to 2 when executed.

Images