System and method for automatically generating shoulder surgery recommendations for patient

Abstract

The invention relates to a system and method for automatically generating shoulder surgery recommendations for a patient. A system for automatically generating shoulder surgery recommendations for a patient, the system comprising: a memory configured to store patient-specific image data for the patient; and processing circuitry configured to: receive patient-specific image data from the memory; determining, based on the patient-specific image data, one or more soft tissue features and a bone mineral density metric associated with the patient's humerus, where the one or more soft tissue features include at least one of a fat penetration value or an atrophy rate of the one or more soft tissue structures of the patient; generating a recommendation for a type of shoulder surgery to be performed by the patient based on at least one of a fat penetration value or an atrophy rate of the one or more soft tissue structures of the patient; generating a recommendation of a humeral implant type for the patient based on the humeral-related bone mineral density metric; and outputting suggestions for the patient of the shoulder surgery type and the humeral implant type.

Description

[0001] This application is a divisional application of the patent application with the national application number 201980091666.3 and the invention title "Soft Tissue Modeling and Planning System for Orthopaedic Surgical Procedures", which entered the national stage on August 9, 2021, from the international application with the international application date of December 11, 2019 and the international application number PCT / US2019 / 065788, the entire content of which is incorporated herein by reference. Technical Field

[0002] The present disclosure relates to surgical joint repair, and more particularly to systems and methods for automatically generating shoulder surgery recommendations for a patient. Background Art

[0003] Surgical joint repair procedures involve the repair and / or replacement of damaged or diseased joints. Often times, surgical joint repair procedures (as an example, such as arthroplasty, etc.) involve replacing a damaged joint with a prosthesis that is implanted into the patient's bone. Appropriately selecting a prosthesis with the appropriate size and shape and appropriately positioning the prosthesis to ensure optimal surgical outcomes is challenging. To assist with positioning, surgical procedures often involve using surgical instruments to control the shaping of the surface of the damaged bone and to control the cutting or drilling of the bone to receive the prosthesis.

[0004] Today, surgeons can use visualization tools, and surgeons use three-dimensional modeling of bone shapes to facilitate preoperative planning for joint repair and replacement. These tools can help surgeons design and / or select surgical guides and implants that closely match the patient's anatomy, and can improve surgical outcomes by customizing surgical plans for each patient. Summary of the Invention

[0005] The present disclosure describes various systems, devices, and techniques for providing patient analysis, preoperative planning, and / or training and education for surgical joint repair procedures. For example, the systems described herein can determine soft tissue (e.g., muscle, connective tissue, adipose tissue, etc.) dimensions and / or characteristics from patient imaging data. Characteristics of the soft tissue can include fat infiltration, atrophy rate, range of motion, or other similar characteristics. The system can use these soft tissue dimensions / characteristics to determine range of motion values for one or more joints of the patient. Based on these range of motion values, the system can recommend one or more types of surgical interventions that may be suitable for treating one or more health conditions of one or more joints. In one embodiment, the system can use soft tissue dimensions to determine whether an anatomical shoulder replacement surgery or a reverse shoulder replacement surgery is appropriate for a particular patient.

[0006] Alternatively, the system described herein can also determine a bone density metric for at least a portion of the patient's humeral head based on the patient-specific image data of the patient. For example, the bone density metric can be the overall density of the humeral head or a single indication of a portion of the humeral head. As another example, the bone density metric can include bone density values for various portions of the patient's humeral head. In fact, the bone density metric may not indicate the density of the bone, but rather can be a metric representing bone density (e.g., voxel intensity from the image data, standard deviation of voxel intensity from the image data, compressibility, etc.). The system can control the user interface to present a graphical representation of the bone density metric and / or generate a recommendation regarding the type of implant for the humeral head based on the bone density metric. For example, a bone density metric indicating sufficient trabecular bone density in the humeral head can cause the system to recommend a stemless humeral implant as opposed to a stemmed humeral implant.

[0007] In one embodiment, a system for modeling a patient's soft tissue structure includes: a memory configured to store patient-specific image data of the patient; and a processing circuit configured to: receive the patient-specific image data; determine a patient-specific shape representing the patient's soft tissue structure based on the intensity of the patient-specific image data; and output the patient-specific shape.

[0008] In another embodiment, a method for modeling a patient's soft tissue structure includes: storing patient-specific image data of the patient in a memory; receiving the patient-specific image data by a processing circuit; determining, by the processing circuit, a patient-specific shape representing the patient's soft tissue structure based on the intensity of the patient-specific image data; and outputting, by the processing circuit, the patient-specific shape.

[0009] In another embodiment, a computer-readable storage medium includes instructions that, when executed by a processing circuit, cause the processing circuit to: store patient-specific image data of the patient in a memory; receive the patient-specific image data; determine a patient-specific shape representing the patient's soft tissue structure based on the intensity of the patient-specific image data; and output the patient-specific shape.

[0010] In another embodiment, a system for modeling a patient's soft tissue structure includes: means for storing patient-specific image data of the patient; means for receiving the patient-specific image data; means for determining a patient-specific shape representing the patient's soft tissue structure based on the intensity of the patient-specific image data; and means for outputting the patient-specific shape.

[0011] Details of various embodiments of the present disclosure are set forth in the accompanying drawings and the following description. From the specification, drawings, and claims, various features, objects, and advantages will become apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of a plastic surgery system according to an embodiment of the present disclosure.

[0013] Figure 2 is a block diagram of a plastic surgery system including a mixed reality (MR) system according to an embodiment of the present disclosure.

[0014] Figure 3 is a flowchart showing exemplary stages of a surgical lifecycle.

[0015] Figure 4 is a flowchart showing preoperative, intraoperative, and postoperative workflows supporting a plastic surgery procedure.

[0016] Figure 5A and Figure 5B are diagrams of exemplary muscles and bones related to a patient's shoulder.

[0017] Figure 6 is a block diagram showing exemplary components of a system configured to determine soft tissue structure dimensions and / or features and other information related to surgical interventions associated with a joint according to an embodiment of the present disclosure.

[0018] Figure 7 is a diagram of an exemplary insertion point of a muscle of the rotator cuff;

[0019] Figure 8A is a conceptual diagram of exemplary patient-specific image data.

[0020] Figure 8B is based on Figure 8A a conceptual diagram of a Hessian feature image generated from patient-specific image data.

[0021] Figure 8C is a conceptual diagram of an exemplary initial shape and an exemplary contour superimposed on patient-specific image data.

[0022] Figure 9 is a conceptual diagram of an exemplary procedure for changing an initial shape towards a patient-specific shape representing a patient's soft tissue structure.

[0023] Figure 10 is a conceptual diagram of an exemplary procedure for changing an intermediate shape towards a patient-specific shape representing a patient's soft tissue structure.

[0024] Figure 11 is a conceptual diagram of an exemplary patient-specific shape representing a patient's soft tissue structure compared to an actual contour in patient-specific image data.

[0025] Figure 12It is a conceptual diagram showing an exemplary initial shape of the scapular muscles and a patient-specific shape superimposed on patient-specific image data.

[0026] Figure 13 It is a conceptual diagram showing an exemplary initial shape of the supraspinatus muscle and a patient-specific shape superimposed on patient-specific image data.

[0027] Figure 14 It is a conceptual axial view showing an exemplary final patient-specific shape of the rotator cuff muscles superimposed on patient-specific image data.

[0028] Figure 15 It is a conceptual sagittal view showing an exemplary final patient-specific shape of the rotator cuff muscles superimposed on patient-specific image data.

[0029] Figure 16A It is a conceptual posterior three-dimensional view showing an exemplary final patient-specific shape of the rotator cuff muscles and the bones from patient-specific image data.

[0030] Figure 16B It is a conceptual anterior three-dimensional view showing an exemplary final patient-specific shape of the rotator cuff muscles and the bones from patient-specific image data.

[0031] Figure 17 It is a conceptual end-face three-dimensional view showing an exemplary final patient-specific shape of the rotator cuff muscles and the bones from patient-specific image data.

[0032] Figure 18A and Figure 18B It is a conceptual diagram of exemplary patient-specific CT data where the initial shape associated with the soft tissue structure is registered with the bone structure and modified to represent the patient-specific shape of the soft tissue structure.

[0033] Figure 19 It is a conceptual diagram of an exemplary final patient-specific shape that is masked and thresholded to determine soft tissue characteristics such as fat infiltration.

[0034] Figure 20 It is a conceptual diagram of an exemplary final patient-specific shape of the soft tissue structure and a pre-disease assessment.

[0035] Figure 21 and Figure 22 It is a conceptual diagram of an exemplary spring-modeled muscle that contributes to the range of motion analysis of the shoulder joint.

[0036] Figure 23A It is a flowchart showing an exemplary process of using patient-specific image data to model soft tissue structures according to the technology of the present disclosure.

[0037] Figure 23B is a flowchart showing another exemplary process for modeling soft tissue structures using patient-specific image data according to the techniques of the present disclosure.

[0038] Figure 24 is a flowchart showing an exemplary process for modeling soft tissue structures using patient-specific image data according to the techniques of the present disclosure.

[0039] Figure 25 is a flowchart showing an exemplary process for determining the fat infiltration value of a patient's soft tissue structure according to the techniques of the present disclosure.

[0040] Figure 26 is a flowchart showing an exemplary process for determining the atrophy rate of a patient's soft tissue structure according to the techniques of the present disclosure.

[0041] Figure 27 is a flowchart showing an exemplary process for determining the type of shoulder treatment based on the determined soft tissue structure of a patient according to the techniques of the present disclosure.

[0042] Figure 28 is a flowchart showing an exemplary process for determining the type of shoulder treatment based on patient-specific image data according to the techniques of the present disclosure.

[0043] Figure 29 is a block diagram showing an exemplary computing system implementing a deep neural network (DNN) that can be used to determine one or more aspects of patient anatomy, diagnosis, and / or treatment recommendations according to the techniques of the present disclosure.

[0044] Figure 30 shows what can be done by Figure 29 the exemplary computing system in

[0045] Figure 31 is a flowchart showing an exemplary operation of a computing system for determining the type of shoulder surgery recommendation for a patient using a DNN according to the techniques of the present disclosure.

[0046] Figure 32 is a diagram showing an exemplary bone related to a patient's shoulder.

[0047] Figure 33A 、 Figure 33B 、and Figure 33C is a conceptual diagram of an exemplary humerus prepared for a humeral implant.

[0048] Figure 34 is a conceptual diagram of an exemplary humeral implant.

[0049] Figure 35 is a conceptual diagram of an exemplary stemmed humeral implant.

[0050] Figure 36 is a conceptual diagram of an exemplary stemless humeral implant implanted on the humeral head.

[0051] Figure 37 is a conceptual diagram of an exemplary reverse humeral implant.

[0052] Figure 38 is a block diagram showing exemplary components of a system configured to determine bone density from patient-specific image data in accordance with an embodiment of the present disclosure.

[0053] Figure 39A is a flowchart showing an exemplary process for determining the type of humeral implant based on bone density.

[0054] Figure 39B is a flowchart showing an exemplary process for applying a neural network to patient-specific image data to determine the stem size of a humeral implant.

[0055] Figure 39C is a flowchart showing an exemplary process for determining recommendations for shoulder treatment based on soft tissue structure and bone density determined from patient-specific image data.

[0056] Figure 40 is a flowchart showing an exemplary process for displaying bone density information.

[0057] Figure 41 is a conceptual diagram of an exemplary user interface including a femoral head and a cutting plane.

[0058] Figure 42 is a conceptual diagram of an exemplary user interface including a femoral head and a representation of internal bone density.

[0059] Figure 43 is a conceptual diagram of an exemplary user interface including a humeral head and a representation of internal bone density associated with a type recommendation for a humeral implant. Detailed Description

[0060] The present disclosure describes various systems, devices, and techniques for providing patient analysis, preoperative planning, and / or training and education for surgical joint repair procedures. Orthopedic surgery can involve implanting one or more prosthetic devices to repair or replace a patient's damaged or diseased joint. Virtual surgical planning tools use image data of the diseased or damaged joint to generate an accurate three-dimensional bone model that can be viewed and manipulated preoperatively by a surgeon. These tools can enhance surgical outcomes by allowing the surgeon to simulate the surgery, select or design implants that more closely match the contours of the patient's actual bone, and select or design surgical instruments and guidance tools that are particularly suited to repair the bones of a specific patient.

[0061] These planning tools can be used to generate a preoperative surgical plan and to complete the plan with implants and surgical instruments selected or manufactured for an individual patient. These systems can rely on the patient's bone model to determine the procedure for an individual patient and / or the type of a specific implant. However, soft tissue structure information (e.g., muscle and / or connective tissue) derived from the patient's imaging data is not available. Without this imaging data of the patient's soft tissue, the planning tools and clinicians can determine certain aspects of the surgery or implant without the help of how the patient's soft tissue affects the current joint and the postoperative joint.

[0062] As described herein, a system can determine soft tissue (e.g., muscle, tendon, ligament, cartilage, and / or connective tissue) dimensions and other characteristics from patient imaging data. Accordingly, the system can be configured to use these soft tissue dimensions and / or other characteristics derived from the patient imaging data to select or recommend certain types of medical interventions, types of surgical treatments, or even types, dimensions, and / or placement of one or more medical implants. Thus, the system can use the soft tissue information derived from the patient imaging data to determine or assist in determining a surgical plan for a specific patient. For example, the system can select between an anatomical shoulder replacement surgery or a reverse shoulder replacement surgery and then output the options to a user such as a surgeon, for example, via a presentation on a display, based on the soft tissue dimensions and other characteristics derived from the patient imaging data. These recommendations for shoulder replacement described herein can apply to a new replacement or first replacement of the shoulder, or in other embodiments, can apply to a revision surgery in which the patient has already had a shoulder replacement. Generally, shoulder surgery can be used to restore shoulder function and / or reduce a patient's pain.

[0063] In one embodiment, the system can receive patient imaging data (e.g., computerized tomography (CT) including X-ray images, magnetic resonance imaging (MRI) images, or other imaging modalities) and construct a three-dimensional (3D) imaging dataset. From this imaging dataset, the system can identify the positions of bones associated with soft tissue structures of interest and the approximate positions of the soft tissue structures themselves. For example, if a patient may require a shoulder replacement surgery, the system can identify portions of the scapula and humerus and the muscles of the rotator cuff. For each soft tissue structure (e.g., for each muscle of the rotator cuff), the system can determine a representation of the soft tissue structure from the imaging data. The system can place an initial shape within the estimated position of the soft tissue structure and then fit the initial shape to the imaging data to determine the representation of the actual soft tissue structure. The estimated position can be based on one or more markers or landmarks (e.g., muscle insertion points or muscle origins) on the associated bone or other bony structures or portions of bony structures. The initial shape can be a statistical mean shape (SMS) derived from multiple subjects or any geometric shape.

[0064] For the initial shape, the system can use vectors orthogonal to the surface of the initial shape to identify voxels outside or inside the initial shape and exceeding an intensity threshold that represents the boundary of the soft tissue structure within the imaging data. In some embodiments, the boundary of the soft tissue structure can be estimated from the separation zones identified between adjacent soft tissue structures. For each vector, from the corresponding position on the initial shape, the system can move the surface of the initial shape towards the corresponding voxel of the identified voxel. This movement of the surface of the initial shape may occur for several iterations until the initial shape is modified to approximate the contour of the identified voxel. In other embodiments, the system can use the correspondence from the initial shape to the associated bone and / or a minimization or maximization algorithm (e.g., a cost function) to fit and scale the initial shape to the patient-specific image data. Then, the finally modified shape can be used as the representation of a soft tissue structure such as the muscles of the patient's rotator cuff.

[0065] The system can determine one or more features of one or more soft tissue structures from the determined representation. For example, the system can employ these features for surgical planning. In some embodiments, the system can calculate the volume of the soft tissue structure used for other determinations. Within the representation of the soft tissue structure, the system can determine the fat infiltration value of the soft tissue structure based on the thresholded intensity of the voxels from the imaging data, i.e., compare the voxel intensity and the threshold intensity value to characterize the voxel as a representation of adipose tissue. For example, voxels within the segmented representation of the soft tissue structure having an intensity value exceeding the threshold can be considered as a representation of adipose tissue rather than muscle tissue. Voxels not exceeding the threshold intensity value or groups of two or more voxels can be considered as representing non-adipose tissue. As used herein, the term "exceeding" can refer to a value greater than or less than the threshold.

[0066] The ratio of adipose tissue within a representation to total tissue (including adipose and non - adipose tissue) can be determined as the fat infiltration value. The system can also calculate an atrophy rate by dividing the volume of the SMS fitted to the patient's bone (e.g., the pre - morbid tissue volume estimated for the patient) by the volume of the representation of the soft tissue structure. Then, the system can determine the spring constant (or some other representation of muscle function) of the soft tissue structure and other soft tissue structures associated with the joint to determine the range of motion of the joint. Then, the system can determine the type of intervention for the joint based on the range of motion and / or other features discussed herein. For example, the system can determine whether an anatomical shoulder replacement or a reverse shoulder replacement is a more appropriate treatment for the patient.

[0067] In some embodiments, the system can determine bone density characteristics of the humeral head of the humerus based on patient - specific image data (e.g., 2D or 3D image data). For example, the system can characterize bone density values assigned to voxels or combinations of voxels of trabecular bone within at least a portion of the humeral head. In other embodiments, the system can determine an overall bone density metric or score indicative of the total volume of trabecular bone in at least a portion of the humeral head. The system can control a display device to display a user interface that includes a representation of bone density, such as a graphical indication of bone density. In some embodiments, the system can generate a recommendation for the type of humeral implant (e.g., stemmed or stemless) based on the determined bone density. In some embodiments, the recommendation for the type of humeral implant can be based on historical surgical data for humeral implants, where the system associates the type of humeral implant used for a patient with the bone density values identified in the patient - specific image data for that same patient.

[0068] As an example embodiment herein, shoulder replacement surgery is described. However, the systems, devices, and techniques described herein can be used to analyze other anatomical structures or groups of structures of a patient, determine the type of treatment for other joints of the patient (e.g., elbow, hip, knee, etc.), or select a certain type of implant for a patient's specific anatomical health condition. Additionally, in other embodiments, the techniques described herein for determining soft tissue structures from patient imaging data can be used to identify other structures such as bones.

[0069] In some embodiments, a system, apparatus, and method may employ a mixed reality (MR) visualization system to assist in creating, implementing, validating, and / or modifying a surgical plan before and during a surgical procedure, such as associated processes for determining why a type of treatment is provided to a patient (e.g., joint replacement surgery such as shoulder replacement). Because MR, or in some instances VR, can be used to interact with the surgical plan, the present disclosure may also refer to the surgical plan as a “virtual” surgical plan. In accordance with the techniques of the present disclosure, a mixed reality visualization system and visualization tools may be used together or visualization tools other than the mixed reality visualization system may be used.

[0070] For example, a surgical plan or recommendation generated by the system BLUEPRINT TM system, available from Wright Medical Limited, or another surgical planning platform may include information defining various features of a surgical procedure, such as a recommended step of a surgical treatment (e.g., an anatomical or reverse shoulder surgery) of a specific surgical procedure step to be performed on a patient by a surgeon in accordance with the surgical plan (e.g., including bone or tissue preparation steps and / or steps for selecting, modifying, or replacing an implant component). In various embodiments, the information may include the dimensions, shape, angle, surface profile, and / or orientation of an implant component selected or modified by a surgeon, the dimensions, shape, angle, surface profile, and / or orientation of bone or soft tissue defined by a surgeon in a bone or tissue preparation step, and / or the position, axis, plane, angle, and / or entry point defining the placement of an implant component relative to a patient's bone or other tissue. Information such as dimensions, shape, angle, surface profile, and / or orientation of a patient's anatomical features may be derived from imaging (e.g., x-ray, CT, MRI, or ultrasound or other images), direct observation, or other techniques.

[0071] Some visualization tools utilize patient image data to generate a three-dimensional model of a bone contour for use in preoperative planning for joint repair and replacement. These tools may allow a surgeon to design and / or select surgical guides and implant components that closely match a patient's anatomy. These tools are capable of improving surgical outcomes by customizing a surgical plan for each patient. An example of such a visualization tool for shoulder repair is the BLUEPRINT TM system. The BLUEPRINT TM system provides a two-dimensional plan view of a bone repair region and a three-dimensional virtual model of the repair region to a surgeon. The surgeon is able to use the BLUEPRINT TMa system to select, design, or modify appropriate implant components, determine how best to position and orient the implant components and how to shape the surface of the bone to receive the components, and design, select, or modify a surgical guidance tool or instrument to complete the surgical plan. By BLUEPRINT TM The information generated by the TM system can be compiled in the patient's pre-operative surgical plan (including before and during the actual surgery) at an appropriate location stored in a database (e.g., on a server of a wide area network, local area network, or global network), where the surgeon or other care provider can access the appropriate location in the database.

[0072] Specific embodiments of the present disclosure are described with reference to the accompanying drawings, in which like reference numerals represent like elements. However, it should be understood that the accompanying drawings only illustrate the various implementations described herein and are not intended to limit the scope of the various techniques described herein. The drawings illustrate and describe various embodiments of the present disclosure.

[0073] In the following description, numerous details are set forth to provide an understanding of the present disclosure. However, those skilled in the art should understand that one or more aspects of the present disclosure may be practiced without these details, and various variations or modifications of the described embodiments are possible.

[0074] Figure 1 is a block diagram of an orthopedic surgery system 100 according to an embodiment of the present disclosure. The orthopedic surgery system 100 includes a set of subsystems. In Figure 1 an embodiment, the subsystems include a virtual planning system 102, a planning support system 104, a manufacturing and delivery system 106, an intraoperative guidance system 108, a medical education system 110, a monitoring system 112, a predictive analytics system 114, and a communication network 116. In other embodiments, the orthopedic surgery system 100 may include more, fewer, or different subsystems. For example, the orthopedic surgery system 100 may omit the medical education system 110, the monitoring system 112, the predictive analytics system 114, and / or other subsystems. In some embodiments, the orthopedic surgery system 100 may be used for surgical tracking, in which case, the orthopedic surgery system 100 may be referred to as a surgical tracking system. In other cases, the orthopedic surgery system 100 may generally be referred to as a medical device system.

[0075] Users of the orthopedic surgery system 100 can use the virtual planning system 102 to plan orthopedic surgeries. As discussed herein, for example, the virtual planning system 102 and / or another surgical planning system can analyze patient imaging data (e.g., bones and / or soft tissues) and determine a recommended surgical treatment based on bone and / or soft tissue features determined from the imaging data. Users of the orthopedic surgery system 100 can use the planning support system 104 to review surgical plans generated using the orthopedic surgery system 100. The manufacturing and delivery system 106 can assist in manufacturing and delivering items needed to perform orthopedic surgeries. The intraoperative guidance system 108 provides guidance to assist users of the orthopedic surgery system 100 in performing orthopedic surgeries. The medical education system 110 can assist in the training of users such as medical professionals, patients, and other types of individuals. The pre-operative and intraoperative monitoring system 112 can assist in monitoring the patient before and after the patient undergoes surgery. The predictive analytics system 114 can assist medical professionals in making various types of predictions. For example, the predictive analytics system 114 can apply artificial intelligence techniques to determine a classification (e.g., diagnosis) of the health condition of an orthopedic joint, determine which type of surgery to perform on a patient and / or which type of implant to use during the procedure, determine the types of items that may be needed during a surgical procedure, and so on.

[0076] Subsystems of the orthopedic surgery system 100 (e.g., the virtual planning system 102, the planning support system 104, the manufacturing and delivery system 106, the intraoperative guidance system 108, the medical education system 110, the pre-operative and intraoperative monitoring system 112, and the predictive analytics system 114) can include various systems. Systems in the subsystems of the orthopedic surgery system 100 can include various types of computing systems, computing devices, including server computers, personal computers, tablets, smartphones, display devices, Internet of Things (IoT) devices, visualization devices (e.g., mixed reality (MR) visualization devices, virtual reality (VR) visualization devices, holographic projectors, or other devices for presenting extended reality (XR) visualizations), surgical tools, and so on. In some embodiments, a holographic projector can project a hologram for viewing by a group of multiple users or a single user without the need for headphones, rather than for viewing only by users wearing headphones. For example, the virtual planning system 102 can include an MR visualization device and one or more server devices, and the planning support system 104 can include one or more personal computers and one or more server devices, and so on. A computing system is one or more computing devices and / or a collection of systems that are configured to operate as a system. In some embodiments, one or more devices can be shared between two or more subsystems of the orthopedic surgery system 100. For example, in a previous embodiment, the virtual planning system 102 and the planning support system 104 can include the same server device.

[0077] Exemplary MR visualization devices include the Microsoft HOLOLENS available from Microsoft Corporation in Redmond, Washington TM headset, which includes a see-through holographic lens, sometimes referred to as a waveguide, that allows a user to see real-world objects through the lens and simultaneously see projected 3D holographic objects. The Microsoft HOLOLENS TM headset or a waveguide-based visualization device is an example of an MR visualization device that can be used in some embodiments of the present disclosure. Some holographic lenses can present holographic objects with a degree of transparency through the see-through holographic lens so that the user can see real-world objects and virtual holographic objects. In some embodiments, some holographic lenses may sometimes completely prevent the user from seeing real-world objects, but instead may allow the user to see a complete virtual environment. The term "mixed reality" can also encompass scenarios in which one or more users can perceive one or more virtual objects generated by holographic projections. In other words, "mixed reality" can encompass situations in which a holographic projector generates a hologram of elements that appear to a user to exist in the user's actual physical environment. Although the MR visualization device is described herein as an example, in other embodiments, display screens such as cathode ray tube (CRT) displays, liquid crystal displays (LCDs), and light-emitting diode (LED) displays can be used to present any aspect of the information described herein.

[0078] In Figure 1 an embodiment, devices included in subsystems of the orthopedic surgery system 100 can communicate using the communication network 16. The communication network 16 can include various types of communication networks, including one or more wide area networks such as the Internet, local area networks, and the like. In some embodiments, the communication network 116 can include wired and / or wireless communication links.

[0079] Various variations of the orthopedic surgery system 100 are possible. The variation can include more or fewer subsystems than the Figure 1 version of the orthopedic surgery system 100 shown. For example, Figure 2 is a block diagram of an orthopedic surgery system 200 that includes one or more mixed reality (MR) systems according to an embodiment of the present disclosure. The orthopedic surgery system 200 can be used to create, validate, update, modify, and / or implement a surgical plan. In some embodiments, such as by using a virtual surgical planning system (e.g., BLUEPRINT TMa system) that can create a surgical plan preoperatively and then verify, modify, update, and view the surgical plan intraoperatively, for example, using MR visualization or other visualization of the surgical plan. In other embodiments, the orthopedic surgery system 200 can be used to create a surgical plan immediately as needed before or during surgery. In some embodiments, the orthopedic surgery system 200 can be used for surgical tracking, in which case the orthopedic surgery system 200 can be referred to as a surgical tracking system. In other cases, the orthopedic surgery system 200 can generally be referred to as a medical device system.

[0080] In Figure 2 an embodiment, the orthopedic surgery system 200 includes a preoperative surgical planning system 202, a healthcare facility 204 (e.g., a surgical center or a hospital), a storage system 206, and a network 208 that allows a user to access stored patient information at the healthcare facility 204, such as medical history, image data corresponding to an injured joint or bone, and various parameters corresponding to a surgical plan that has been created preoperatively (as an example). The preoperative surgical planning system 202 can be equivalent to Figure 1 the virtual planning system 102 in TM and, in some embodiments, can generally correspond to a virtual planning system similar to or the same as the BLUEPRINT

[0081] In Figure 2In an embodiment, the healthcare facility 204 includes a mixed reality (MR) system 212. In some embodiments of the present disclosure, the MR system 212 includes one or more processing devices (P) 210 to provide functions such as presenting visual information related to preoperative planning, intraoperative guidance, or even postoperative review and follow-up to a user. The processing device 210 may also be referred to as a processor. In addition, one or more users of the MR system 212 (e.g., a surgeon, a nurse, or other healthcare provider) can use the processing device (P) 210 to generate a request for a specific surgical plan or other patient information, and the specific surgical plan or other patient information is sent to the storage system 206 via the network 208. In response, the storage system 206 returns the requested patient information to the MR system 212. In some embodiments, a user can use other processing devices to request and receive information, such as one or more processing devices that are part of the MR system 212 but not part of any visualization device, or one or more processing devices that are part of a visualization device (e.g., visualization device 213) of the MR system 212, or a combination of one or more processing devices that are part of the MR system 212 but not part of any visualization device and one or more processing devices that are part of a visualization device (e.g., visualization device 213) of the MR system 212, and the visualization device (e.g., visualization device 213) is part of the MR system 212. In other words, an exemplary MR visualization device such as the Microsoft HOLOLENS TM devices may include all components of the MR system 212 or perform some or all of the processing functions necessary for the passive visualization device 213 using one or more external processors and / or memories.

[0082] In some embodiments, multiple users can use the MR system 212 simultaneously. For example, the MR system 212 can be used in an audience mode, in which multiple users each use their own visualization devices so that the users can see the same information simultaneously and from the same perspective. In some embodiments, the MR system 212 can be used in a mode in which multiple users each use their own visualization devices so that the users can see the same information from different perspectives.

[0083] In some embodiments, the processing device 210 is capable of providing a user interface for displaying data and receiving input from a user at the healthcare facility 204. The processing device 210 may be configured to control the visualization device 213 to present the user interface. Further, the processing device 210 may be configured to control the visualization device 213 (e.g., one or more optical waveguides such as a holographic lens) to present virtual images, such as 3D virtual models, 2D images, surgical planning information, etc. The processing device 210 can include various different processing or computing devices, such as servers, desktop computers, laptop computers, tablets, mobile phones, and other electronic computing devices, or processors within such devices. In some embodiments, one or more processing devices 210 can be located away from the healthcare facility 204. In some embodiments, the processing device 210 is located within the visualization device 213. In some embodiments, at least one processing device 210 is located outside the visualization device 213. In some embodiments, one or more processing devices 210 are located within the visualization device 213 and one or more processing devices 210 are located outside the visualization device 213.

[0084] In Figure 2 the embodiments of, the MR system 212 further includes one or more memories or storage devices (M) 215 for storing data and instructions of software executable by the processing device 210. The instructions of the software can correspond to the functions of the MR system 212 described herein. In some embodiments, functions of virtual surgical planning applications such as the BLUEPRINT TM system, etc. can also be stored and executed by the processing device 210 in combination with the memory storage device (M) 215. For example, the memory or storage system 215 may be configured to store data corresponding to at least a portion of the virtual surgical plan. In some embodiments, the storage system 206 may be configured to store data corresponding to at least a portion of the virtual surgical plan. In some embodiments, the memory or storage device (M) 215 is located within the visualization device 213. In some embodiments, the memory or storage device (M) 215 is located outside the visualization device 213. In some embodiments, the memory or storage device (M) 215 includes a combination of one or more memories or storage devices within the visualization device 213 and one or more memories or storage devices located outside the visualization device.

[0085] Network 208 may be equivalent to network 116. Network 208 can include one or more wide area networks, local area networks, and / or global networks (e.g., the Internet) that connect the pre-operative surgical planning system 202 and the MR system 212 to the storage system 206. The storage system 206 can include one or more databases that contain patient information, medical information, patient image data, and parameters defining a surgical plan. For example, in preparing for an orthopedic surgical procedure, medical images of the patient's diseased or damaged bone and / or soft tissue are typically generated pre-operatively. The medical images can include images of the bone and / or soft tissue taken along the sagittal and coronal planes of the patient's body. The medical images can include X-ray images, magnetic resonance imaging (MRI) images, computed tomography (CT) images, ultrasound images, and / or any other type of 2D or 3D image that provides information about the surgical area. The storage system 206 can also include data identifying the implant components selected for a particular patient (e.g., type, size, etc.), the surgical guidance selected for a particular patient, and details of the surgical procedure such as entry points, cutting planes, drill axes, reaming depths, etc. As an example, the storage system 206 can be a cloud-based storage system (as shown) or can be located at the healthcare facility 204 or at the location of the pre-operative surgical planning system 202, or can be part of the MR system 212 or the visualization device (VD) 213.

[0086] Before (e.g., preoperatively) or during (e.g., intraoperatively) a surgical procedure, a surgeon can use the MR system 212 to create, review, validate, update, modify, and / or implement a surgical plan. In some embodiments, the MR system 212 can also be used after a surgical procedure (e.g., postoperatively) to review the results of the surgical procedure, evaluate the need for corrections, or perform other postoperative tasks. Thus, the MR system 12 can enable a user to see a real-world scene such as an anatomical object in addition to virtual images (e.g., virtual scapula or humerus images, guide images, or other text or images) placed at the real-world scene. To this end, the MR system 212 can include a visualization device 213 worn by the surgeon and (as described in further detail below) operable to display various types of information, including 3D virtual images of the patient's diseased, damaged, or postoperative joint and details of the surgical plan, such as images of the patient's bones and / or soft tissues derived from patient imaging data, generated models of bones or soft tissues, 3D virtual images of the prosthetic implant components selected for the surgical plan, 3D virtual images of the entry points for positioning the prosthetic components, surgical guides, and instruments and their placement on the damaged joint for properly orienting and positioning the prosthetic components during the surgical procedure, 3D virtual images of the alignment axes and cutting planes for shaping the bone planes to align cutting or reaming tools or 3D virtual images of drilling tools defining one or more holes in the bone surface, and any other information that the surgeon can use to implement the surgical plan. The MR system 212 can generate images of this information that are perceivable by a user of the visualization device 213 before and / or during the surgical procedure.

[0087] In some embodiments, the MR system 212 includes multiple visualization devices (e.g., multiple instances of the visualization device 213) to enable multiple users to simultaneously view the same image and share the same 3D scene. In some such embodiments, one visualization device can be designated as the primary device and other visualization devices can be designated as viewers or spectators. The users of the MR system 212 can, as needed, re-designate any viewer device as the primary device at any time.

[0088] Figure 3 is a flowchart showing exemplary stages of a surgical lifecycle (process cycle) 300. At Figure 3In an embodiment, the surgical procedure lifecycle 300 begins with a preoperative phase (302). During the preoperative phase, a surgical plan is developed. After the preoperative phase is the manufacturing and delivery phase (304). During the manufacturing and delivery phase, patient-specific items such as parts and equipment required to perform the surgical plan are manufactured and delivered to the surgical site. In some embodiments, patient-specific items are not required to perform the surgical plan. The intraoperative phase (306) follows the manufacturing and delivery phase. The surgical plan is executed during the intraoperative phase. In other words, one or more individuals perform surgery on the patient during the intraoperative phase. After the intraoperative phase is the postoperative phase (308). The postoperative phase includes activities that occur after the surgical plan is completed. For example, the patient's complications can be monitored during the postoperative phase.

[0089] As described in the present disclosure, the orthopedic surgery system 100 can be used in one or more of the preoperative phase 302, the manufacturing and delivery phase 304, the intraoperative phase 306, and the postoperative phase 308 ( Figure 1 ). For example, the virtual planning system 102 and the planning support system 104 can be used in the preoperative phase 302. In some embodiments, the preoperative phase 302 can include systems for analyzing patient imaging data, modeling bones and / or soft tissues, and / or determining or recommending the type of surgical treatment based on the patient's health condition. The manufacturing and delivery system 106 can be used in the manufacturing and delivery phase 304. The intraoperative guidance system 108 can be used in the intraoperative phase 306. In Figure 3 multiple phases can be used Figure 1 some of the systems in. For example, the medical education system 110 can be used in one or more of the preoperative phase 302, the intraoperative phase 306, and the postoperative phase 308; the preoperative and postoperative monitoring system 112 can be used in the preoperative phase 302 and the postoperative phase 308. The predictive analytics system 114 can be used in the preoperative phase 302 and the postoperative phase 308. Figure 3 There can be various workflows during the surgical procedure. For example, Figure 3 different workflows within the surgical procedure may be suitable for different types of surgeries.

[0090] Figure 4 is a flowchart showing exemplary preoperative, intraoperative, and postoperative workflows that support an orthopedic surgery procedure. In Figure 4In an embodiment, a surgical procedure begins with a medical consultation (400). During the medical consultation (400), a medical professional assesses the patient's medical health. For example, the medical professional may ask the patient about the patient's symptoms. During the medical consultation (400), the medical professional may also discuss various treatment options with the patient. For example, the medical professional may describe one or more different surgical procedures to address the patient's symptoms.

[0091] Further, Figure 4 The embodiments in include a case creation step (402). In other embodiments, the case creation step is performed before the medical consultation step. During the case creation step, a medical professional or other user creates an electronic case file for the patient. The patient's electronic case file may include information related to the patient, such as data about the patient's symptoms, observations of the patient's range of motion, data about the patient's surgical plan, the patient's medical images, instructions for the patient, billing information for the patient, and so on.

[0092] Figure 4 The embodiments in include a preoperative patient monitoring phase (404). During the preoperative patient monitoring phase, the patient's symptoms may be monitored. For example, the patient may be suffering from pain associated with arthritis in the patient's shoulder. In this embodiment, the patient's symptoms may not have risen to the level that requires arthroplasty to replace the patient's shoulder. However, arthritis typically worsens over time. Accordingly, the patient's symptoms may be monitored to determine whether it is time to perform surgery on the patient's shoulder. Observations from the preoperative patient monitoring phase may be stored in the patient's electronic case file. In some embodiments, a predictive analytics system 114 may be used to predict when the patient may need surgery, predict a delay in the treatment process or avoid surgery, or make other predictions about the patient's health.

[0093] In addition, in Figure 4 the embodiments of, a medical image acquisition step (406) is performed during the preoperative phase. During the image acquisition step, medical images of the patient are generated. Medical images of a particular patient may be generated in a variety of ways. For example, a computed tomography (CT) procedure, a magnetic resonance imaging (MRI) procedure, ultrasound processing, or other imaging procedures may be used to generate the images. The medical images generated during the image acquisition step include images of the anatomical structures of interest of a particular patient. For example, if the patient's symptoms involve the patient's shoulder, medical images of the patient's shoulder may be generated. The medical images may be added to the patient's electronic case file. The medical professional can use the medical images during one or more of the preoperative, intraoperative, and postoperative phases.

[0094] Further, in Figure 4In an embodiment, an automatic processing step (408) can be performed. During the automatic processing step, the virtual planning system 102 ( Figure 1 ) can automatically develop a preliminary surgical plan for the patient. For example, the virtual planning system 102 can generate a model or representation of the patient's bones and / or soft tissues. Based on these representations, the virtual planning system 102 can determine bone and / or soft tissue characteristics such as soft tissue volume, fat infiltration of the muscle, atrophy rate of the muscle, and range of motion of the bone. The virtual planning system 102 can determine what type of treatment should be performed (e.g., whether a shoulder replacement is an anatomical replacement or a reverse replacement) based on these characteristics. In some embodiments of the present disclosure, the virtual planning system 102 can use machine learning techniques to develop a preliminary surgical plan based on the information in the patient's virtual case file.

[0095] Figure 4 The embodiment also includes a manual correction step (410). During the manual correction step, one or more human users can check and correct the determinations made during the automatic processing step. In some embodiments of the present disclosure, one or more users can use a mixed reality or virtual reality visualization device during the manual correction step. In some embodiments, the changes made during the manual correction step can be used as training data to improve the machine learning techniques applied by the virtual planning system 102 during the automatic processing step.

[0096] The virtual planning step (412) can be after the Figure 4 manual correction step in. During the virtual planning step, a medical professional can develop a surgical plan for the patient. In some embodiments of the present disclosure, one or more users can use a mixed reality or virtual display visualization device during the process of developing a surgical plan for the patient.

[0097] Further, in Figure 4 the embodiment, intraoperative guidance (414) can be generated. The intraoperative guidance can include guiding the surgeon on how to perform the surgical plan. In some embodiments of the present disclosure, the virtual planning system 102 can generate at least a part of the intraoperative guidance. In some embodiments, the surgeon or other users can contribute to the intraoperative guidance.

[0098] In addition, in Figure 4 the embodiment, the step of selecting and manufacturing surgical articles (416) is performed. During the step of selecting and manufacturing surgical articles, the manufacturing and delivery system 106 ( Figure 1) Surgical articles used in a surgical procedure as described by a surgical plan can be manufactured. For example, surgical articles can include surgical implants, surgical tools, and other articles required to perform the surgical procedure described by the surgical plan.

[0099] In Figure 4 an embodiment, a surgical procedure (418) can be performed using guidance from the intraoperative system 108 ( Figure 1 ). For example, a surgeon can perform a surgical procedure while wearing a head-mounted MR visualization device of the intraoperative system 108 that presents guidance information to the surgeon. The guidance information can assist the surgeon in completing the surgical procedure, thereby providing guidance for each step in the surgical workflow, including the sequence of steps, details of each step, and tool or implant selection, implant placement and positioning, and bone surface preparation for each step in the surgical workflow.

[0100] After the surgical procedure is completed, postoperative patient monitoring (420) can be performed. During the postoperative patient monitoring step, the patient's recovery outcome can be monitored. Recovery outcomes can include symptom relief, range of motion, complications, performance of the implanted surgical articles, etc. The pre- and postoperative monitoring system 112 ( Figure 1 ) can assist in the postoperative patient monitoring step.

[0101] Figure 4 The medical consultation, case creation, pre-operative patient monitoring, image acquisition, automatic processing, manual correction, and virtual planning steps in Figure 3 are part of the pre-operative phase 302 in Figure 4 . The surgical procedure with the guidance step in Figure 3 is part of the intraoperative phase 306 in Figure 4 . The postoperative patient monitoring step in Figure 3 is part of the postoperative phase 308 in

[0102] As described above, one or more subsystems of the orthopedic surgery system 100 can include one or more mixed reality (MR) systems such as the MR system 212 ( Figure 2 ). Each MR system can include a visualization device. For example, in Figure 2In an embodiment, the MR system 212 includes a visualization device 213. In some embodiments, in addition to including a visualization device, the MR system may further include external computing resources that support the operation of the visualization device. For example, the visualization device of the MR system may be communicatively coupled to a computing device (e.g., a personal computer, a laptop computer, a tablet computer, a smart phone, etc.) that provides the external computing resources. Alternatively, sufficient computing resources for performing the necessary functions of the visualization device may be provided on or within the visualization device 213.

[0103] The virtual planning system 102 and / or other systems may analyze patient imaging data that is also used for planning surgical interventions such as joint surgery. As discussed herein, as an example, shoulder replacement surgery is one type of surgical procedure that can be planned using the systems and techniques herein. Figure 5A and Figure 5B is a diagram of exemplary muscles and bones related to a patient's shoulder.

[0104] As Figure 5A shown in an embodiment, the front view of patient 500 includes the sternum 502, the shoulder 504, and the ribs 506. Some of the bones associated with the structure and function of the shoulder 504 include the coracoid process 510 and the scapula 512 of the scapula (not all of which are shown). Muscles associated with the shoulder 504 include the serratus anterior 508, the teres major, and the biceps 518. The subscapularis 514 is Figure 5A a rotator cuff muscle shown in Figure 5B In an embodiment, other rotator cuff muscles, the supraspinatus 526, the infraspinatus 530, and the teres minor 532 are shown in the rear view of patient 500. Figure 5B The skeletal features of the humeral head 520 and the spine of the scapula 528 are also shown. Other muscles associated with the shoulder 504 include the triceps 522 and the deltoid 524.

[0105] When evaluating the shoulder 504 for treatment, such as what type of shoulder treatment or replacement may be appropriate, the system may analyze patient-specific imaging data of these bones and soft tissues such as Figure 5A and Figure 5B discussed herein. For example, the virtual planning system 102 may generate a representation of soft tissues (e.g., muscles) from the patient imaging data and determine various characteristics of the soft tissues. These characteristics may include muscle volume associated with the muscle, fat infiltration (e.g., fat ratio), muscle atrophy rate, and range of motion of the joint.

[0106] From this information, the virtual planning system 102 can determine the type of treatment being proposed, such as whether the patient would benefit from an anatomical shoulder replacement or a reverse shoulder replacement. In an anatomical shoulder replacement, the humeral head is replaced with an artificial humeral head (e.g., a partial sphere), and the glenoid surface of the scapula is replaced with an artificial surface that mates with the artificial humeral head. In a reverse shoulder replacement, an artificial partial sphere is implanted into the glenoid surface and an artificial surface (e.g., a cup) that mates with the sphere is implanted to replace the bone head. The virtual planning system 102 can also recommend the size and / or placement of the implant based on patient imaging data and / or muscle characteristics.

[0107] In one embodiment, a system such as the virtual planning system 102 can be configured to model a patient's soft tissue structure. The virtual planning system 102 can include a memory and processing circuitry configured to store patient-specific image data of the patient. The processing circuitry can be configured to receive patient-specific image data (e.g., CT data), determine a patient-specific shape representing the patient's soft tissue structure based on the intensity of the patient-specific image data, and output the patient-specific shape. Thus, the patient-specific shape can be a model of the patient's actual soft tissue structure.

[0108] The virtual planning system 102 can generate the patient-specific shape of the soft tissue structure using various methods. For example, the processing circuitry can be configured to receive an initial shape (e.g., a geometry or a statistical average shape based on a patient population) and determine a plurality of surface points on the initial shape. Then, the virtual planning system 102 can register the initial shape with the patient-specific image data (e.g., place the initial shape into the patient-specific image data based on muscle insertion points on adjacent bones) and identify one or more contours in the patient-specific image data that represent the boundaries of the patient's soft tissue structure. The one or more contours can be voxels or pixels in the patient-specific imaging data that have an intensity above a threshold indicating the boundary of the soft tissue structure. In some embodiments, the contours can be determined by identifying the separation zones between adjacent soft tissue structures (e.g., using a Hessian feature image representing the intensity gradient within the patient-specific image data). Contrary to identifying the structure boundaries based solely on very similar intensities between muscles (e.g., and adipose tissue), the Hessian feature image that identifies the separation zones between adjacent structures can improve the precision between these structure boundaries. Then, the virtual planning system 102 iteratively moves the plurality of surface points towards the corresponding positions of the one or more contours to transform the initial shape into a patient-specific shape representing the patient's soft tissue structure. Thus, each iterative movement causes the modified initial shape to become increasingly similar to the actual shape of the patient's soft tissue structure indicated in the image data.

[0109] In some embodiments, the virtual planning system 102 may display a patient-specific shape that has been modeled using imaging data. The virtual planning system 102 may also perform additional determinations as part of a surgical plan. For example, the virtual planning system 102 may use patient-specific imaging data to determine a fat volume ratio of the patient-specific shape, determine an atrophy rate of the patient-specific shape, determine a range of motion of the patient's humerus based on the fat volume ratio and atrophy rate of the patient-specific shape of the patient's soft tissue structure, and then determine one type of a plurality of types of shoulder treatment procedures for the patient based on the range of motion of the humerus.

[0110] The virtual planning system 102 may determine the range of motion of the humerus by determining the range of motion of the humerus based on the fat volume ratio and atrophy rate of one or more muscles of the patient's rotator cuff. Based on this information, the virtual planning system 102 may select the type of shoulder treatment from among anatomic shoulder replacement surgery or reverse shoulder replacement surgery. In some embodiments, for situations where the patient's bones and / or muscles cannot support anatomic shoulder replacement, the virtual planning system 102 may recommend reverse shoulder replacement surgery. Thus, a patient determined to have greater fat infiltration and a greater atrophy rate may be more suitable for reverse shoulder replacement (e.g., compared to one or more appropriate thresholds). In some embodiments, the planning system 102 may employ a decision tree or neural network and use the fat infiltration value and other parameters such as patient age, gender, activity level, and / or other factors that may indicate whether the patient is more suitable for reverse or anatomic shoulder replacement as inputs. In some embodiments, the fat infiltration value may be a type of quality metric of a soft tissue structure such as muscle. In other embodiments, the quality of the muscle may be represented by other types of values that may or may not incorporate the presence of fat in the muscle.

[0111] Figure 6 is a block diagram showing exemplary components of a system 540 configured to determine soft tissue structure metrics and other information related to surgical interventions associated with joints. The system 540 may be similar to Figure 1 the virtual planning system 102 herein and / or a system configured to perform the processes discussed herein. In Figure 6 an embodiment, the system 514 includes a processing circuit 542, a power supply 546, a display device 548, an input device 550, an output device 552, a storage device 554, and a communication device 544. In Figure 6In an embodiment, the display device 548 can display an image to present a user interface such as an opaque or at least partially transparent screen to the user. The display device 548 can present visual information and, in some embodiments, present audio information or other information to the user. For example, the display device 548 can include one or more speakers, haptic devices, etc. In other embodiments, the output device 552 can include one or more speakers and / or haptic devices. The display device 548 can include an opaque screen (e.g., an LCD or LED display). Alternatively, the display device 548 can include an MR visualization device (e.g., including a see-through holographic lens) and a projector that allows the user to see real-world objects in the real-world environment through the lens and also see virtual 3D holographic images projected into the lens and onto the user's retina, such as through a holographic projection system such as the Microsoft HOLOLENS TM device. In this embodiment, the virtual 3D holographic object may appear to be placed in the real-world environment. In some embodiments, the display device 548 includes one or more display screens such as an LCD display screen, an OLED display screen, etc. The user interface can present a virtual image of the details of a virtual surgical plan for a specific patient.

[0112] In some embodiments, the user can interact with and control the system 540 in various ways. For example, the input device 550 can include one or more microphones and associated speech recognition processing circuitry or software that can recognize voice commands issued by the user and, in response, perform various arbitrary operations such as selection, activation, or deactivation of various functions associated with surgical planning, intraoperative guidance, etc. As another embodiment, the input device 550 can include one or more cameras or other optical sensors that detect and interpret gestures to perform the operations described above. As yet another embodiment, the input device 550 includes one or more devices that sense the direction of gaze and perform various operations as described elsewhere in this disclosure. In some embodiments, the input device 550 can receive manual input from the user via a handheld controller that includes, for example, one or more buttons, keypads, keyboards, touchscreens, joysticks, trackballs, and / or other manual input media, and perform the various operations described above in response to the manual user input.

[0113] The communication device 544 may include one or more circuits or other components that facilitate data communication with other devices. For example, the communication device 544 may include one or more physical drives (e.g., DVD, Blu-ray, or Universal Serial Bus (USB) drives) that allow data to be transferred between the system 540 and the drive when physically connected to the system 540. In other embodiments, the communication device 544 may include. The communication device 544 may also support wired and / or wireless communication with another computing device and / or network.

[0114] The storage device 544 may include one or more memories and / or repositories that store corresponding types of data in common and / or separate devices. For example, the user interface module 556 may include instructions that define how the system 540 controls the display device 548 to present information to the user. The preoperative module 558 may include instructions for the analysis of patient data such as imaging data and / or the determination of treatment options based on patient data. The intraoperative module 560 may include instructions that define how the system 540 operates when providing information such as details about a planned surgery and / or feedback about a surgical procedure to a clinician for display.

[0115] The surface fitting module 562 may include instructions that define how the processing circuit 542 determines a representation of soft tissue (e.g., a patient-specific shape) from patient-specific imaging data. For example, the surface fitting module 564 may specify an initial shape, the number of iterations, and other details regarding adjusting the initial shape to a patient-specific shape based on the intensity of the patient imaging data. The image registration module 564 may include instructions that define how to register the initial shape or other anatomical structures with the patient image data. For example, before generating a patient-specific shape during the surface fitting process, the image registration module 564 may instruct the processing circuit 542 how to register a statistical mean shape (SMS) (e.g., an anatomical shape derived from a multi-population) with the bones of the patient imaging data. The patient data 566 may include any type of patient data, such as patient imaging data (e.g., CT scan, X-ray scan, or MRI data), patient characteristics (e.g., age, height, weight), patient diagnosis, patient health status, previous surgeries or implants, or any other information related to the patient.

[0116] As discussed above, the surgical lifecycle 300 may include a preoperative phase 302( Figure 3)。One or more users may use the orthopedic surgery system 100 during the preoperative phase 302. For example, the orthopedic surgery system 100 may include a virtual planning system 102 (which may be similar to system 540) that aids one or more users in generating a virtual surgical plan customized for the anatomical structure of interest of a particular patient. As described herein, the virtual surgical plan may include a three-dimensional (3D) virtual model corresponding to the anatomical structure of interest of a particular patient and 3D models of one or more prosthetic components that are matched to the particular patient to repair the anatomical structure of interest or are selected to repair the anatomical structure of interest. The virtual surgical plan may also include a 3D virtual model of guidance information that guides the surgeon during the performance of the surgical procedure, e.g., when preparing the bone surface or tissue and placing implantable prosthetic hardware relative to the bone surface or tissue.

[0117] As discussed herein, system 540 may be configured to model the soft tissue structures of a patient using patient imaging data. For example, system 540 may include a memory (e.g., storage device 554) configured to store patient-specific image data of the patient (e.g., patient data 566). System 540 also includes processing circuitry 542 configured to receive the patient-specific image data and determine a patient-specific shape representative of the soft tissue structures of the patient based on the intensity of the patient-specific image data. The processing circuitry 542 can then output the patient-specific shape for display or use in further analysis of the patient. For example, the processing circuitry 542 may use the patient-specific shape or other features from the patient-specific image data to generate surgical procedure recommendations as described herein (e.g., what type of treatment should be performed on the patient).

[0118] The processing circuit 542 can use one or more processes to determine a patient-specific shape. For example, the processing circuit 542 can receive an initial shape (e.g., a geometry or an SMS), determine a plurality of surface points on the initial shape, and register the initial shape with patient-specific image data. The processing circuit 542 can register the initial shape by determining one or more muscle insertion points and / or origins on a pre-segmented bone in the patient-specific image data or by otherwise identifying an approximate location of a soft tissue structure of interest. Then, the processing circuit 542 can identify one or more contours in the patient-specific image data that represent the boundaries of the patient's soft tissue structures (which can be based on a separation zone between soft tissue structures) and iteratively move the plurality of surface points toward the corresponding locations of the one or more contours to change the initial shape into a patient-specific shape that represents the patient's soft tissue structures. Thus, as the boundaries of the initial shape are iteratively moved toward more closely fitting the contours, the processing circuit 542 can generate one or more intermediate shapes. The contours can represent a set of voxels that exceed a certain threshold or fall within a threshold range and indicate the boundaries of the soft tissue structures.

[0119] In some embodiments, the initial shape and the patient-specific shape are three-dimensional shapes. However, in other embodiments, the initial shape and / or the patient-specific shape can be defined as two-dimensional. In these embodiments, a collection of several two-dimensional shapes can be used to define the entire volume, or three-dimensional shape. In one embodiment, the processing circuit 542 can iteratively move the surface points of the initial shape and the intermediate shapes in the directions of respective vectors in three dimensions such that the processing circuit 542 processes data in three-dimensional space. In other embodiments, the processing circuit 542 can operate in two-dimensional slices to change the initial shape toward the contours in the patient-specific image data. Then, the processing circuit 542 can combine several two-dimensional slices to generate a full three-dimensional volume of the patient's final patient-specific shape.

[0120] Soft tissue structures can include muscles, tendons, ligaments, or other connective tissues that are not bone. Even though joint replacement treatments typically may involve modification of bone (e.g., replacing at least a portion of the bone with an artificial material such as metal and / or polymer), nevertheless, the soft tissue state can inform what type of replacement may be suitable for the particular joint being replaced. Thus, system 540 can analyze a patient's soft tissues such as the muscles around a joint to obtain information that may affect the type of joint replacement. In the case of shoulder replacement, the soft tissue structures of interest for the joint can include rotator cuff muscles such as the subscapularis, supraspinatus, infraspinatus, and teres minor muscles. Other muscles associated with the shoulder such as the latissimus dorsi, deltoid, serratus anterior, triceps, and biceps can be analyzed for shoulder replacement treatment. For surgical planning purposes, system 540 can determine the respective characteristics of each soft tissue structure for the purpose of determining what type of range of motion and / or stress the new reconstructed joint can withstand.

[0121] In some embodiments, processing circuit 542 can determine the type of treatment for a patient's shoulder based on various criteria such as the range of motion of the humerus relative to the glenoid surface or the rest of the scapula. The type of shoulder treatment can include anatomic shoulder replacement or reverse shoulder replacement, and processing circuit 542 can recommend what type of replacement is preferred for the patient based on the soft tissue characteristics. In addition, processing circuit 542 recommends other parameters for the treatment such as implant placement location, angle, orientation, type of implant, etc. For example, processing circuit 542 can determine the fat volume ratio (e.g., fat infiltration value) of a patient-specific shape from patient-specific image data. Processing circuit 542 can also determine the atrophy rate of a patient-specific shape based on the pre-morbid or previous healthy state of the soft tissue structures of interest. Then, processing circuit 542 can determine the range of motion of the patient's humerus based on the fat volume ratio and atrophy rate of the patient-specific shape of the patient's soft tissue structures. For example, higher fat infiltration and atrophy of the muscles may indicate a lower (or narrower) range of motion of the joint. In some embodiments, the range of motion can be one or more specific angles of the corresponding movement of the joint or a metric or other composite value representing the overall range of motion of the joint. Processing circuit 542 can determine the range of motion of the joint based on several muscles. For example, processing circuit 542 can determine one or more range of motion values (e.g., one or more individual angles or one or more composite values representing the overall range of motion) of the humerus relative to the scapula based on the fat ratios and atrophy rates of several corresponding muscles of the rotator cuff and / or other muscles or connective tissues associated with the shoulder joint. In addition, the range of motion may be affected by bone-to-bone collisions or other mechanical impacts. From this information, processing circuit 542 can recommend the type of treatment for the patient's shoulder during the pre-operative planning phase.

[0122] Figures 7 to 18B Illustrates exemplary steps involved in modeling a patient's soft tissue structure from patient-specific image data. Processing circuitry 542 of system 540 is described as an exemplary system that performs these processes, however, other devices, systems, or combinations thereof may perform similar determinations. Figure 7 Is a diagram of exemplary insertion points of some of the muscles of the rotator cuff.

[0123] As Figure 7 As shown in the embodiment of [], processing circuitry 542 may begin soft tissue modeling by registering an initial shape with the patient's associated bones. Glenoid cavity 578 of the scapula is shown together with the head 520 of the humerus 570. The patient's bones may be determined from an automatic segmentation process in which processing circuitry 542 or other systems determine the bones based on intensity values of patient-specific image data. From these bones, processing circuitry 542 may identify insertion points or attachment points (or additional origins of muscles) of one or more soft tissue structures of interest in the shoulder joint. In other embodiments, the insertion points or other bony landmarks may be determined directly from patient-specific image data rather than from bone segmentation. For example, the insertion of the supraspinatus muscle 572 indicates the location where the supraspinatus muscle attaches to the head 520 of the humerus, the insertion of the infraspinatus muscle 574 indicates the location where the infraspinatus muscle attaches to the head 520 of the humerus, and the insertion of the scapula 576 indicates the location where the scapular muscle attaches to the head 520 of the humerus. Processing circuitry 542 may identify each of these insertion points based on a comparison with an anatomical diagram or other instructions based on general human anatomy. In some embodiments, processing circuitry 542 may determine additional insertion points on the scapula or other bones as additional points for registering the initial shape with the patient's bones.

[0124] As discussed herein, the initial shape that is transformed and fitted to the patient's image data can begin as a geometry or more specifically an SMS. The SMS may be selected for a general population or from a plurality of different SMSs based on one or more demographic factors (e.g., gender, age, race, etc.) of the patient population. In some embodiments, the SMS may be used because it may more closely match the patient's muscles. Thus, processing circuitry 542 may reduce the number of iterations or calculations required to modify the initial shape and generate a patient-specific shape that fits the image data. Additionally, the SMS may include pre-identified locations that match the identified insertion points on the associated bones.

[0125] Figure 8A Is a conceptual diagram of exemplary patient-specific image data. As Figure 8AAs shown, the display shows two-dimensional image data 600 of the intensity of a CT or x-ray image. Higher intensities represent dense tissues that absorb x-ray energy, such as bone in the form of the scapula 602. Generally, because tissues absorb x-ray energy at similar low levels, it is more difficult to identify soft tissue structures such as muscle in CT data. For example, the infraspinatus muscle may have a boundary partially defined by edge 601 (i.e., the edge of the scapula 602) and edge 603 (i.e., the separation zone between the outer edge of the infraspinatus muscle and an adjacent layer of adipose tissue). Although edge 603 may be difficult to identify from x-ray data, however, specific data processing techniques can assist in identifying edge 603 or the separation zone from which edge 603 can be determined.

[0126] Figure 8B is based on Figure 8A the patient-specific image data in to generate a conceptual diagram of the Hessian feature image 605. As Figure 8B shown in the embodiment of, the processing circuit 542 can determine the Hessian feature image 605 from patient-specific CT data such as Figure 8A the image data 600. The Hessian feature image 605 is shown as a two-dimensional image, however, it can be part of a three-dimensional data field. The Hessian feature image 605 indicates regions of the patient-specific CT data that include higher intensity gradients between two or more voxels in the patient-specific CT data. For example, the processing circuit 542 can calculate second derivatives between adjacent voxels, or groups of voxels, to determine the gradients between the voxels or groups of voxels, and then generate the Hessian feature image based on the second derivatives. Although the Hessian feature image is described as a three-dimensional image, however, the same techniques can also be used to generate two-dimensional images.

[0127] Because there are variations in intensity between the voxels of these structures, such as between bone objects and soft tissues, between two sets of bone objects close to each other, and between two sets of soft tissues close to each other, etc., the Hessian feature image can show the separation between two anatomical objects. The processing circuit 542 can determine one or more contours, or at least a portion of a contour, based on voxel-based separation information (e.g., based on the Hessian feature image).

[0128] Although the contours indicate the spacing between anatomical objects, the contours may not provide the complete shape of the anatomical objects. For example, due to imperfect imaging, lack of intensity gradients between voxels or groups of voxels, or noise, there may be holes, gaps, or other errors that cause discontinuities in the contours representing the boundaries of anatomical structures (e.g., bone or soft tissue). As an example, the contours may be incomplete and may not be a closed surface representing the anatomical object. Generally, the contours can provide an initial estimate of the size, shape, and location of the anatomical object. However, as described in more detail, because the contours are based on image information and 2D scans may be imperfect, the contours may be an inaccurate indicator of the actual anatomical object (e.g., due to holes or other missing parts, as well as protrusions in the contours). Thus, the processing circuit 542 can use an initial shape having a closed surface and modify the initial shape to approximate the contours defined at least by the Hessian feature image portion.

[0129] The Hessian feature image 605 includes a number of lines of various intensities (e.g., white indicates regions where the gradient between voxels is higher than in darker or black regions). The whiter and wider lines indicate regions of higher gradient. The clavicle 602 can be identified as the line around the outer surface of the clavicle 602, i.e., indicating a larger gradient between voxels having a higher intensity starting from the bone and those having a lower intensity starting from the soft tissue. As Figure 8B shown, the infraspinatus muscle can have boundaries partially defined by contour 607A (i.e., the edge of the scapula 602) and contour 607B (i.e., the separation zone between the outer edge of the infraspinatus muscle and the adjacent layer of adipose tissue). Each of the contours 607A and 607B can be identified as extending through the middle of the separation zone indicated by the lines of the Hessian feature image 605. Thus, the contours 607A and 607B can indicate the correspondence between adjacent structures near the boundaries of each structure.

[0130] Each of the contours 607A and 607B can be discontinuous, however, in other embodiments, these contours can be continuous separation zones. Thus, the contours 607A and 607B can form at least a part of the contour representing the boundary of the infraspinatus muscle. In some embodiments, other separation zones can be used to provide contours for other muscles. For example, the Hessian feature image 605 can represent skin boundaries such as contour 607C, which can be used to identify the boundaries of these muscles (which are typically located adjacent to the skin having the least adipose tissue between the muscle and the skin).

[0131] Based on the Hessian feature image 605, the processing circuit 542 can identify one or more separation regions between the soft tissue structure and adjacent soft tissue structures. In other words, the separation region can indicate the intensity gradient between two soft tissue structures in the patient-specific image data. Thus, the processing circuit 542 can determine that at least a portion of one or more contours passes through one or more separation regions. The processing circuit 542 can determine that the contour passes through the middle of the separation region or through the intensity-based weighted middle of the separation region.

[0132] Figure 8C is a conceptual diagram of an exemplary initial shape 604 and an exemplary segmentation contour superimposed on patient-specific image data. As Figure 8C shown, the display shows two-dimensional image data 600 of a process of modeling a soft tissue structure according to the exemplary process described herein. Although the soft tissue structure of the infraspinatus muscle is shown as an example, the same process can be performed on any soft tissue structure. Although a two-dimensional image is shown for illustrative purposes in Figure 8C , the processing circuit 542 can perform these processes in three-dimensional space.

[0133] An initial shape such as the initial shape 604 shown as a dashed line is selected for the soft tissue structure of interest (i.e., the infraspinatus muscle in this embodiment). Then, the initial shape 604 is registered with one or more associated bones including the scapula 602. For example, the insertion points identified on the scapula 602 can be used to match the corresponding attachment points identified from the initial shape 604. The initial shape 604 can be an SMS, i.e., an anatomical shape representing the soft tissue structures of multiple subjects different from the patient. Since the SMS is specific to the muscle of interest, the initial shape 604 may be similar to the patient's structure. However, as Figure 8C shown, the initial shape 604 does not accurately reflect the boundary of the scapula 602 or other intensities of the image data 600. Thus, the processing circuit 542 can deform or modify the initial shape 604 to fit the image data of this particular patient.

[0134] The processing circuit 542 can move portions of the initial shape 604 toward the actual soft tissue structure of the patient represented by the segmentation contour 606 (e.g., similar to the synthesized patient-specific shape) through one or more iterations. As shown by the arrow groups 608A, 608B, and 608C (collectively "arrow group 608"), the portions of the initial shape 604 are deformed toward other corresponding portions of the segmentation contour 606. Figure 8CEach arrow in the corresponding group of arrows shown moves at least a portion of the distance toward a corresponding surface point on the surface of the initial shape 604 during one iteration toward the segmentation contour 606. As such, after two or more iterations, the processing circuitry 542 can deform the initial shape 604 to fit the segmentation contour 606. For example, a portion of the segmentation contour 606 is moved in the direction of the group of arrows 608B to fit a portion of the scapula 602. Now the portions of the initial shape 604 that are also moved in the directions of the group of arrows 608A and the group of arrows 608C are fit to the contours of the soft tissue structures indicated by the intensities of the image data 600. In some embodiments, this process may be referred to as closed surface fitting.

[0135] In some embodiments, the processing circuitry 542 can receive the segmentation contour 606, i.e., an initial analysis of patient-specific image data, to generate a representation of a soft tissue structure of interest such as the infraspinatus muscle. For example, the segmentation contour 606 can be a three-dimensional shape or a plurality of two-dimensional slices. However, the segmentation contour 606 may not be a complete model representing the soft tissue structure. For example, individual voxels in the image data may be inaccurate or missing, and the synthesized segmentation contour 606 may be incomplete. For example, the segmentation contour 606 can be determined at least in part from contours identified from the separation region based on the Hessian feature image. Thus, deforming a fully enclosed SMS or geometry to fit the image data 600 may result in generating a complete model of the soft tissue structure.

[0136] In other embodiments, the processing circuitry 542 may not receive any segmentation contour 606 of the soft tissue structure. Instead, the processing circuitry 542 can place the initial shape based on insertion points on the associated bone within the patient-specific image data. Then, the processing circuitry 542 can extend vectors from various positions on the surface of the initial shape toward corresponding voxels, or otherwise indicate the edges of the soft tissue structures or bones in the image data 600 that exceed a threshold intensity, differ from surrounding voxels by a predetermined value or percentage. For example, these voxels that exceed the threshold intensity can together constitute one or more contours of the soft tissue structure represented by the patient-specific image data. Alternatively, as discussed herein, contours representing the edges of the soft tissue structure can be determined based on gradients between voxels in, for example, the Hessian feature image. Thus, after one or more iterations of deforming the initial shape 604 toward the identified voxels that exceed the threshold or exceed the relative change in voxel intensity, the initial shape 604 can then be transformed into a patient-specific shape representing the soft tissue structure of the patient. Thereby, the patient-specific shape can be a model of the structure of the patient, and various features such as volume, length, etc. can be calculated by the processing circuitry 542. Figures 9 to 11 An exemplary process is shown for determining a patient-specific shape from patient-specific image data by iteratively moving points on the surface of an initial shape.

[0137] Figure 9 It is a conceptual diagram of an exemplary procedure for changing an initial shape 616 towards a patient-specific shape 614 representing the soft tissue structure of a patient. Figure 9 The embodiment in [the figure] shows patient image data, the initial shape 616, and a sagittal view 610 of the scapula 612. The initial shape 616 can be placed and deformed to represent the soft tissue structure of the subscapularis muscle, i.e., it can be similar to the contour 614 represented by the dotted line. The processing circuit 542 registers the initial shape 616 within the image data by registering a plurality of positions on the initial shape 616 with corresponding insertion positions on one or more bones identified in the patient image data. For example, the initial shape 616 can be registered with the insertion positions on the scapula 612. This registration may not require any part of the initial shape 616 to actually contact the scapula 612. However, the registration uses the insertion positions on the scapula 612 as a guide for registering the initial shape 616 within the patient-specific image data. The initial shape 616 is shown as an SMS to approximate the patient's structure. In other embodiments, geometric shapes such as spheres, ellipsoids, or other structures can be used as the initial shape 616.

[0138] Because the initial shape 616 needs to be deformed or modified, the processing circuit 542 can move each part of the initial shape 616 towards the voxels of the image data representing the surface of the soft tissue structure as needed. For example, the processing circuit 542 can select a plurality of surface points on the initial shape 616. Each of the vectors 618A and 618B (collectively referred to as "vector 618") extends from a corresponding surface point on the initial shape 616 and in a direction orthogonal to the surface of the initial shape 616. The processing circuit 542 can extend the vector 618 inside and outside the initial shape 616, which can enable the processing circuit 542 to identify the contour of the soft tissue structure located outside or inside the initial shape 616 based on the position where the initial shape 616 is initially registered with the imaging data. For example, because a part of the scapula 612 is located inside the initial shape 616, the vector 618B points inward from the surface of the initial shape 616. In contrast, the vectors 618A around the remaining part of the initial shape 616 eventually point outward from the surface of the initial shape 616.

[0139] Thus, as indicated by the surfaces of the contour 614 and the scapula 612, the processing circuit 542 can identify one or more contours corresponding to the edges of the soft tissue structures using the vectors 618. For example, the processing circuit 542 can extend the respective vectors 618 from each surface point (e.g., the point based on each vector 618) in at least one of the ways of extending outwardly or inwardly from the respective surface points among the plurality of surface points on the initial shape 616. Thus, the processing circuit 542 can determine the respective positions in the patient-specific image data where the voxel intensity exceeds the threshold intensity value for the vectors of each surface point. These respective positions of at least one of the plurality of surface points at least partially define one or more contours. In other words, for each of the vectors 618, it can be determined that the voxels or pixels including intensity values exceeding the threshold intensity value are part of the contour 614 or the adjacent bone of the soft tissue structure. In other embodiments, the processing circuit 542 can receive the contour 614 from a previous segmentation of the image data indicating the soft tissue structures of the patient, or can use the Hessian feature data to determine at least a portion of the contour 614 identified from the separated regions in the Hessian feature image. However, the processing circuit 542 can still deform the initial shape 616 towards the known contour to generate a fully enclosed surface model of the soft tissue structure of interest. This fully enclosed surface model can also be pre-labeled (e.g., the initial shape 616 can be pre-labeled) as to which parts should face the bone or which parts should be adjacent to another specific muscle. This labeling can enable or initiate further segmentation of the soft tissue structure.

[0140] As discussed above, the positions being searched for by each vector 618 in the patient-specific image can be based on the separated regions and the contours determined from the Hessian feature image. In other embodiments, the processing circuit 542 can determine the respective positions in the patient-specific image data where the voxel intensity exceeds the threshold intensity value by determining the respective positions in the patient-specific image data that are greater than or less than a predetermined intensity value. In other words, the intensity can exceed (e.g., become higher than) a high threshold intensity value indicating that the voxel represents bone or exceed (e.g., become lower than) a low threshold intensity value indicating that the voxel represents fluid, adipose tissue, or other tissue indicating the boundary of the soft tissue structure. For example, the threshold intensity value can represent bone strength. Since soft tissue (e.g., muscle) exists against the bone surface, once the vector reaches the position of the bone, the processing circuit 542 infers this position as the boundary of the soft tissue structure. The threshold intensity value can also be set to a structure outside the muscle to identify where the vector has left the volume of the soft tissue structure. In other embodiments, the threshold intensity value can be less than the expected intensity value of the muscle. For example, once the threshold intensity in the path of the vector decreases below the threshold intensity value, the processing circuit 542 can interpret the lower intensity value as a different fluid or other structure from the soft tissue structure of interest. In some embodiments, the threshold intensity value can be of a predetermined magnitude.

[0141] In other embodiments, the threshold intensity value can be a difference calculated based on where the vector originated and / or the previous voxel or pixel that the vector most recently intersected. As such, the processing circuit 542 can identify relative changes in the intensity of the patient image data (e.g., voxel-to-voxel changes) or a filter mask that can indicate the boundaries of soft tissue structures or other structures associated with a portion of the contour 614 of the soft tissue structure of interest. For example, the processing circuit 542 can be configured to determine the corresponding locations in the patient-specific image data that exceed the threshold intensity value by determining the corresponding locations in the patient-specific image data that are greater than a difference threshold, where the difference threshold is the difference between the intensity associated with the corresponding surface point and the intensity at the corresponding location in the patient-specific image data. In some embodiments, the processing circuit 542 can employ one or more of these types of thresholds when analyzing the image data of the boundary of the soft tissue structure of interest.

[0142] Generally, the processing circuit 542 can identify the corresponding locations in the patient-specific image data that represent the boundaries of the soft tissue structure for each vector. However, in some embodiments, the processing circuit 542 may not identify the voxels or pixels that exceed the threshold intensity value. This problem can occur due to incomplete information in the patient-specific image data, data corruption, patient movement during the data generation process, or any other type of anomaly. The processing circuit 542 can employ algorithms to avoid this problem during the deformation process of the initial shape 616. For example, the processing circuit 542 can employ the maximum distance in the image data for which voxels or pixels that exceed the threshold intensity value are identified. The maximum distance can be a predetermined distance or a scaled distance from a point on the initial shape 616 where the vector begins. The maximum distance can be selected in the range from about 5 millimeters (mm) to about 50 mm. In one embodiment, the maximum distance can be set to about 20 mm.

[0143] If the processing circuit 542 does not identify voxels that exceed the threshold intensity value within this distance from the origin of the vector, the processing circuit 542 can then remove the vector and the corresponding point on the surface of the initial shape 616 from the deformation process and rely only on the other points and vectors for deformation. Alternatively, the processing circuit 542 can select a new surface point from the initial shape 616 and extend a new vector from the new surface point to attempt to find voxels that exceed the threshold intensity value or voxels that correspond to the contour or boundary of the soft tissue structure determined from the Hessian feature image. The new surface point on the surface of the initial shape 616 can be within or at a predetermined distance from the removed surface point, at a predetermined distance between the removed surface point and another surface point, or at some other location on the initial shape 616 that the processing circuit 542 selects to replace the removed surface point.

[0144] Then, the processing circuit 542 can deform the surface of the initial shape 616 using one or more iterations to cause the enclosed initial shape 616 to approximate or fit portions of the contour 614 and the scapula 612. During each iteration, the processing circuit 542 can move some or all of the surface points on the initial shape 616. For example, the processing circuit 542 can extend a corresponding vector 618 from each respective surface point of the plurality of surface points and the vector 618 is orthogonal to the surface of the initial shape 616 including the respective surface point. As discussed above, these vectors can point inwardly and / or outwardly from the surface of the initial shape 616. Then, the processing circuit 542 can determine corresponding points in the patient-specific image data that exceed a threshold intensity value for the corresponding vector 618 of each surface point. These points that exceed the threshold intensity value can form one or more contours similar to the contour 614 and / or the bone surface.

[0145] At each respective point of the contour 614, the processing circuit 542 can determine a plurality of potential positions that are located within the envelope of the respective point and exceed the threshold intensity value in the patient-specific image data. Thus, the plurality of potential positions of the single vector can at least partially define the surface of the contour 614. These potential positions within the envelope indicate the potential directions in which the surface points on the initial shape 616 should move. The processing circuit 542 selects one of these potential positions to guide the movement of the respective surface point to address how the surface of the contour at these potential positions is oriented relative to the surface points on the initial shape 616. In other words, the processing circuit 542 can select such a potential position such that the difference between the orientation of the surface of the initial shape 616 and the orientation of the surface of the contour 614 at the potential position is reduced.

[0146] For example, the processing circuit 542 can determine a corresponding normal vector orthogonal to the surface for each of the plurality of potential positions. Taking the vector 620 of the respective surface point on the surface of the initial shape 616 as an example, a number of normal vectors are generated from each respective potential position. An exemplary normal vector is the vector 622 from one potential position. Then, the processing circuit 542 can determine the angle between the corresponding normal vector and the vector from the respective surface point for each of the corresponding normal vectors. Using the embodiment of the vector 624, the processing circuit 542 can determine the angle 622 between the vector 624 and the vector 620. This angle 622 can be referred to as the cosine angle. The processing circuit 542 can perform this calculation for each potential position corresponding to a respective vector 618 such as the exemplary vector 620.

[0147] Then, the processing circuit 542 can select, for each respective surface point on the initial shape 616, such a potential position among a plurality of potential positions that has the minimum angle between the vector of the respective surface point (e.g., vector 620) and the respective normal vector of each of the plurality of potential positions (e.g., vector 624). In other words, the processing circuit 542 can identify positions on the contour 614 that provide appropriate movement of the surface points and deformation of the surface points. Then, the processing circuit 542 can cause each respective surface point to move at least partially toward the selected one potential position. This movement of the respective surface points causes the initial shape 616 to be modified or deformed toward a patient-specific shape corresponding to the contour 614 and a portion of the scapula 612. The processing circuit 542 can repeat this process at each iteration until the initial shape 616 has been deformed to approximate voxels or pixels that exceed a threshold intensity value or correspond to a previously segmented boundary of the soft tissue structure of interest.

[0148] The processing circuit 542 can cause the respective surface points to move at least half the distance between the respective surface points and the selected one potential position corresponding to the contour 614. However, in other cases or during the iterative process, the distance moved can vary. Since the deformation caused in a single step may not result in an accurate final patient-specific shape, the surface points may not move completely toward the potential position. In other words, the small adjustments made to the selected positions on the contour 614 based on the normal vectors of each potential position in each iteration can provide a final shape that more closely matches the contour 614. In other words, each surface point on the initial shape 616 may not necessarily move completely along a linear direction throughout the iterative process. This combination of non-linear movement of the individual surface points can allow the final patient-specific shape to more closely match or fit the contour 614 and, if applicable, the adjacent bone surface.

[0149] As Figure 10 shown in the sagittal view 640 Figure 9 in, the initial shape 616 has been deformed to an intermediate shape 632 during one iteration, and the intermediate shape 632 is closer to the contour 614 than the initial shape 616. When the processing circuit 542 determines that the intermediate shape 632 is still not close enough to the contour 614 or otherwise requires additional deformation, the processing circuit 542 can perform one or more additional iterative deformations. For example, the processing circuit 542 can determine vectors 634A and 634B (collectively "vector 634") from the respective surface points on the intermediate shape 632. Similar to the vector 618 described above with reference to Figure 9 the vector 634B points inward toward the surface of the scapula 612, and the vector 634B points outward toward portions of the scapula 612 and the contour 614.

[0150] Then, the processing circuit 542 determines another set of potential positions of each vector 634 at a point where the corresponding vector reaches a voxel or pixel with an intensity value exceeding a threshold. From each potential position of each vector 634, the processing circuit 542 can select the potential position of the normal vector having the smallest angle when compared with the vector of the surface point. For example, for the exemplary vector 636 of the surface point on the intermediate shape 632, the processing circuit 542 can determine the vector 637 as a vector from a potential position in the contour 614, which is within the envelope of the point where the vector 636 in the contour 614 reaches the contour 614. When the angle 638 is the smallest angle between the vector of the intermediate shape 632 and the normal vector of the potential position, the processing circuit 542 can select the potential position associated with the vector 637. Thus, the processing circuit 542 can select, at least for some vectors 634, positions on the contour 614 that are different from the points where the vectors 634 reach the contour 614. This process enables the processing circuit 542 to more closely approach the contour 614 by moving the surface points on the intermediate shape 632 in directions different from the direction orthogonal to the surface at the corresponding surface points.

[0151] Then, the processing circuit 542 can move the surface points of the intermediate shape 632 at least partially toward the selected potential positions on the contour 614. For example, the processing circuit 542 can deform the intermediate shape 632 into Figure 11 the fully enclosed final patient-specific shape 642 shown in Figure 11 . As shown in Figure 11 , the patient-specific shape 642 can approximate the contour 614 and at least some surfaces of the scapula 612 against which the soft tissue structure of the patient is arranged. In some embodiments, the patient-specific shape 642 can be identical or similar to the contour 614. The contour 614 can represent the initial segmentation of the soft tissue structure in the patient-specific image data and / or represent the surface of the super-threshold voxels identified from each vector. In other embodiments, before reaching the final patient-specific shape 642, the processing circuit 542 can deform the initial shape 616 more than twice. In some embodiments, the processing circuit 542 can perform a predetermined number of iterations to approach the contour of the voxels or pixels with an intensity value exceeding the threshold. In other embodiments, the processing circuit 542 can continue to perform additional iterative deformations of the initial shape until a certain number, a certain percentage, or all of the surface points of the shape are within a predetermined distance of the voxels or pixels above the threshold. In other words, the processing circuit 542 can continue to deform the initial shape and its intermediate shapes until the deformed shape is within a certain acceptable tolerance or error of the contour in the patient-specific image data.

[0152] In some embodiments, the processing circuitry 542 may follow the same deformation instructions in each iteration. Alternatively, the processing circuitry 542 may adjust one or more factors that determine how the surface points of the shape are moved during an iteration of the deformation process. For example, these factors may specify the number of surface points from an initial or intermediate shape, the number of potential positions identified within the contour, the envelope size of the potential positions that can be selected for each vector of the surface points, the distance that each surface point can move in one iteration, the allowable deviation that each surface point can move relative to each other in one iteration, or other such factors. These types of factors can limit the extent to which the processing circuitry 542 deforms the initial or intermediate shape in a single iteration.

[0153] For example, the processing circuitry 542 may deform the initial shape in a more uniform manner and deform the intermediate shape in a less uniform manner to more closely approximate the actual scale of the soft tissue structure identified in the image data. In one embodiment, the processing circuitry 542 may be configured to generate a second shape (e.g., intermediate shape 632) by moving each of the plurality of surface points from the initial shape 616 a first corresponding distance in a first iteration such that the plurality of surface points iteratively move toward corresponding potential positions of one or more contours (e.g., contour 614), the first corresponding distance being within a first tolerance of a first modified distance. The first tolerance may be selected by the processing circuitry 542, the user, or may be otherwise predetermined to maintain the smoothness of the second shape relative to the initial shape 616. In other words, the tolerance may be an allowable deviation from a value. The tolerance may be as low as zero such that all surface points must move the same distance. However, the tolerance may be larger to allow the processing circuitry 542 to vary the distance that each surface point can move in an iteration.

[0154] Then, the processing circuit 542 may cause each surface point among the plurality of surface points of the second shape to move a second corresponding distance in a second iteration after the first iteration to generate a third shape (e.g., another intermediate shape or a final shape) from the second shape, the second corresponding distance being within a second tolerance of the second modification distance. In this second iteration, the second tolerance is greater than the first tolerance to enable a greater change in the distance by which each surface point moves than was allowed in the previous iteration. Thus, a smaller tolerance can promote smoothness when deforming the shape, while a larger tolerance can promote further precision in the proximity of the shape of the next deformation to the soft tissue structure. Generally, the processing circuit 542 may increase the tolerance of the modification distance of the surface points between iterations. However, in some embodiments, the processing circuit 542 may switch between increasing or decreasing the tolerance, or the processing circuit 542 may maintain the same value of tolerance for each iteration. In other words, the elasticity of later iterations may increase to enable each surface point to move more precisely to its corresponding point (e.g., towards the contour representing the boundary of the soft tissue structure). Additionally, since the system becomes more confident in the correspondence with the contour 614, the search distance may increase for each iteration. For a more confident correspondence, such as the correspondence of the portion of the contour 614 associated with the bone structure, etc., the system may use a higher iteration that provides a more elastic or tolerant movement for each surface point on the initial shape 616. Conversely, since the confidence in the correspondence with the contour 614 is lower, the portion of the contour 614 associated with other soft tissues may require an initial iteration with lower elasticity.

[0155] In some embodiments, this process of registration and modification may be similar to the B-spline algorithm. The internal parameters of the B-spline algorithm may include the spline order, the number of control points, and the number of modification iterations of the shape. The first two parameters of the spline order and the number of control points can control the "elasticity" degree of the algorithm. The higher the spline order and the number of control points, the more likely each surface point on the shape is to exhibit elastic behavior. In other words, as long as the confidence in the correspondence with the contour is maintained at a relatively high level, later iterations result in a more specific solution for the modified shape.

[0156] In one embodiment, for the first iteration, the algorithm is capable of dividing the space of the initial shape 616 into a plurality of surface points. The processing circuit 542 can determine the deformation field based on these surface points and the spline order using a spline relationship. For further iterations, the number of surface points can be doubled, tripled, etc., to enable a more specific deformation field for changing the initial shape 616. Each "iteration" may refer to an internal iteration of the B-spline algorithm. The processing circuit 542 can use an external iteration to perform the registration of the initial shape or the modified shape, which has "N" internal iterations. "N" may increase with the external iteration to enable a more resilient output. An exemplary manner in which the processing circuit 542 performs B-spline registration is described in Lee et al., "Scattered Data Interpolation with Multilevel B-Splines", IEEE Transactions on Visualization and Computer Graphics, Vol. 3, No. 3, July - September, 1997. In https: / / itk.org / Doxygen411 / html / classitk_1_1BSplineScatteredDataPointSetToI mageFilter.html and http: / / www.insight-journal.org / browse / publication / 57. You can find the execution of b among them additional exemplary ways of spline registration.

[0157] In some embodiments, the processing circuit 542 can move the surface points of the initial shape or the intermediate shape a greater distance or the entire distance towards the contour based on the identified intensity value of the voxel or pixel at that location. For example, a voxel with a higher intensity can indicate the presence of bone. Generally, soft tissue structures can be arranged against a part of the bone. Therefore, if a voxel is identified as bone, the processing circuit 542 can move the corresponding surface point of the initial shape or the intermediate shape directly to or adjacent to the identified bone structure. In other embodiments, when bone is identified as part of the contour, the processing circuit 542 can increase the tolerance of the modification distance so that the next iteration can more precisely approach the contour of the bone. As discussed herein, in other embodiments, the contour 614 can be determined based on the Hessian feature image representing the separation zone between adjacent structures. In some embodiments, the processing circuit 542 can track the distribution (profile, curve) behavior of the Hessian feature image along a vector to determine the correspondence with the boundary of the soft tissue structure. The Hessian feature image can include a distribution similar to the moment shape function that provides voxels for the correspondence of the vector. For a bone structure, the processing circuit 542 can know the voxels on the bone surface so that the surface point is directly moved to that voxel.

[0158] Once the final patient-specific shape 642 is determined, the processing circuitry 542 can output the patient-specific shape 642. In some embodiments, the processing circuitry 542 can control the patient-specific shape 642 presented to the user. In other embodiments, the processing circuitry 542 can perform additional calculations on the patient-specific shape 642. For example, the processing circuitry 542 can determine the volume, linear dimensions, cross-sectional dimensions, or other characteristics of the patient-specific shape 642. As described herein, the processing circuitry 542 can use these characteristics in other determinations.

[0159] Figures 12 to 1 8 shows an exemplary modeling of the rotator cuff muscles based on the deformation process described herein. In some embodiments, a system such as system 540 can display similar images to the user via a user interface. Some of the views are two-dimensional, while other views are three-dimensional of the same modeled structure. Figure 12 is a conceptual diagram of an exemplary axial view 650 of patient image data including the scapula 652. The initial shape 654 represents the SMS of the scapular muscles, which has been deformed into a patient-specific shape 656 representing the scapular muscles of the patient. Figure 13 is a conceptual diagram of an exemplary sagittal view 660 of patient image data including the scapula 662. The initial shape 664 represents the SMS of the supraspinatus muscle, which has been deformed into a patient-specific shape 666 representing the supraspinatus muscle of the patient.

[0160] Figure 14 is a conceptual axial view 670 showing exemplary final patient-specific shapes 676, 678, and 680 of three rotator cuff muscles superimposed on patient-specific image data. As Figure 15 shown, the scapula 672 is shown relative to the humeral head 674. Specifically, the patient-specific shape 676 represents the subscapularis muscle, the patient-specific shape 678 represents the supraspinatus muscle, and the patient-specific shape 680 represents the infraspinatus muscle. As Figure 15 shown, a sagittal view 682 of the patient image data includes the scapula 673 relative to the patient-specific shape 676 (e.g., subscapularis muscle), the patient-specific shape 678 (e.g., supraspinatus muscle), and the patient-specific shape 680 (e.g., infraspinatus muscle). It should be noted that additional rotator cuff muscles or other muscles associated with the joint of interest may be determined but are not shown, although they can be in other embodiments.

[0161] Figure 16A is a conceptual posterior three-dimensional view 690 showing an example final patient-specific shape of three rotator cuff muscles and the bones from patient-specific image data. As Figure 16AAs shown, the subscapularis muscle represented by the patient-specific shape 676 relative to the scapula 692 and the humeral head 694 is shown. View 690 also shows the supraspinatus muscle as the patient-specific shape 678 and the infraspinatus muscle as the patient-specific shape 680. As Figure 16B shown, the front view 702 is a three-dimensional view of a similar structure shown in the rear view 690, such as the subscapularis muscle as the patient-specific shape 676, the supraspinatus muscle as the patient-specific shape 678, and the infraspinatus muscle as the patient-specific shape 680, etc. As Figure 17 shown, the end view 704 is a three-dimensional view of a similar structure shown in the rear view 690. The end view 704 is a view of these shoulder structures in a plane aligned with the glenoid surface 706. Thus, the subscapularis muscle as the patient-specific shape 676 is shown, the supraspinatus muscle as the patient-specific shape 678 between the coracoid process 708 and the acromion 710 is shown, and the infraspinatus muscle as the patient-specific shape 680 is shown.

[0162] Figure 18A With Figure 18B is a conceptual diagram of exemplary patient-specific CT data in which an initial shape associated with a soft tissue structure is registered with a bone structure and modified to represent a patient-specific shape of the soft tissue structure. For example, the processing circuit 542 can initially identify the bone structure (e.g., one or more x-ray images) within the CT data, i.e., which may be a basic landmark for registering the initial shape of the soft tissue structure and then scaling the initial shape to fit the patient's CT data.

[0163] As Figure 18A shown in the axial slice embodiment of, the processing circuit 542 can determine a set of basic landmarks 713A, 713B, and 713C (collectively "landmarks 713") from the patient image based on the target muscle (e.g., soft tissue structure). For example, the subscapularis muscle is adjacent to the thoracic cage, and thus, the ribs may be segmented and used to locate the landmark (the rib of landmarks 713). The initial shape 715 can be the statistical mean shape (SMS) of the subscapularis muscle that is strictly registered with the patient. The initial shape 715 can be a pathological shape, such as an SMS generated from other patients with similar health conditions to the patient. In other words, since the SMS is only registered and scaled, a healthy SMS may not provide an appropriate model of the patient's soft tissue. However, in some embodiments, the registered and / or scaled SMS can be further modified according to the closed surface modification described herein Figures 8A to 11 As described.

[0164] The connections 714A, 714B, and 714C (collectively "connections 714") indicate the correspondence between the landmarks 713 and the corresponding points on the SMS of the initial shape 715.Figure 18A is an axial slice of patient-specific CT data.

[0165] Figure 18B is a sagittal slice of patient-specific CT data and shows different views of the rib and the subscapularis muscle. The single rib 716 is a landmark that includes a number of points corresponding to the corresponding points on the initial shape 715, and the connection 717 is a line indicating these correspondences. Figure 18A and Figure 18B is a two-dimensional representation of the registration process. In some embodiments, three-dimensional registration can be performed by the processing circuit 542. The process described with respect to the subscapularis muscle can also be performed on other muscles. For example, landmarks related to the supraspinatus muscle can be identified on the lower side of the clavicle and the acromion, and landmarks related to the infraspinatus muscle can be identified based on the surrounding skin.

[0166] Generally, once the basic landmarks (e.g., landmarks 713 and 716) are identified, their closest correspondences on the SMS are located. The processing circuit 542 can determine an intensity-based curve along the line connecting landmark '1' to its corresponding point on SMS 'c' (e.g., connecting 713 and 717). When a specific change in the curve is detected at a certain position 'v', the Euclidean distance (d) between 'v' and 'c' and the intensity value (i) of 'v' are stored. Thus, the processing circuit 542 can use the following equation:

[0167] Cf = fun(d n , i n ), n ∈ landmarks(1)

[0168] The intensity-based metric can be only the intensity value of the CT image data or its gradient. For intensity, a specific change can be a step from high (bone density) to low (soft tissue structure), and for the gradient embodiment, there may be a positive spike along the curve connecting indicating the boundary of the soft tissue structure. The processing circuit 542 can use a minimization algorithm such as a cost function to determine the registration of the initial shape 715. The minimization algorithm can refer to a general type of algorithm, where the processing circuit 542 deforms the initial shape 715 (e.g., SMS) by satisfying the threshold of the algorithm so that the deformed version of the SMS fits the bone-to-muscle scale of the soft tissue structure. For example, the cost function can be a combination of the Euclidean distance d = ||v - c|| and i; for example,

[0169] Cf = ∑ n w1 × d n + w2 × i n , (2)

[0170] where w1 and w2 are empirically determined weights.

[0171] The cost function can also have another term that is independent of the patient and is used to smooth the final estimate of the registration. This term is called the regularization term and can provide adaptive weights for the parameters of the initial shape 715 (based on their relevance and the noise in the SMS). This term (Cf2) can be added to the previous "difference" term (now called (Cf1)) and the optimization gives the sum:

[0172] Cf = Cf1 + Cf2 (3)

[0173] For comparison, Cf1 can be in the same scale range as Cf2. Cf2 can normalize its value (i.e., between 0 and 1). To scale Cf into the same interval, after strict registration (i.e., between the SMS and the patient-specific CT data), the Cf1 value is used for normalization.

[0174] Once the initial shape 715 is strictly registered with the patient, the initial shape 715 can be elastically deformed to match the soft tissue structure of the patient using its finite parameter equation:

[0175]

[0176] where s’ is the initial shape (e.g., the SMS), λ i is the eigenvalue and v i is the eigenvector of the corresponding covariance matrix (e.g., also called the mode of variation). The covariance matrix represents the variables of the data set, such as the variables in the patient-specific patient data, etc. b i 's value determines the shape of s (e.g., the final patient-specific shape of the soft tissue structure). The term b i is the scaling factor of the initial shape 715. The processing circuit 542 can use this process to find the value of b i so as to provide the final shape (s) that estimates the patient-specific structure of the target muscle. For example, the processing circuit 542 performs its best fit so as to minimize the cost function that defines the "difference" between s and the patient muscle (e.g., m) in the patient-specific CT image. In other words, the optimization algorithm can minimize Cf = |s - m| or maximize Cf’ = |s - m| -1 maximizes.

[0177] Since it is based on its convexity, it may now be important to define Cf or Cf’, and the optimizer will fall or not fall into the global or local minimum or maximum. To obtain a good estimate at the end of the optimization process, the processing circuit 542 can determine Cf that reflects the "shape difference" between the modified final object (s) and the target (m). The variable for calculating this difference may be related to the Euclidean distance estimated from the final shape to a finite number of fiducial marks located on the patient.

[0178] The processing circuit 542 can use an optimization algorithm that minimizes a cost function by applying the SMS parameter equation iteratively and changing the parameter values based on the value of Cf after each iteration. When no further optimization of Cf can be performed (minimized or maximized), i.e., the optimal value is reached, or when the maximum number of iterations is reached, the processing circuit 542 can stop the loop. When the optimization algorithm is completed, the processing circuit 542 can cause the modified initial shape 715 to ultimately become the final patient-specific shape of the soft tissue structure. The minimization or maximization algorithm can refer to a general type of algorithm in which the processing circuit 542 deforms the initial shape 715 (e.g., SMS) by meeting the threshold of the algorithm to fit the deformed version of the SMS to the bone-to-muscle scale of the soft tissue structure. The threshold can indicate when the deformed version of the SMS is the best fit or reduces the error with the soft tissue structure in the patient-specific image data.

[0179] Figure 19 Is a conceptual diagram of an exemplary final patient-specific shape that is masked and thresholded to determine fat infiltration. Fat infiltration, or the value representing the ratio or volume of fat within muscle, can indicate changes in the structure and / or function relative to other healthy muscle. Thus, the amount of fat within muscle can indicate the health status of that muscle. Furthermore, the health status of the muscle can affect what type of joint treatment is suitable for the patient.

[0180] As discussed herein, systems such as system 540 can determine the fat infiltration value of the modeled soft tissue structure relative to a threshold intensity value from patient-specific image data within a representation of the soft tissue structure based on thresholded voxels or groups of voxels. The ratio of adipose tissue to all tissue within the representation can be determined as the fat infiltration value. System 540 can determine the fat infiltration value from the pixels of multiple two-dimensional slices of patient-specific image data or from the voxels of a three-dimensional image dataset. Here, a three-dimensional scheme is described as an example, but for illustrative purposes, two-dimensional views are used. For example, for CT image data, the intensity threshold for fat can be approximately -29 Hounsfield units (HU) and the intensity threshold for muscle can be approximately 160 HU. In one embodiment, the fat infiltration (FI) value can be calculated as: FI = 100*(1 - x / X), where x is the muscle volume within the mask and X is the total volume within the mask. In some embodiments, regions with fat or muscle less than a specified threshold (e.g., less than one cubic millimeter) can be considered noise and eliminated.

[0181] As Figure 19As shown, the patient image data 720 includes a patient-specific shape 722 for the muscle of interest generated using a mask applied to the voxels. First, the system can apply a mask to the patient-specific shape 722 to remove data outside the patient-specific shape 722. Next, the system can apply a threshold to the voxels under the mask. In some embodiments, a threshold can be applied to the intensity of a group of voxels or the average voxel intensity of two or more voxels to avoid noise that may be present in a single voxel. Thus, the system can analyze the intensity and / or spatial characteristics of the patient-specific image data to determine regions of adipose tissue. The black regions indicated by the voxels 724 indicate muscle tissue above the threshold intensity indicating muscle tissue. In contrast, the lighter voxels 726 are below the threshold, have a lower intensity, and indicate adipose tissue. In the absence of the mask not shown, the intensity of the adipose tissue voxels would be lower than the intensity of the voxels associated with muscle. Then, the system can determine the adipose volume of the soft tissue structure by adding the voxels 726 below the threshold. Then, the system can determine the adipose infiltration value based on the adipose volume and the total volume of the patient-specific shape 722 of the soft tissue structure. For example, the system 540 can divide the adipose volume by the total volume to determine the adipose ratio (e.g., percentage) of the soft tissue structure. Then, the system 540 can output the adipose volume ratio of the soft tissue structure. In some embodiments, when determining what type of joint replacement may be suitable for a patient, the system 540 can use the adipose volume ratio as an input. In some embodiments, the system 540 can use muscle mass indicators such as adipose infiltration and range of motion to determine the positioning and / or orientation of implant components such as humeral implants or glenoid implants. For example, the system 540 can recommend moving one or more implants laterally or centrally to improve the range of motion and / or strength of the patient's shoulder. Because the strength or flexibility of the muscle may depend at least in part on the distance between the glenoid and the humeral head, changing the position of the humeral implant (or selecting a different size humeral implant) such that the humerus is moved closer to or farther from the glenoid can enable the clinician to improve the range of motion and / or strength of the shoulder after implantation. In some embodiments, bone grafting can be used to add to the humeral head or glenoid to achieve the desired lateralization or centralization of the humerus (with or without a humeral implant).

[0182] Figure 20 is a conceptual diagram of an exemplary final patient-specific shape 736 and pre-disease prediction 734 of the soft tissue structure. As Figure 20 shown in the embodiment of, the sagittal view 730 includes the scapula 732 and the patient-specific shape 736 of the subscapularis muscle. The patient-specific shape 736 can be determined based on the closed surface fitting described above. However, it can provide useful information when selecting the type of joint treatment and understanding how the muscle changes from a healthy state to a pre-disease state. Thus, the system 540 can determine the atrophy rate of the muscle.

[0183] For example, the processing circuitry 542 of system 540 can be configured to determine the bone-to-muscle scale of a patient's soft tissue structure. The processing circuitry 542 can determine the length, width, and / or volume of the scale of a bone, such as the scapula 732, relative to muscle according to the patient-specific shape 736. The bone-to-muscle scale can identify the specific anatomical structure dimensions of the patient. The processing circuitry 542 can obtain a statistical mean shape (SMS) of the soft tissue structure. The SMS can be a representation of a typical muscle calculated based on a large number of healthy subjects. However, in some embodiments, an SMS of a pathological structure can be used in some embodiments.

[0184] Next, the processing circuitry 542 can use a minimization algorithm to deform the SMS so that the SMS fits the bone-to-muscle scale of the soft tissue structure. For example, the processing circuitry 542 can modify the SMS to fit more tightly to the patient's bone-to-muscle scale and thereby estimate the pre-morbid state of the muscle. The resulting pre-morbid prediction 734 can be used as an estimate of the health state of the muscle. Then, the processing circuitry 542 can determine the atrophy rate of the soft tissue structure (e.g., a muscle such as the supraspinatus muscle represented by the patient-specific shape 736) by dividing the volume of the deformed SMS by the volume of the soft tissue structure represented by the patient-specific shape 736. The result is the atrophy rate of the soft tissue structure. In some embodiments, in addition to the calculations during the pre-operative planning process, the processing circuitry 542 can output the atrophy rate for display or further use.

[0185] Figure 21 And Figure 22 is a conceptual diagram of exemplary spring-modeling muscles contributing to the range of motion analysis of the shoulder joint. Although Figure 21 and Figure 22 illustrate the pre-operative planning of a reverse shoulder replacement, however, in other embodiments, an anatomical shoulder replacement can be planned. As Figure 21 shown, the virtual view 740 includes a posterior view of the humeral head 744 and scapula 742 on the left side and an anterior view of the humeral head 744 and scapula 742 on the right side. The humeral head has been fitted with a spacer 748 configured to mate the already-fitted glenoid sphere 750 with the glenoid surface. Other components may also be involved in implants such as cups and plates like the spacer 748. In an anatomical shoulder replacement, these components are inverted so that the implant sphere is instead attached to the humeral head 744. Some exemplary components can include a reverse glenoid component having a wedge-shaped substrate, a central screw, a glass sphere, and a symmetric graft. In a reverse shoulder replacement, a stemless retrograde (Ascend) FLEX reverse humeral component with a standard insert and a 1.5 offset disc can also be used.

[0186] For example, when the shoulder joint has deteriorated, system 540 can perform a determination as to whether the patient should receive an anatomical or reverse shoulder replacement. Part of this determination can include consideration of engaging one or more muscles to move the joint. In one embodiment, the processing circuitry 542 of system 540 can model three rotator cuff muscles as springs having spring constants K1, K2, and K3. As Figure 21 shown, the infraspinatus muscle has been modeled as spring 746 and the subscapularis muscle has been modeled as spring 747. In other embodiments, fewer or additional muscles can be modeled as part of this analysis. As Figure 22 shown, virtual view 760 shows the subscapularis muscle modeled as spring 762. The processing circuitry 542 is capable of assigning spring constants to the respective springs, such as assigning spring constants K1, K2, and K3 to springs 746, 747, and 762, etc.

[0187] As described herein, the processing circuitry 542 can determine each spring constant based on the calculated fat infiltration (e.g., fat volume ratio) and atrophy rate of the respective muscles that have been modeled. For example, the processing circuitry 542 can employ an equation such as K = f(R(FI), R(A)) to determine the spring constant of each muscle, where K is the spring constant, FI is the fat infiltration, and A is the atrophy of the respective muscle. In other embodiments, additional factors can be used when determining the spring constant, such as the overall volume, length, and / or cross-sectional thickness of the patient-specific shape of the muscle, patient age, patient gender, the patient's injury history, or any other type of factor that may affect the function of the muscle, etc. For example, the processing circuitry 542 can determine the attachment points of the respective springs 746, 747, and 762 based on the insertion points of the muscles. In some embodiments, each muscle can be represented by two or more than two springs, such as different springs representing the respective attachment points from the muscle to the bone. In other embodiments, each muscle can be represented by a more complex model of muscle function other than a spring. In some embodiments, the processing circuitry 542 can determine the load applied to the spring. The load can combine the weight of the bone structure with an external load (e.g., lifting a standard object). Although springs can be used as some models, however, in other embodiments, a finite element model can be used.

[0188] The processing circuit 542 can determine the range of motion of the patient's humerus by determining the range of motion of the patient's humerus based on the fat volume ratio and atrophy rate of one or more corresponding muscles of the patient's rotator cuff. For example, the processing circuit 542 can calculate each spring constant K1, K2, and K3, and then determine the range of motion of the humerus relative to the scapula. In some embodiments, the processing circuit 542 can determine the range of motion in one or more planes or in three dimensions. The range of the axis of motion can be predefined, and in some embodiments, when soft tissue is not a limiting factor along these axes, bone collisions can be determined to establish the range of motion for some angles. The processing circuit 542 can perform this calculation for each possible type of treatment, such as anatomic replacement or reverse replacement, etc. In other embodiments, the processing circuit 542 can perform a range of motion analysis on the currently damaged joint and bones and use this calculation to identify which type of treatment is suitable for the patient. In any case, the processing circuit 542 can determine which type of shoulder treatment, such as anatomic shoulder replacement or reverse shoulder replacement, etc., should be selected for the patient and then output the selected type of shoulder treatment for display. The processing circuit 542 can present the analysis of each type of treatment to the user, such as presenting a numerical score or calculation of each type of treatment. Then, the user can determine which type of treatment to provide for the patient from this presented information of each type of treatment.

[0189] Figure 23A is a flowchart showing an exemplary program for modeling soft tissue structures using patient-specific image data according to the techniques of the present disclosure. The processing circuit 542 of the system 540 is described as performing Figure 23A the embodiments in, however, other devices or systems, such as the virtual planning system 102, etc., can perform one or more parts of the present technology. Further, some parts of the present technology can be performed by a combination of two or more than two devices and / or systems via a distributed system. Figure 23A The embodiments of Figures 8A to 11 can be similar to the example diagrams and discussions referred to above

[0190] As Figure 23A shown, the processing circuit 542 can obtain patient-specific image data (800) of a patient of interest. The patient-specific image data can be generated by one or more imaging modalities (e.g., x-ray, CT, MRI, etc.) and stored in a data storage device. Then, the processing circuit 542 obtains an initial shape (802) of the soft tissue structure of interest. The initial shape can be a geometric shape or a statistical mean shape (SMS). The soft tissue structure can be a muscle or other non-bone structure. However, in other embodiments, the Figure 23AThe process or other techniques in []. Then, the processing circuit 542 registers the initial shape with the patient-specific image data (804). This registration may include registering the initial shape with the bone insertion points and / or bones identified by the segmented bones in the patient-specific image data. In other embodiments, where preliminary muscle segmentation has been performed on the soft tissue structures of interest in the patient-specific image data, the processing circuit 542 may register the initial shape with the preliminary muscle segmentation.

[0191] Then, the processing circuit 542 identifies one or more contours in the patient-specific image data that represent the boundaries of the soft tissue structures (806). The one or more contours may be identified as voxels associated with the segmented bones and / or muscles in the patient-specific image data. In other embodiments, the processing circuit 542 may determine each contour by extending normal vectors from the surface of the initial shape inward and / or outward. The voxels or pixels that are encountered by each vector and exceed a threshold intensity value in the patient-specific image data may be identified as defining at least a portion of the contour.

[0192] Then, the processing circuit 542 moves the surface points on the surface of the initial shape toward the corresponding points on the one or more contours (808). The movement of these surface points causes the entire surface of the initial shape to deform. If the processing circuit 542 determines that the surface points need to be moved again to more tightly fit the initial shape to the one or more contours (the "yes" branch in block diagram 810), the processing circuit 542 moves the surface points of the deformed surface of the initial shape again (808). If the processing circuit 542 determines that the surface points do not need to be moved again and the deformed shape fits the one or more contours (the "no" branch in block diagram 810), the processing circuit 542 outputs the final deformed shape as the patient-specific shape representing the soft tissue structure of the patient (812). The patient-specific shape may be presented via a user interface and / or used for further analysis, such as as part of a preoperative plan for treating the patient.

[0193] Figure 23B is a flowchart showing another exemplary procedure for modeling soft tissue structures using patient-specific image data according to the techniques of the present disclosure. The processing circuit 542 of the system 540 is described as performing Figure 23B the embodiments in []. However, other devices or systems, such as the virtual planning system 102, may perform one or more parts of the present technique. Further, a combination of two or more than two devices and / or systems may perform some parts of the present technique via a distributed system. Figure 23B The embodiments in [] may be similar to the illustrations and discussions above with reference to Figures 8A to 11 In some aspects, Figure 23B the techniques in [] may also be related to Figure 23Ais similar to the technology in

[0194] As Figure 23B shown in, the processing circuit 542 can obtain patient-specific image data (820) of a patient of interest. The patient-specific image data can be generated by one or more imaging modalities (e.g., x-ray, CT, MRI, etc.) and stored in a data storage device. Then, the processing circuit 542 obtains an initial shape, which is an SMS (821) of a soft tissue structure of interest. Then, the processing circuit 542 registers the SMS with the insertion points of one or more corresponding bones of the patient-specific image data (822). In other embodiments, the registration may include registering the SMS with a preliminary muscle segmentation (in addition to the bone insertion points) in the patient-specific image data.

[0195] Then, the processing circuit 542 selects a plurality of surface points around the surface of the SMS and determines vectors orthogonal to the surface of each surface point (823). The surface points can be evenly distributed around the surface of the SMS at a predetermined density, predetermined interval, and / or according to other selection factors. In some embodiments, the processing circuit 542 can direct these normal vectors both outwardly and inwardly from at least some of the surface points. For each of these vectors, the processing circuit 542 determines points in the patient-specific image data that exceed a threshold intensity value and potential positions within the envelope located at that point (824). The determined points can be voxels, pixels, or points in space associated with the voxel or pixel. The determined points can correspond to the outer surface (e.g., one or more contours) of the identified soft tissue structure of interest within the patient-specific image data. The potential positions within the envelope line are those positions that also exceed the threshold and are part of one or more contours. The envelope line can be determined as a predetermined distance from the point identified by the vector, the number of potential positions adjacent to the point (e.g., the eight potential positions closest to the point), or other such criteria.

[0196] The potential positions are analyzed to identify changes in the contours that fit the SMS. In other words, the potential positions are more precisely analyzed in the direction in which the surface points of the SMS are moved. The processing circuit 542 determines, for each vector, the potential position having the smallest angular difference between its vector and the vector of the surface point (825). This angle can be referred to as the cosine between the two vectors. Figure 9The angle 622 therein is an example of this angle between the vector 620 of the surface point and the vector 624 of the potential position on the contour. After selecting the potential position for each surface point on the SMS, the processing circuit 542 moves each surface point a certain distance (826) towards the corresponding potential position on the contour. The distance moved can be a part, or a percentage, of the total distance to the potential position. In one embodiment, the distance is approximately half of the total distance to the potential position. Thus, each iteration brings the surface points closer to one or more contours, however, the distance of each iteration gets smaller and smaller. In other embodiments, the distance can be less than half of the total distance to the potential position or greater than half of the total distance to the potential position.

[0197] The movement of these surface points causes the entire surface of the SMS to deform. If the processing circuit 542 determines that the surface points need to be moved again to fit the SMS more tightly to one or more contours (the "Yes" branch in block diagram 827), the processing circuit 542 updates the SMS shape change function (828) before re-determining the points that exceed the threshold of each vector and the SMS surface points. The SMS change function can define how the processing circuit 542 deforms the SMS in this iteration. For example, the SMS change function can define how the deformation of the SMS balances the "smoothness" and "precision" of the next shape. For example, the early deformations may be smoother, or more uniform, than the later deformations, and the later deformations can optimize the deformation precision of one or more contours in the patient-specific image data.

[0198] In one embodiment, the SMS change function can utilize a tolerance factor that defines the amount of movement by which each surface point deviates from another surface point. For example, zero tolerance can indicate that all surface points must move the same distance in this iteration. A larger tolerance can allow the surface points to move different distances, which may result in a lower smoothness of the deformed SMS, however, a higher precision towards the contours in the patient-specific image data. In some embodiments, the SMS change function can specify different tolerances for different threshold intensities. For example, if the points in the patient-specific image data exceed the threshold indicating bone, the SMS change function can specify a larger tolerance because it can be expected that the muscle is arranged against the bone surface, so the processing circuit 542 can move the surface point closer to the bone surface. In some embodiments, the processing circuit 542 may not change the SMS shape change function between two iterations.

[0199] If the processing circuit 542 determines that the surface points do not need to be moved again and the deformed SMS shape fits one or more contours (the "No" branch in block diagram 827), the processing circuit 542 then outputs the final deformed shape as the patient-specific shape representing the soft tissue structure of the patient (829). As Figure 25 、Figure 26 and Figure 27 As described in one or more embodiments of Figure 27 , the patient-specific shape can be presented via a user interface and / or used for further analysis, such as being part of a preoperative plan for treating the patient.

[0200] Figure 24 is a flowchart showing an exemplary procedure for modeling soft tissue structures using patient-specific image data according to the techniques of the present disclosure. The processing circuit 542 of system 540 is described as performing Figure 24 the embodiments of Figure 24 , however, other devices or systems, such as the virtual planning system 102, may perform one or more parts of the present technology. Further, a combination of two or more than two devices and / or systems may perform some parts of the present technology via a distributed system. Figure 24 The embodiments of Figure 24 may be similar to the illustrations and discussions above with reference to Figure 18A and Figure 18B the illustrations and discussions of Figure 18B .

[0201] As Figure 24 shown in Figure 24 , the processing circuit 542 can obtain patient-specific image data (830) of a patient of interest. The patient-specific image data can be generated by one or more imaging modalities (e.g., x-ray, CT, MRI, etc.) and stored in a data storage device. Then, the processing circuit 542 obtains an initial shape (831) of the soft tissue structure of interest. The initial shape can be a geometric shape or a statistical mean shape (SMS). For example, the SMS can be a pathological shape to capture a health condition similar to the patient. The soft tissue structure can be a muscle or other non-skeletal structure. Then, the processing circuit 542 registers the initial shape with one or more positions associated with one or more bones of the patient-specific CT data (832). The registration can include registering points on the initial shape with corresponding points or positions associated with adjacent bones.

[0202] Then, the processing circuit 542 determines the correspondence (833) between each position of the bone and the corresponding point on the initial shape. Then, the processing circuit 542 determines the distance (834) between each position and the corresponding point on the initial shape based on the intensity curve of the correspondence. For example, intensity or gradient curves can be used to identify the boundary of the soft tissue structure in the patient-specific CT data. The processing circuit 542 can use a cost function to fit the initial shape to all available basic positions of the bone.

[0203] Then, the processing circuitry 542 may select a scaling factor (835) that minimizes the difference between the initial shape and the changes in the patient-specific CT data. For example, the processing circuitry 542 may analyze different scaling factors and use a cost function to obtain the best fit between the initial shape and the patient-specific CT data. Then, the processing circuitry 542 may output a final patient-specific shape representative of the soft tissue structure of the patient. For example, the processing circuitry 542 may perform this analysis on several muscles associated with the shoulder to be replaced.

[0204] Figure 25 is a flow diagram showing an exemplary procedure for determining the fat infiltration value of the soft tissue structure of a patient according to the techniques of the present disclosure. The processing circuitry 542 of the system 540 is described as performing Figure 25 the embodiments in, however, other devices or systems, such as the virtual planning system 102, may perform one or more portions of the present techniques. Further, combinations of two or more than two devices and / or systems may perform some portions of the present techniques via a distributed system. Figure 25 The embodiments in may be similar to the example diagrams and discussions referenced above Figure 19 Regarding the three-dimensional data set, the Figure 25 processes in are described, however, in other embodiments, several two-dimensional slice data may be analyzed in a similar manner.

[0205] As Figure 25As shown in FIG. 542, the processing circuit 542 may obtain or receive the final patient-specific shape (840) of the patient's soft tissue structure. Then, the processing circuit 542 applies a mask (842) to the patient-specific shape. The mask may remove data outside the patient-specific shape. Next, the processing circuit 542 may apply a threshold (844) to the voxels or volume under the mask. In some embodiments, the processing circuit 542 may apply a threshold to the average intensity of two or more voxels and / or groups of voxels to determine whether the group of voxels should be identified as adipose tissue. This grouping of voxels may reduce the influence of noise from determining whether a voxel is adipose tissue. Then, the processing circuit 542 determines the adipose volume of the soft tissue structure (846) by adding voxels having intensity values determined to be below the threshold. In other words, voxels determined to be less than the threshold are determined to be adipose tissue and voxels greater than the threshold are determined to be muscle. Then, the processing circuit 542 determines an adipose infiltration value (848) based on the adipose volume (i.e., the number of voxels having intensity values less than the intensity threshold, representing the adipose volume) and the total volume (i.e., all represented by all voxels in the masked patient-specific shape, including the number of voxels having intensity values less than the intensity threshold and the number of voxels having intensity values greater than or equal to the intensity threshold) of the patient-specific shape of the soft tissue structure. For example, the processing circuit 542 can divide the adipose volume by the total volume of the patient-specific shape to determine the adipose ratio of the soft tissue structure. The processing circuit 542 may calculate the total volume of the patient-specific shape or obtain the previously calculated volume. Then, the processing circuit 542 may output the adipose volume ratio of the soft tissue structure as the adipose infiltration value (850). The adipose volume ratio may be presented via a user interface and / or used for additional analysis.

[0206] Figure 25 FIG. 4 is a flowchart showing an exemplary procedure for determining the atrophy rate of a patient's soft tissue structure according to the techniques of the present disclosure. The processing circuit 542 of the system 540 is described as performing Figure 26 the embodiments in FIG. 4, however, other devices or systems such as the virtual planning system 102 may perform one or more parts of the present technology. Further, a combination of two or more devices and / or systems may perform some parts of the present technology via a distributed system. Figure 26 The embodiments in FIG. 4 may be similar to the example diagrams and discussions referenced above with respect to Figure 20 FIG. 3. The processes in FIG. 4 are described with reference to a three-dimensional data set, however, in other embodiments, several two-dimensional slice data may be analyzed in a similar manner. Figure 26

[0207] Figure 26 ​​As shown in the embodiments, first, the processing circuit 542 determines the bone-to-muscle scale (860) of the patient's soft tissue structure. The processing circuit 542 can determine the length, width, and / or volume of the scale of bones such as the scapula relative to muscles according to the patient-specific shape. The bone-to-muscle scale can identify the specific anatomical structure dimensions of the patient. Then, the processing circuit 542 obtains the statistical mean shape (SMS) (862) of the soft tissue structure. The SMS can be a representation of the typical muscle scale calculated based on a large number of subjects. The SMS can be based on a healthy population. In some embodiments, the SMS can be based on the patient's race, gender, age, height, or other factors similar to the patient.

[0208] Next, the processing circuit 542 deforms the SMS using a minimization algorithm to fit the SMS to the bone-to-muscle scale of the soft tissue structure (864). For example, the processing circuit 542 can modify the SMS to fit more tightly to the patient's bone-to-muscle scale and thereby evaluate the pre-morbid state of the muscle. Various different types of minimization algorithms can be employed to make the SMS fit the patient's anatomy. The generated pre-morbid prediction of the muscle can be used as an assessment of the scale of the health state of the muscle. Then, the processing circuit 542 determines the atrophy rate of the soft tissue structure (866) by dividing the volume of the deformed SMS by the volume of the soft tissue structure represented by the patient-specific shape. The result is the atrophy rate of the soft tissue structure, that is, the ratio of the volume of healthy tissue to the volume of actual tissue. In other embodiments, the atrophy rate can be calculated based on other scale comparisons between the healthy state and the current state of the soft tissue structure. Then, in addition to the calculations during the pre-operative planning process, the processing circuit 542 outputs the atrophy rate for display to the user or for further use (868). For example, the processing circuit 542 can output the current tissue or patient-specific shape superimposed on a pre-morbid or healthy assessment of the same soft tissue structure. When performing pre-operative planning on a joint of interest (e.g., the rotator cuff muscles of the shoulder joint of interest), the processing circuit 542 can perform this atrophy rate calculation for each muscle of interest.

[0209] Figure 27 is a flowchart showing an exemplary procedure for determining the type of shoulder treatment based on the determined soft tissue structure of a patient according to the techniques of the present disclosure. The processing circuit 542 of the system 540 is described as performing Figure 27 the embodiments, however, other devices or systems such as the virtual planning system 102 can perform one or more parts of the present technology. Further, a combination of two or more than two devices and / or systems can perform some parts of the present technology via a distributed system. The processes are described with reference to three-dimensional datasets, Figure 26 however, in other embodiments, several two-dimensional slice data can be analyzed in a similar manner.

[0210] As Figure 27 shown in the embodiments of , the processing circuit 542 may model one or more rotator cuff muscles associated with the shoulder joint, and / or other muscles, as springs (870). The processing circuit 542 then obtains the fat infiltration value and atrophy rate (872) of each soft tissue structure. The processing circuit 542 then determines the spring constant (874) for each muscle based on the fat infiltration value and atrophy rate. The processing circuit 542 can then determine the range of motion of the humerus in the shoulder joint based on the spring constants of one or more muscles (876). The processing circuit 542 then determines the type of shoulder treatment based on the determined range of motion (878). For example, the processing circuit 542 may select between an anatomical shoulder replacement and a reverse shoulder replacement. In some embodiments, pathologies such as concentric osteoarthritis or massive rotator cuff tears, muscle mass metrics, age, and glenoid deformity status may also affect whether an anatomical or reverse shoulder replacement is suitable for the patient. In some embodiments, the processing circuit 542 may determine other aspects of the surface from this information, such as the size of the implant, the position of the implant, or other relevant information. In some embodiments, the processing circuit 542 may use one or more decision trees or neural networks to determine the type of shoulder replacement. The recommended shoulder treatment may also be based on other patient information such as age, gender, activity level, or any other aspect.

[0211] The processing circuit 542 may control the user interface to display the recommended shoulder replacement. In response to a clinician selecting, accepting, or confirming the recommendation, the processing circuit 542 may initiate other preoperative planning for the patient and the selected type of shoulder replacement. For example, the processing circuit 542 or another system such as the virtual planning system 102 of Figure 1 may generate a surgical plan for the selected shoulder replacement. The processing circuit 542 may control the user interface to guide the clinician through additional customization steps for the patient, such as the implants required, cutting planes, anchor positions, reaming axes, screw drilling and / or placement, sizes, implant placement, specific steps of the surgery, and any other aspects of the selected surgery. The clinician may also interact with the surgical plan by observing system-generated visualizations of the procedure, anatomy, and / or implants for the patient. These types of processes may be applied to other joints such as the ankle, elbow, wrist, hip, knee, etc. Additionally, Figure 27The process herein can be used to determine the position or rotational angle of one or more implants associated with shoulder treatment. For example, the processing circuit 542 can determine whether to centralize (e.g., move the humeral head implant closer to the scapula or move the glenoid closer to the patient's midline) or lateralize (e.g., move the humeral head implant farther from the scapula or move the glenoid closer to the humerus) to improve the range of motion based on the spring constant. In some embodiments, the processing circuit 542 can recommend bone grafting (removing or adding bone) to the size and / or position of the humeral head and / or glenoid to achieve the desired position of the humerus relative to the scapula. The processing circuit can use the stiffness of the shoulder muscles (e.g., the spring constant when used to model the muscles) to determine the appropriate position of the humeral head to achieve the appropriate range of motion and / or strength of the shoulder.

[0212] As described herein, the processing circuit 542 and / or other systems can use patient-specific imaging data to determine multiple measurements of the patient's morphological features. The measurements can include distance measurements, angle measurements, or measurement relationships of the patient's structures and / or other types of numerical characterizations between the patient's structures. For example, the measurements can include any combination of values related to one or more of the following:

[0213] ● Glenoid version: The angular orientation of the axis of the glenoid joint surface relative to the transverse axis of the scapula.

[0214] ● Glenoid tilt: The up / down tilting of the glenoid relative to the scapula.

[0215] ● Glenoid orientation / direction: The three-dimensional orientation of the glenoid in three-dimensional space.

[0216] ● Radius of the best-fit sphere of the glenoid: The radius of the best-fit sphere of the patient's glenoid. The best-fit sphere is a conceptual sphere sized such that the sector of the sphere sits as flat as possible with the patient's glenoid.

[0217] ● Root mean square error of the best-fit sphere of the glenoid: The mean square error of the difference between the patient's glenoid and the sector of the best-fit sphere.

[0218] ● Reverse shoulder angle: The inclination of the lower part of the glenoid.

[0219] ● Critical shoulder angle: The angle between the plane of the glenoid fossa and the connecting line to the lowest point of the acromion.

[0220] ● Acromiohumeral space: The space between the acromion and the top of the humerus.

[0221] ● Glenohumeral space: The space between the glenoid and the humerus.

[0222] ● Humeral version: The angle between the humeral orientation and the axis of the lateral condyle.

[0223] ● Humeral neck axis angle: The angle between the normal vector of the anatomical neck of the humerus and the intramedullary axis.

[0224] ● Radius and root mean square error of the best-fit sphere of the humeral head: The radius of the best-fit sphere of the humeral head of the patient. The best-fit sphere refers to a conceptual sphere with dimensions such that the sectors of the sphere fit the surface of the humeral head as closely as possible. The root mean square error refers to the error between the best-fit sphere and the actual humeral head of the patient.

[0225] ● Humeral subluxation: Measurement of the subluxation of the humerus relative to the glenoid.

[0226] ● Humeral orientation / direction: The orientation of the humeral head in three-dimensional space.

[0227] ● Measurement of the epiphysis of the patient's humerus.

[0228] ● Measurement of the metaphysis of the patient's humerus.

[0229] ● Measurement of the diaphysis of the patient's humerus.

[0230] ● Retroversion of the bone.

[0231] Figure 28 is a flowchart showing an exemplary procedure for determining the type of shoulder treatment based on patient-specific image data according to the technology of the present disclosure. The processing circuit 542 of the system 540 is described as performing Figure 28 in the embodiments, however, other devices or systems such as the virtual planning system 102 can perform one or more parts of the present technology. Further, a combination of two or more than two devices and / or systems can perform some parts of the present technology via a distributed system. The process is described with reference to a three-dimensional data set Figure 28 in, however, in other embodiments, a number of two-dimensional slice data can be analyzed in a similar manner.

[0232] As Figure 28 shown in the embodiments in, the processing circuit 542 can receive patient-specific image data (e.g., CT image data) of the patient. Then, the processing circuit 542 determines one or more soft tissue features (892) from the patient-specific image data of one or more soft tissue structures of the patient. Exemplary soft tissue features can include soft tissue shape and volume, fat infiltration value, atrophy rate, range of motion value, or any other such parameter. Then, the processing circuit 542 generates a recommendation (894) for the type of shoulder surgery based on the determined one or more soft tissue features. For example, the processing circuit 542 can select between an anatomical total shoulder replacement or a reverse total shoulder replacement. Then, the processing circuit 542 can output the determined recommendation for the type of shoulder surgery of the patient for display on the user interface (896).

[0233] One or more determinations described herein and related to orthopedic classification and surgical planning may employ artificial intelligence (AI) techniques such as neural networks. In one embodiment, processing circuitry 542 may employ various AI techniques to generate one or more characteristics of tissue, such as atrophy rate, fat infiltration, range of motion, recommendations for the type of implant (e.g., stem size of a humeral implant), and / or recommendations for a specific type of surgical treatment (e.g., anatomic or reverse shoulder replacement), among others. In some embodiments, such AI techniques may be employed during the preoperative phase 302( Figure 3 ) or other phases of the surgical lifecycle. A deep neural network (DNN) is a class of artificial neural network (ANN) that has shown great promise as a classification tool. A DNN includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. A DNN may also include one or more other types of layers such as pooling layers.

[0234] Each layer may include a set of artificial neurons, i.e., typically simply referred to as "neurons". Each neuron in the input layer receives input values from an input vector. The output of the neurons in the input layer is provided as the input to the next layer in the DNN. Each neuron in the layer after the input layer may apply a propagation function to the output of one or more neurons in the previous layer to generate input values to the neuron. The neuron may then apply an activation function to the input to compute an activation value. The neuron may then apply an output function to the activation value to generate an output value of the neuron. The output vector of the DNN includes the output values of the output layer of the DNN.

[0235] There are several challenges associated with the application of DNNs to planned orthopedic surgery, specifically, with regard to shoulder pathologies. For example, some challenges relate to how to construct and train a DNN such that the DNN can provide meaningful outputs regarding shoulder pathologies. In another embodiment of the challenges associated with the application of DNNs to planned orthopedic surgery, it can be understood that patients and medical professionals are reluctant to trust decisions made by a computer, especially when it is not clear how the computer made those decisions. Thus, there is a question regarding how to generate outputs in a manner that helps ensure that patients and medical professionals are willing to trust the outputs of the DNN.

[0236] This disclosure describes techniques that can address these challenges and provide a DNN architecture that provides meaningful outputs regarding shoulder pathologies and / or recommended shoulder treatments based on one or more inputs. For example, an artificial neural network (ANN) has an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. The input layer includes a plurality of input layer neurons. Each input layer neuron among the plurality of input layer neurons corresponds to a different input element among a plurality of input elements. The output layer includes a plurality of output layer neurons.

[0237] Each output layer neuron among the multiple output layer neurons corresponds to a different output element among the multiple output elements. Each output element among the multiple output elements corresponds to a different classification in one or more shoulder pathology classification systems. In this embodiment, the computing system may generate multiple training data sets from previous shoulder surgery cases. Each corresponding training data set corresponds to a different training data patient among the multiple training data patients and includes a corresponding training input vector and a corresponding target output vector.

[0238] For each corresponding training data set, the training input vector of the corresponding training data set includes the value of each element among the multiple input elements. For each corresponding training data set, the target output vector in the corresponding training data set includes the value of each element among the multiple output elements. In this embodiment, the computing system may use the multiple training data sets to train the neural network. Additionally, in this embodiment, the computing system may obtain a current input vector corresponding to a current patient. The computing system may apply the DNN to the current input vector to generate a current output vector. Then, the computing system may determine a diagnosis of the shoulder health condition of the current patient based on the current output vector, which may also be referred to as shoulder classification.

[0239] In this embodiment, by making different output elements among the multiple output elements correspond to different categories in one or more shoulder pathology classification systems, the DNN can provide meaningful output information that can be used in the diagnosis of a patient's shoulder health condition, the determination of anatomical structure features, or treatment recommendations. For example, this can be more efficient in terms of computation and training time than a system in which different values of neurons in the output layer correspond to different categories. Further, in some embodiments, the output value of a neuron in the output layer indicates a measure of the confidence with which the classified shoulder health condition of the patient belongs to the corresponding category in a shoulder pathology classification system. This confidence value can help a user consider the possibility that the patient may have a shoulder health condition of a different category than that determined by the computer system using the DNN. Further, for the output of the same output layer neuron, it may be computationally efficient in terms of expressing a confidence level and serving as a basis for determining a diagnosis (e.g., classification) of the patient's shoulder health condition, specific features of tissue (e.g., fat infiltration value, atrophy value, range of motion value), or even a recommendation for a surgical procedure type based on one or more such tissue features.

[0240] Figure 29 is a block diagram showing an exemplary computing system 902 of a DNN that can be used to implement one or more aspects of determining a patient's anatomy, diagnosis, and / or treatment recommendations according to the techniques of the present disclosure. The computing system 902 may be part of an orthopedic surgery system 100 ( Figure 1)。The computing system 902 can use a DNN to determine soft tissue characteristics and / or recommendations for the type of treatment, such as whether a patient would benefit from an anatomic or reverse total shoulder replacement. In some embodiments, the computing system 902 includes an XR visualization device (e.g., an MR visualization device or an XR visualization device), and the XR visualization device includes one or more processors that perform the operations of the computing system 902.

[0241] As Figure 29 shown in the embodiments of, system 900 includes a computing system 902 and a collection of one or more client devices (collectively referred to as "client devices 904"). In other embodiments, system 900 may include more, fewer, or different devices and systems. In some embodiments, the computing system 902 and the client devices 904 may communicate via one or more communication networks such as the Internet.

[0242] The computing system 902 may include one or more computing devices. The computing system 902 and the client devices 904 may include various types of computing devices, such as server computers, personal computers, smartphones, laptop computers, and other types of computing devices. In Figure 29 the embodiments of, the computing system 902 includes a processing circuit 908, a data storage system 910, and a collection of one or more communication interfaces 912A through 912N (collectively referred to as "communication interfaces 912"). The data storage system 910 is configured to store data. The communication interfaces 912 are capable of enabling the computing system 902 to communicate with other computing systems and devices such as client devices 912 (e.g., wirelessly or using a wire). For ease of explanation, as performed by the computing system 902 as a whole, the present disclosure may describe actions performed by the processing circuit 906, the data storage system 910, and the communication interfaces 912. One or more subsystems in the orthopedic surgery system 100 ( Figure 1 ) may include the computing system 902 and the client devices 904. For example, the virtual planning system 102 may include the computing system 902 and the client devices 904.

[0243] A user may use the client device 904 to access information generated by the computing system 902. For example, the computing system 902 may generate a recommendation for the type of shoulder treatment for the current patient. The recommendation may be represented by a shoulder category among a plurality of shoulder categories in the shoulder treatment classification system. In this embodiment, the user may use the client device 904 to access information about the intervention recommendation. Since the computing system 902 may be far from the client device 904, the user of the client device 904 may consider the computing system 902 to be in a cloud-based computing system. In other embodiments, some or all of the functions of the computing system 902 may be performed by one or more client devices 904.

[0244] Computing system 902 may implement a DNN. Storage system 910 may include one or more computer-readable data storage media. Storage system 910 may store the parameters of the DNN. For example, storage system 910 may store the weights of the neurons of the DNN, the bias values of the neurons of the DNN, etc.

[0245] Computing system 902 may determine a treatment recommendation for a patient's shoulder health condition based on the output of the DNN. According to the techniques of the present disclosure, the output elements of the DNN include output elements corresponding to different categories in one or more shoulder recommendation classification systems. The shoulder recommendation classification system may include different classification systems for each type of treatment of the patient, or different classifications for different pathological types that may lead to different recommendations. For example, different treatments may include anatomic shoulder replacement and reverse shoulder replacement. However, other treatments or surgeries may have corresponding classification systems. For example, the Walch classification system and the Favard classification system are two different classification systems for primary glenohumeral osteoarthritis. The Warner classification system and the Goutallier classification system are two different classification systems for rotator cuff. In some embodiments, the shoulder pathology classification system may include classifications of more general categories of shoulder pathologies, such as one or more of the following: primary glenohumeral osteoarthritis (PGHOA), rotator cuff tear arthropathy (RCTA) instability, massive rotator cuff tear (MRCT), rheumatoid arthritis, post-traumatic arthritis, and osteoarthritis. These classification systems may be used to determine treatment recommendations.

[0246] For example, the Walch classification system defines five categories: 1A, 1B, 2A, 2B, and 3. As another example, the Favard classification system defines five categories: E0, E1, E2, E3, and E4. As yet another example, the Warner classification system defines four categories of rotator cuff atrophy: none, mild, moderate, and severe. As yet another example, the Goutallier classification system defines five categories: 0 (completely normal muscle), I (some fatty streaks), II (muscle mass greater than fat infiltration), III (muscle mass equal to fat infiltration), IV (amount of fat infiltration greater than muscle). In other embodiments, the classification system may use a fat infiltration value or an atrophy rate calculated from patient-specific image data.

[0247] In some embodiments, the computing system 902 may determine a treatment recommendation based on a diagnosis of a patient's shoulder health condition according to a comparison with the values of output elements generated by the DNN. For example, the value of an output element may correspond to a confidence value that indicates the confidence level to which the patient's shoulder health condition belongs in the category corresponding to the output layer neuron that generated the value. For example, the value of the output element may be a confidence value or the computing system 902 may calculate a confidence value based on the value of the output element.

[0248] In some embodiments, the output function of the output layer neuron generates a confidence value. Further, the computing system 902 may identify which confidence value is the highest. In this embodiment, the computing system 902 may determine that the shoulder pathology category corresponding to the highest confidence value is the diagnosis of the current patient's shoulder health condition. In some embodiments, if no confidence value is higher than a threshold, the computing system 902 may generate an output indicating that the computing system 902 cannot make a conclusive diagnosis. Thus, the computing system 902 cannot determine a treatment recommendation.

[0249] As mentioned above, in some embodiments, the output elements of the DNN include confidence values. In one such embodiment, a confidence value function outputs a confidence value. The confidence value function may be the output function of the output layer neuron of the DNN. In this embodiment, all possible confidence values output by the confidence value function are within a predetermined range. Further, in this embodiment, the computing system 902 may apply the DNN to the input vector to generate an output vector. As part of applying the DNN, the computing system 902 may calculate the output value of each corresponding output layer neuron for the plurality of output layer neurons.

[0250] Then, the computing system 902 may apply the confidence value function using the output value of the corresponding output layer neuron as an input to the confidence value function. The confidence value function outputs the confidence value of the corresponding output layer neuron. In this embodiment, for each corresponding output layer neuron of the plurality of output layer neurons, the output element corresponding to the corresponding output layer neuron specifies the confidence value of the corresponding output layer neuron. Further, for each corresponding output layer neuron of the plurality of output layer neurons, the confidence value of the corresponding output layer neuron is a measure of the confidence that the current patient's shoulder health condition belongs to a category corresponding to the output element (corresponding to the corresponding output layer neuron) in one or more shoulder pathology classification systems.

[0251] The computing system 902 may use various confidence value functions. For example, the computing system 902 may apply a hyperbolic tangent function, a sigmoid function, or other types of functions that output values within a predetermined range. The hyperbolic tangent function (tanh) has the form γ(c)=tanh(c)=(e c –e -c) / (e c +e -c ). The hyperbolic tangent function takes a real-valued argument, such as the output value of an output layer neuron, and transforms it into the range (–1, 1). The sigmoid function has the form γ(c) = 1 / (1 + e -c ). The sigmoid function takes a real-valued argument, such as the output value of an output layer neuron, and transforms it into the range (0, 1).

[0252] Computing system 902 can train the DNN using multiple training data sets. Each corresponding training data set can correspond to a different training data patient among a plurality of previously diagnosed training data patients. For example, the first training data set can correspond to the first training data patient, the second training data set can correspond to the second training data patient, and so on. To the extent that the training data set can include information about the patient, the training data set can correspond to the training data patient. The training data patient can be a real patient with a diagnosed shoulder health condition. In some embodiments, the training data patient can include a simulated patient.

[0253] Each corresponding training data set can include a corresponding training input vector and a corresponding target output vector. For each corresponding training data set, the training input vector of the corresponding training data set includes the value of each element among a plurality of input elements. In other words, the training input vector can include the value of each input layer neuron of the DNN. For each corresponding training data set, the target output vector of the corresponding training data set can include the value of each element among a plurality of output elements. In other words, the target output vector can include the value of each output layer neuron of the DNN.

[0254] In some embodiments, the values in the target output vector are based on a confidence value. This confidence value, in turn, can be based on the confidence level expressed by one or more trained medical professionals, such as an orthopedic surgeon. For example, the information in the training input vector of the training data set (or information from which the training input vector of the training data set is derived) can be given to the trained medical professional and the trained medical professional can be consulted to provide the confidence level that the training data patient has a shoulder health condition belonging to each category in each shoulder pathology classification system.

[0255] For example, in an embodiment where the shoulder pathology classification system includes the Walch classification system, a medical professional may indicate that her confidence level that the shoulder health condition of the training data patient belongs to category A1 is 0 (i.e., she does not believe at all that the shoulder health condition of the training data patient belongs to category A1), indicate that her confidence level that the shoulder health condition of the training data patient belongs to category A2 is 0, indicate that her confidence level that the shoulder health condition of the training data patient belongs to category B1 is 0.75 (i.e., she is fairly confident that the shoulder health condition of the training data patient belongs to category B1), indicate that her confidence level that the shoulder health condition of the training data patient belongs to category B2 is 0.25 (i.e., she believes that the chance that the shoulder health condition of the training data patient belongs to category B2 is small), and may indicate that her confidence level that the shoulder health condition of the training data patient belongs to category C is 0. In some embodiments, the computing system 902 may apply the inverse of the confidence value function to the confidence values provided by the medical professional to generate the values included in the target output vector. In some embodiments, the confidence values provided by the medical professional are the values included in the target output vector.

[0256] For the shoulder health condition of the same training data patient belonging to each category in each shoulder pathology classification system, different medical professionals may have different confidence levels. Thus, in some embodiments, the confidence values on which the values in the target output vector are based may be an average value or otherwise determined from the confidence levels provided by multiple medical professionals. Similar confidence values for recommendations of treatment types can be calculated based on the identified pathology or features determined from the DNN.

[0257] In some such embodiments, greater weight may be given to the confidence levels of some medical professionals than to the confidence levels of other medical professionals in the weighted average of the confidence levels. For example, greater weight may be given to the confidence level of a prominent orthopedic surgeon than to the confidence level of other orthopedic surgeons. In another embodiment, greater weight may be given to the confidence levels of medical professionals or training data patients in a particular region or hospital than to the confidence levels of medical professionals or training data patients in other regions or hospitals. Advantageously, this weighted average may allow the DNN to be adjusted according to various criteria and preferences.

[0258] For example, a medical professional may preferably use a trained DNN to weight the confidence level in a specific manner. In some embodiments where the training dataset includes a training dataset based on the medical professional's own cases, the medical professional (e.g., an orthopedic surgeon) may prefer to use a DNN trained using such a training dataset in which the medical professional's own cases are weighted more heavily or exclusively using the medical professional's own cases. Thus, the DNN can generate an output customized to the medical professional's own practice style. Also, as mentioned above, the medical professional and the patient may have different levels of confidence in the output of the computing system. Accordingly, in some embodiments, the computing system 902 can provide information indicating that the DNN is trained to simulate the decisions of the medical professional himself and / or an orthopedic surgeon with a specific confidence.

[0259] In some embodiments, different medical professionals' confidence levels in the same training data patient can be used when generating different training datasets. For example, the confidence level of a first medical professional for a specific training data patient can be used to generate a first training dataset and the confidence level of a second medical professional for the same training data patient can be used to generate a second training dataset.

[0260] Further, in some embodiments, the computing system 902 can provide a confidence value for the output to one or more users. For example, the computing system 902 can provide a confidence value to the client device 904 for display to one or more users. Thus, one or more users can better understand how the computing system 902 can obtain a diagnosis and / or recommendation for the treatment of a patient's shoulder.

[0261] In some embodiments, to expand the training dataset universe, the computing system 902 can automatically generate confidence values from electronic medical records. For example, in one embodiment, a patient's electronic medical record can include data from which the computing system 902 can form an input vector and can include data indicating a surgeon's diagnosis of the patient's shoulder health and the selected shoulder treatment. In this embodiment, the computing system 902 can infer a default confidence level from the diagnosis. The default confidence level can have various values (e.g., 0.75, 0.8, etc.). While this default confidence level may not reflect the surgeon's actual confidence level, the input confidence level can help increase the number of available training datasets, which can improve the accuracy of the DNN.

[0262] In some embodiments, the training dataset is weighted based on the health outcomes of the training data patients. For example, if the training data patients associated with the training dataset have all positive health outcomes, a higher weight can be given to the training dataset. However, if the associated training data patients have less positive health outcomes, a lower weight can be given to the training dataset. During the training process, the computing system 902 can use a loss function that weights the training dataset based on the weight given to the training dataset.

[0263] In some embodiments, as part of generating the training dataset, the computing system 902 can select multiple training datasets from a database of training datasets based on one or more training dataset selection criteria. In other words, if a training dataset does not meet the training dataset selection criteria, the computing system 902 can exclude a particular training dataset from the training process of the DNN. In Figure 29 an embodiment, the data storage system 910 stores a database 914 containing training datasets from previous shoulder surgery cases.

[0264] There can be a wide variety of training dataset selection criteria. For example, in one embodiment, one or more training dataset selection criteria can include which surgeon performed the surgery on the multiple training data patients. In some embodiments, one or more training dataset selection criteria include the region where the training data patients live. In some embodiments, one or more training dataset selection criteria include the region associated with one or more surgeons (e.g., the region where one or more surgeons interned, lived, obtained a license, trained, etc.).

[0265] In some embodiments, one or more training dataset selection criteria include the postoperative health outcomes of the training data patients. In this embodiment, the postoperative health outcomes of the training data patients can include one or more of the following: postoperative range of motion, the presence of a postoperative infection, or postoperative pain. Thus, in this embodiment, the training dataset on which the DNN is trained can exclude training datasets in which adverse health outcomes occur.

[0266] Additional training datasets can be added to the database over time and the computing system 902 can use the additional training datasets to train the DNN. Thus, as more training datasets are added to the database, the DNN can continue to improve over time.

[0267] The computing system 902 may apply one of various techniques to train the DNN using a training data set. For example, the computing system 902 may use one of various standard backpropagation algorithms known in the art. For example, as part of training the DNN, the computing system 902 may apply a cost function to determine a cost value based on the difference between the output vector generated by the DNN and the target output vector. Then, the computing system 902 may use the cost value in the backpropagation algorithm to update the weights of the neurons in the DNN. In this way, the computing system 902 may train the DNN to determine various characteristics (e.g., fat infiltration, atrophy rate, range of motion, etc.) of a patient's soft tissue based on inputs such as tissue volume, voxel grouping, pre-morbid shape volume. In some embodiments, the computing system 902 may train the DNN to determine recommendations for shoulder treatment using inputs from patient-specific image data and / or determined characteristics of soft tissue such as those that may be determined from different DNNs. In some embodiments, the computing system 902 may train the DNN to determine recommendations for shoulder treatment using bone density metrics indicative of or related to the bone density of one or more bones (e.g., the humeral head) receiving an implant. For example, the computing system 902 may train the DNN using patient-specific imaging data (e.g., CT data) associated with the humeral head and the surgeon-selected type of humeral implant for each respective patient (e.g., stemmed, which may include or not include the length of the stem, or stemless humeral implant). The output of this training is a recommended type of humeral implant based on patient-specific image data associated with the humeral head of a new patient. In this way, the type of shoulder treatment and / or implant may be determined based on the density of trabecular bone or other characteristics related to the density within the humerus.

[0268] Figure 30 An exemplary DNN 930 is shown that may be implemented by the computing system 902 having Figure 29 the system in Figure 30 In an embodiment of, the DNN 930 includes an input layer 932, an output layer 934, and one or more hidden layers 936 located between the input layer 932 and the output layer 934. In Figure 30 an embodiment of, neurons are represented as circles. Although in Figure 30 an embodiment of, each layer is shown as including six neurons, however, the layers in the DNN 930 may include more or fewer neurons. Further, although in Figure 30 the DNN 930 is shown as a fully connected network in Figure 30 however, the DNN 930 may have a different architecture. For example, the DNN 930 may not be a fully connected network, may have one or more convolutional layers, or may otherwise have a different architecture than that shown in

[0269] In some implementations, the DNN 930 can be or include one or more artificial neural networks (also simply referred to as neural networks). A neural network can include a set of connected nodes, which can also be referred to as neurons or perceptrons. A neural network can be organized into one or more layers. A neural network that includes multiple layers can be referred to as a “deep” network. A deep network can include an input layer, an output layer, and one or more hidden layers located between the input layer and the output layer. The nodes of a neural network can be fully or partially connected.

[0270] The DNN 930 can be or include one or more feedforward neural networks. In a feedforward neural network, the connections between nodes do not form loops. For example, each connection can connect a node from a previous layer to a node from a subsequent layer.

[0271] In some instances, the DNN 930 can be or include one or more recurrent neural networks. In some instances, at least some of the nodes in a recurrent neural network can form loops. Recurrent neural networks can be particularly useful for processing inherently sequential input data. Specifically, in some instances, a recurrent neural network can pass or save information from a previous part of an input data sequence to a subsequent part of the input data sequence by using recurrent or directed cyclic node connections.

[0272] In some embodiments, the sequential input data can include event sequence data (e.g., sensor data over time or images captured at different times). For example, a recurrent neural network can analyze sensor data over time to detect or predict swipe directions, perform handwritten recognition, etc. The sequential input data can include words in a sentence (e.g., for natural language processing, language detection or processing, etc.), musical notes in a piece of music, sequential actions made by a user (e.g., to detect or predict sequential application usage), sequential target states, etc. Exemplary recurrent neural networks include long short-term (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous-time recurrent neural networks, neural history compressors, echo state networks, Elman networks, Jordan networks, recurrent neural networks, Hopfield networks, fully recurrent networks, sequence-to-sequence configurations, etc.

[0273] In some implementations, the DNN 930 can be or include one or more convolutional neural networks. In some instances, a convolutional neural network can include one or more convolutional layers that perform convolutions on input data using learned filters. Filters can also be referred to as kernels. Convolutional neural networks can be particularly useful for vision problems such as when the input data includes images such as still images or videos. However, convolutional neural networks can also be applied to natural language processing.

[0274] The DNN 930 can be or include one or more other forms of artificial neural networks such as, for example, deep Boltzmann machines, deep belief networks, stacked autoencoders, etc. Any neural networks described herein can be combined (e.g., stacked) to form a more complex network.

[0275] In Figure 30 an embodiment, the input vector 938 includes a plurality of input elements. Each input element can be a numerical value. The input layer 932 includes a plurality of input layer neurons. Each input layer neuron among the plurality of input layer neurons included in the input layer 932 can correspond to a different input element among the plurality of input elements. In other words, for each input element in the input vector 938, the input layer 932 can include a different neuron.

[0276] Further, in Figure 30 an embodiment, the output vector 940 includes a plurality of output elements. Each output element can be a numerical value. The output layer 934 includes a plurality of output layer neurons. Each output layer neuron among the plurality of output layer neurons corresponds to a different output element among the plurality of output elements. In other words, each output layer neuron in the output layer 934 corresponds to a different output element in the output vector 940.

[0277] The input vector 938 can include a wide variety of information. For example, the input vector 938 can include morphometric measurements of a patient. In some embodiments where the input vector 938 includes morphometric measurements of a patient, the input vector 938 can be determined based on medical images of the patient such as CT images, MRI images, or other types of images. For example, the computing system 902 can obtain medical images of the current patient (e.g., patient-specific image data). For example, the computing system 902 can obtain medical images from an imaging machine (e.g., a CT machine, an MRI machine, or other types of imaging machines), the patient's electronic medical record, or another data source. As described herein, in this embodiment, the computing system 902 can segment the medical images to identify internal structures of the current patient such as soft tissue and bone, and in some embodiments, generate patient-specific shapes of the bone and / or soft tissue. Further, in this embodiment, the computing system 902 can determine multiple measurements based on the relative positions of the identified internal structures of the current patient. In this embodiment, the plurality of input elements can include input elements for each of the multiple measurements. Other inputs can include other determinations from patient-specific image data such as fat infiltration values, atrophy rates, and / or joint range of motion values, etc.

[0278] As mentioned elsewhere in this disclosure, computing system 902 may include one or more computing devices. Accordingly, the various functions of computing system 902 may be performed by various combinations of the computing devices in computing system 902. For example, in some embodiments, a first computing device in computing system 902 may segment an image, a second computing device in computing system 902 may train a DNN, and a third computing device in computing system 902 may apply the DNN, etc. In other embodiments, a single computing device in computing system 902 may segment an image, train a DNN, and apply the DNN.

[0279] In some embodiments, input vector 938 may include information based on a patient's rotator cuff assessment (e.g., in combination with zero or more other exemplary types of input data described herein). For example, input vector 938 may include, alone or in combination with the morphological inputs described above, information about the fatty infiltration of the rotator cuff, atrophy of the rotator cuff, and / or other information about the patient's rotator cuff. In some embodiments, for example, measurements of fatty infiltration and atrophy of soft tissue used as inputs to a neural network may be derived by any of the soft tissue modeling techniques described in this application. In some embodiments, information about a patient's rotator cuff may be expressed in terms of categories in a shoulder pathology classification system such as the Warner classification system or the Goutallier classification system.

[0280] In some embodiments, input vector 938 may include (e.g., in combination with zero or more other exemplary types of input data described herein) the patient's range of motion information. As described elsewhere in this disclosure, in some embodiments, a motion tracking device may be used to generate the patient's range of motion information. As described herein, in other embodiments, one or more range of motion values may be determined from the analysis of patient-specific image data.

[0281] Further, in some embodiments, input vector 938 may include (e.g., in combination with zero or more other exemplary types of input data described herein) information specifying a category in one or more shoulder pathology classification systems. In this embodiment, the output vector may include output elements corresponding to the category in one or more different shoulder recommendations for shoulder treatment. For example, input vector 938 may include information specifying a category in a rotator cuff muscle classification system and output vector 940 may include output elements corresponding to the type of shoulder treatment recommended (e.g., anatomic or reverse shoulder replacement).

[0282] In some embodiments, the input vector 938 can include (e.g., in combination with zero or more other exemplary types of input data described herein, including morphological input and / or rotator cuff input) information specifying the bone density score of the humerus and / or glenoid. Other information included in the input vector 938 can include demographic information such as patient age, patient activity, patient gender, patient body mass index (BMI), etc. In some embodiments, the input vector 938 can include information about the rate of onset of symptoms (e.g., gradual or sudden). The plurality of input elements in the input vector 938 can also include patient goals for participating in activities such as specific exercise / sports types, range of motion, etc.

[0283] In some embodiments, the output vector can include output elements for a variety of surgical types. The output elements for each surgical type can correspond to different types of shoulder surgeries. Exemplary types of shoulder surgeries that can be presented as output can include stemless standard total shoulder arthroplasty, stemmed standard total shoulder arthroplasty, stemless reverse shoulder arthroplasty, stemmed reverse shoulder arthroplasty, augmented glenoid standard total shoulder arthroplasty, augmented glenoid reverse shoulder arthroplasty, and other types of orthopedic shoulder surgeries. A shoulder surgery can be "standard" in terms of the patient's shoulder joint having a standard anatomical configuration after the surgery, wherein the glenoid side of the shoulder joint has a concave surface and the humeral side of the shoulder surgery has a convex surface. On the other hand, a "reverse" shoulder surgery results in the opposite configuration, wherein the convex surface is attached to the scapula and the concave surface is attached to the humerus.

[0284] Furthermore, the computing system 902 can determine a recommended type of shoulder surgery for the patient based on the current output vector. For example, the computing system 902 can determine which output element in the output vector corresponds to the type of shoulder surgery with the highest confidence value.

[0285] Figure 31 is a flowchart showing an exemplary operation of a computing system for determining a recommended type of shoulder surgery for a patient using a DNN according to the techniques of the present disclosure. In Figure 31 embodiments, the computing system 902 generates a plurality of training data sets (950). In this embodiment, the DNN has an input layer, an output layer, and one or more hidden layers located between the input layer and the output layer. The input layer includes a plurality of input layer neurons. Each input layer neuron in the plurality of input layer neurons corresponds to a different input element among the plurality of input elements. The output layer includes a plurality of output layer neurons.

[0286] Each output layer neuron among the multiple output layer neurons corresponds to a different output element among the multiple output elements. The multiple output elements include output elements of multiple surgical operation types. Each output element of each surgical operation type among the output elements of multiple surgical operation types corresponds to a different type of shoulder surgery among multiple types of shoulder surgeries. Each corresponding training data set corresponds to a different training data patient among the multiple training data patients, and includes a corresponding training input vector and a corresponding target output vector. For each corresponding training data set, the training input vector in the corresponding training data set includes the value of each element among the multiple input elements. For each corresponding training data set, the target output vector in the corresponding training data set includes the value of each element among the multiple output elements.

[0287] Further, in Figure 31 In an embodiment, the computing system 902 trains the DNN using multiple training data sets (952). Additionally, the computing system 902 may obtain a current input vector corresponding to a current patient (954). The computing system 902 may apply the DNN to the current input vector to generate a current output vector (956). The computing system 902 may determine a recommended type of shoulder surgery for the current patient based on the current output vector (958). The computing system 13 / 208 may perform these activities according to the embodiments provided elsewhere in the present disclosure.

[0288] Figure 32 is an illustration of an exemplary bone related to the shoulder 1000 of a patient. As Figure 32 shown in an embodiment, the shoulder 1000 includes the humerus 1004, the scapula 1010, and the clavicle 1016. The shaft 1006 of the humerus 1004 is connected to the humeral head 1008, and the humeral head 1008 and the glenoid 1012 form the glenohumeral joint 1002. The acromion 1014 of the scapula 1010 is attached to the clavicle 1016 at the acromioclavicular joint.

[0289] The humeral head 1008, the glenoid 1012, and / or the connective tissue between the humeral head 1008 and the glenoid 1012 may degenerate over time due to wear and / or disease. In some cases, a patient with degeneration of the glenohumeral joint 1002 may benefit from a shoulder replacement surgery, in which at least a part of the humeral head 1008, the glenoid 1012, or both are replaced with artificial implants. For example, the humeral head 1008 may be cut to expose the lower density trabecular bone within the humeral head 1008. A humeral implant may be inserted into the trabecular bone to fix the new humeral implant to the shaft 1006 of the humerus 1004.

[0290] Figure 33A 、 Figure 33B 、and Figure 33Cis a conceptual diagram of an exemplary humeral head 1022 for a humeral implant. As Figure 33A shown, as part of a shoulder arthroplasty procedure, a clinician can perform the surgical step of resection of the humeral head 1022 of the humerus 1020 by visually evaluating (e.g., "eyeballing") and marking the anatomic neck 1024 of the humeral head 1020. The anatomic neck 1024 can refer to a plane that bisects a portion of the humeral head 1022 to create or expose a surface to which a humeral implant can be attached to the humerus 1020. Then, as Figure 33B shown in the embodiment of, a clinician can perform the resection of the humeral head 1022 by guiding a cutting tool 1026 (e.g., the blade of an oscillating saw) along the marked anatomic neck 1024 using the clinician's free hand without mechanical or visual guidance. After completion of the resection of the humeral head 1022, a trabecular bone region 1028 is exposed along a plane corresponding to the anatomic neck 1024. Generally, a humeral implant can be inserted into a portion of the trabecular bone region 1028 and fixed in place. The density of the trabecular bone region 1028, or the density variation within the volume of the trabecular bone region 1028, can affect what type of humeral implant can be used for the humerus 1020. For example, a trabecular bone region 1028 of greater density can support a humeral implant with a shorter "stem" of a humeral implant compared to trabecular bone of lower density, or softer trabecular bone, which may require a longer stem on the humeral implant.

[0291] Figure 34 is a conceptual diagram of an exemplary humeral implant intended for anatomic shoulder replacement procedures. In total shoulder arthroplasty surgery (e.g., the type of shoulder replacement or shoulder treatment), the mechanical fixation strength of the humeral implant within the humeral shaft can be determined primarily by the fixation of the implant's diaphyseal stem. When introducing the stem of the humeral implant, the implantation of the stem can subject the shaft to high mechanical forces, as well as other blunt impact forces, during drilling, reaming, and broaching procedures. These procedures can lead to complications related to intraoperative fractures (humeral diaphysis), as well as postoperative loosening and stress shielding that often result in revision surgery. The extraction of a revision humeral head implant can also be difficult, especially when using a cemented implant.

[0292] A stemless humeral implant or a humeral implant with a shorter stem can eliminate potential shaft fractures and allow preservation of native bone stock by avoiding implantation in the shaft or portions of the shaft of the humerus, which is beneficial in the case of revision. The stemless design can also enable a surgeon to restore the glenoid-humeral center of rotation independent of the humeral shaft orientation and avoid complications associated with stem implantation. Skeletal fixation of a stemless humeral implant can be achieved primarily within the humeral head and the trabecular bone network therein.

[0293] Contraindications for a stemless humeral implant for sparing the shaft may include poor bone quality (osteopenia, osteoporosis), other metabolic bone diseases (cysts, tumors, etc.), or the presence of a prior fracture that may affect bone support, ingrowth, and integration of the metal components. As such, the density of the bone within the humeral head (such as the density and / or location of different bone densities, etc.) may need to be sufficient to support a shorter shaft or a stemless femoral implant type. The surgeon can use the "thumb test" as the primary intraoperative assessment tool, whereby pressing the surface of the neck incision of the humeral head with the thumb (proximal humeral tissue) can determine the basal viability for implantation. This basal viability may involve or be represented as the density of the trabecular bone within the humeral head. In other words, the bone density of the trabecular bone within the humeral head can be evaluated using the surgeon's thumb. Bone that is easily compressible with minimal force (e.g., low density) is considered unacceptable for stemless component implantation. Bone that is not easily compressible or provides higher resilience (e.g., higher density) can be considered acceptable for a stemless type of humeral implant.

[0294] As such, the surgeon can make a choice of stemless or stemmed (or even the length of the stem) based on the results of the "thumb test". As described herein, historical data related to the humeral head bone quality (such as density, compression rate, choice of stemless or stemmed implant, or other suitable properties of the humeral head supporting the implant, etc.) may be associated with patient-specific imaging data of these patients. For example, such an association can map the intensity threshold, range of Hounsfield units, or standard intensity deviation of the voxels in the humeral head of the CT data for a particular patient to the type of humeral implant selected by the patient for these corresponding patients. Once the association is completed, the system can then use this association to recommend a specific type of humeral implant based on the CT data mapped to each type of humeral implant. As such, as described herein, the system is capable of recommending a specific type of humeral implant (e.g., stemmed, stemless, and / or length of the stem) based on the analysis of patient-specific image data (e.g., intensity magnitude and / or magnitude location of a voxel or a set of voxels).

[0295] As Figure 34 As shown in the embodiment of FIG. [FIG. number not provided in the original, assuming it's a reference to a figure], the humeral implant 1040 is an embodiment of a "stemless" humeral implant having a gliding surface 1042 configured to contact the glenoid or glenoid implant. The fixation structure 1044 includes projections configured to be embedded into the trabecular bone 1028 to fix the humeral implant 1040, however, there is no stem that extends down near or into the shaft of the humerus.

[0296] The humeral implant 1050 is an example of a "stemmed" humeral implant. However, the stem fixation structure 1054 includes a short stem that facilitates anchoring the humeral implant 1050 to the trabecular bone of the humerus. The humeral implant 1050 includes a glide surface 1052 configured to contact the glenoid or glenoid implant. The humeral implant 1060 is an example of a "stemmed" humeral implant. However, the stem fixation structure 1064 includes a long stem that facilitates anchoring the humeral implant 1060 to the trabecular bone of the humerus. The humeral implant 1060 includes a glide surface 1062 configured to contact the glenoid or glenoid implant. The humeral implants 1050 and 1060 can be used when the trabecular bone is damaged from healthy bone such that a stem is needed to provide sufficient anchorage of the humeral implant. For example, the trabecular bone may not provide sufficient bone density to anchor the humeral implant 1040 and thus a "stemmed" humeral implant such as the humeral implants 1050 or 1060 may be recommended or selected for the patient instead.

[0297] For the humerus with the lowest density trabecular bone or otherwise requiring greater stability, the long stem fixation structure 1064 can be used. In contrast, when the density of the trabecular bone is high enough to sufficiently anchor the humeral implant with the stemless fixation structure 1044, the humeral implant 1040 can be used. The benefits of a stemless design similar to the fixation structure 1044 may include less trabecular bone removal from the humerus, faster healing, and a lower risk of cortical bone damage during the process of stem reaming and / or insertion into the humerus.

[0298] However, until the humeral head has been resected and manually manipulated, the clinician cannot determine whether the trabecular bone within the humeral head can support a stemless design such as the humeral implant 1040. As described herein, the system can use patient-specific image data (e.g., CT data) to determine a humeral head trabecular bone density metric to assist in shoulder replacement planning prior to surgery. Thus, exemplary techniques provide ways to improve the determination of bone density using computational analysis for various practical applications, reducing surgical time (e.g., determining what type of humeral head has been selected to use) and / or increasing the accuracy of the selection of the humeral head implant type prior to surgery (e.g., improving preoperative planning).

[0299] Figure 35 is a conceptual diagram 1070 of an exemplary stemmed humeral implant 1080 implanted within the humerus 1072. As Figure 35 shown in the embodiment of, the stemmed humeral implant 1080 includes a glide surface 1084 and a stem 1080. The stem 1080 has been inserted into the trabecular bone 1078 to anchor the humeral implant 1080 to the humeral head 1076. Although the stem 1080 can be the same as Figure 34a short stem similar to the humeral implant 1050 in [reference], however, the stem 1080 can still be at least partially inserted into the humeral shaft 1074. An exemplary humeral implant 1080 can be similar to the Aequalis Ascend TM Flex manufactured by Wright Medical Group N.V. of Memphis, Tennessee.

[0300] Figure 36 is a conceptual diagram 1100 of an exemplary stemless humeral implant 1112 implanted on the humeral head 1110. As Figure 36 shown in the embodiments in [reference], the stemless humeral implant 1112 includes a gliding surface 1116 and a fixation structure 1114. The fixation structure 1114 has been stemlessly embedded into the trabecular bone of the humeral head 1110. The humeral implant 1112 can be similar to the humeral implant 1040 in Figure 34 [reference]. An exemplary humeral implant 1112 can be similar to the Simpliciti TM Shoulder System manufactured by Wright Medical Group N.V. of Memphis, Tennessee. The gliding surface 1112 can be configured to contact the glenoid implant 1106 implanted in the glenoid surface of the scapula 1102. Since the humeral implant 1112 includes a spherical surface similar to the spherical surface of a healthy humeral head, the stemless humeral implant 1112 and the glenoid implant 1106 can be part of an anatomical shoulder replacement.

[0301] Figure 37 is a conceptual diagram of an exemplary reverse humeral implant 1124. The reverse humeral implant 1124 can be configured with a stem (similar to the humeral implant 1050 or 1060) or a stemless design (similar to the implant 1040) and implanted into the humeral head 1122 of the humerus 1020. However, the reverse humeral implant 1124 has a concave shape and is designed to contact the gliding surface of the spherical or convex-shaped contact surface of the corresponding glenoid implant. The system can recommend a reverse shoulder replacement including the reverse humeral implant 1124 based on the characteristics of the soft tissue structure (e.g., one or more muscles of the rotator cuff muscle or other shoulder muscles) and / or the bone density of the humeral head 1122.

[0302] Figure 38 is a block diagram showing exemplary components of a system 1140 configured to determine the evaluated bone density from patient-specific image data according to an embodiment of the present disclosure. The system 1140 and the components therein can be similar to the system 540 and components described in Figure 6 [reference] and / or Figure 1 the virtual planning system 102 in [reference]. Thus, the system 540 or the virtual planning system 102 can perform the functions ascribed to the system 1140 herein.

[0303] As Figure 38 shown in the embodiments of, system 1140 may include processing circuitry 1142, a power supply 1146, a display device 1148, an input device 1150, an output device 1152, a storage device 1154, and a communication device 1144. The display device 1148 may display images to present a user interface such as an opaque or at least partially transparent screen to the user. The display device 1148 may present visual information and, in some embodiments, audio information or other information presented to the user. For example, the display device 1148 may include one or more speakers, haptic devices, etc. In other embodiments, the output device 1152 may include one or more speakers and / or haptic devices. The display device 1148 may include an opaque screen (e.g., an LCD or LED display). Alternatively, the display device 1148 may include an MR visualization device, e.g., including a see-through holographic lens to be combined with a projector to allow the user to see real-world objects in the real-world environment through the lens and also see virtual 3D holographic images projected into the lens and onto the user's retina through, for example, a holographic projection system such as the Microsoft HOLOLENS TM device. In this embodiment, the virtual 3D holographic objects may seemingly be placed in the real-world environment. In some embodiments, the display device 1148 includes one or more display screens such as an LCD display screen, an OLED display screen, etc. The user interface may present a virtual image of details of a virtual surgical plan for a particular patient, such as information related to bone density, etc.

[0304] The input device 1150 may include one or more microphones and associated speech recognition processing circuitry or software that can recognize voice commands issued by the user and perform any of a variety of operations in response, such as selection, activation, or deactivation of various functions associated with surgical planning, intraoperative guidance, etc. As another embodiment, the input device 1150 may include one or more cameras or other optical sensors that detect and interpret gestures to perform the operations described above. As yet another embodiment, the input device 1150 includes one or more devices that sense the direction of gaze and perform various operations as described elsewhere in this disclosure. In some embodiments, the input device 1150 may receive manual input from the user via, for example, a handheld controller including one or more buttons, a keypad, a keyboard, a touch screen, a joystick, a trackball, and / or other manual input media and perform the various operations described above in response to the manual user input.

[0305] The communication device 1144 may include one or more circuits or other components that facilitate data communication with other devices. For example, the communication device 1144 may include one or more physical drives (e.g., DVD, Blu-ray, or Universal Serial Bus (USB) drives) to permit data transfer between the system 1140 and the drive when physically connected to the system 1140. In other embodiments, the communication device 1144 may include. The communication device 1144 may also support wired and / or wireless communication with another computing device and / or network.

[0306] The storage device 1154 may include one or more memories and / or repositories that store corresponding types of data in common and / or separate devices. For example, the user interface module 1156 may include instructions that define how the system 1140 controls the display device 1148 to present information such as information related to the bone density of the humerus or information associated with a recommended surgical treatment of the casing to the user. The preoperative module 1158 may include instructions for the analysis of patient data such as imaging data and / or the determination of treatment options based on the patient data. The intraoperative module 1160 may include instructions that define how the system 1140 operates when providing information for display to a clinician such as details about a planned surgery and / or feedback about a surgical procedure. The patient data 1166 may be a repository that stores patient-specific image data.

[0307] The bone density modeling module 1162 may include instructions that define how the processing circuit 1142 determines one or more bone density metrics for at least one or more portions of a bone such as the humeral head. For example, the bone density modeling module 1162 may determine a bone density metric based on the intensity of voxels within patient-specific patient data (e.g., CT image data). The processing circuit 1142 may execute the bone density modeling module 1162 to determine different bone density classes for a set of pixels or voxels based on a predetermined range of intensities of a single or a set of pixels or voxels (e.g., Hounsfield units). In some embodiments, the processing circuit 1142 may generate a bone density metric based on the standard deviation of voxels within patient-specific image data. The bone density metric may include different bone density values across a two-dimensional or three-dimensional region of the humeral head. In some embodiments, the bone density metric may be a single value determined based on the average pixel or voxel intensity across the humeral head or a region of the humeral head. In some embodiments, the bone density modeling module 1162 may include instructions for determining the type of humeral implant (e.g., stemmed or stemless) and / or the location within the humeral head where the humeral implant can be implanted. In fact, the bone density metric may not indicate the density of the bone but may be a metric representing bone density. For example, the bone density metric may only indicate the type of implant (e.g., stemmed or stemless) corresponding to the patient-specific image data being analyzed. As another example, the bone density metric may include voxel intensity from the image data, the standard deviation of voxel intensity from the image data, compressibility, an index, or some other indication related to or representing density without actually providing a density measurement of the bone.

[0308] Processing circuitry 1142 may execute a calibration module 1164 to calibrate bone density metrics for patient-specific image data and a selected implant type from other patients of a historical surgery (e.g., an implant type selected based on a history of thumb test information during the surgical procedure). Historically, a clinician may use their thumb to press on the trabecular bone within the humeral head (exposed via the cutting head) to determine the stiffness of the trabecular bone and thereby determine the density of the trabecular bone. The thumb test may be performed to identify what type of stem (if any) is required for a humeral implant. The calibration module 1164 may use this thumb test data obtained from historical patients to correlate known surgical decisions of the humeral implant type made based on the thumb test procedure with patient-specific image data of the same corresponding patient to determine the bone density metrics of the current patient. As such, the calibration module 1164 may be used to identify one or more ranges of bone density metrics corresponding to the respective humeral implant type. For example, using the calibration module 1164, the processing circuitry 1142 may determine whether a stemless humeral implant 1040 is for bone density metrics within a first range, whether a short-stem humeral head 1050 is for bone density metrics within a second range, and whether a long-stem humeral head 1060 is for bone density metrics within a third range.

[0309] As discussed above, the surgical lifecycle 300 may include a preoperative phase 302( Figure 3 ). One or more users may use the orthopedic surgery system 100 during the preoperative phase 302. For example, the orthopedic surgery system 100 may include a virtual planning system 102 (which may be similar to system 1140) to assist one or more users in generating a virtual surgical plan customized for the anatomy of interest of a specific patient. As described herein, the virtual surgical plan may include a three-dimensional virtual model corresponding to the anatomy of interest of a specific patient and a three-dimensional model of one or more prosthetic components (e.g., implants) that match the specific patient to repair the anatomy of interest or are selected to repair the anatomy of interest. The virtual surgical plan may also include a three-dimensional virtual model of guidance information to guide the surgeon during the performance of the surgical procedure, e.g., when preparing the bone surface or tissue and placing implantable prosthetic hardware relative to the bone surface or tissue.

[0310] As discussed herein, the processing circuitry 1142 can be configured to determine a bone density metric of at least a portion of the patient's humeral head based on the patient-specific image data of the patient. For example, the bone density metric can be the overall density of the humeral head or a single indication of a portion of the humeral head. As another example, the bone density metric can include bone density values of corresponding portions of the patient's humeral head. The system can control the user interface via the user interface module 1156 to present a graphical representation of the bone density metric (which can directly or indirectly indicate bone density) and / or generate a recommendation for the type of implant for the humeral head based on the bone density metric. For example, a bone density metric indicating sufficient trabecular bone density in the humeral head can result in a system that recommends a stemless humeral implant as opposed to a stemmed humeral implant.

[0311] In one embodiment, the processing circuitry 1142 can be configured to identify the humeral head in the patient-specific image data by, for example, segmenting the bone or otherwise identifying landmarks or shapes indicative of the humeral head. The processing circuitry 1142 can then determine a bone density metric representing the bone density of at least a portion of the humeral head based on the patient-specific image data. Based on the bone density metric, the processing circuitry 1142 can generate a recommendation for the type of humeral implant for the patient. For example, the processing circuitry 1142 can recommend a stemmed humeral implant (stemmed implant type) for a bone density metric indicating lower density bone and the processing circuitry 1142 can recommend a stemless humeral implant (stemless implant type) for a bone density metric indicating higher density bone. The processing circuitry 1142 can then output a recommendation for the type of humeral implant for the patient for display via the user interface.

[0312] In some embodiments, the processing circuitry 1142 can determine a stem length including the stemmed humeral implant type. The processing circuitry 1142 can determine that a longer stem is needed to adequately anchor to less dense bone of the humerus or to determine the location of less dense trabecular bone within the humerus requires a longer stem. The stem length itself can be identified and presented to the user, or the processing circuitry 1142 can recommend a specific humeral implant that meets the recommended length range. As such, the processing circuitry 1142 can recommend a specific implant or implant type selected among three or more different types of humeral implants based on the bone density metric determined from the patient-specific image data.

[0313] In some embodiments, the bone density metric may represent an overall density score of trabecular bone within at least a portion of the humeral head (e.g., a value, index, or category based on voxel or pixel values from the image data). For example, processing circuit 1142 may determine an average or weighted average density of a region of the humeral head and assign a specific metric value to that region of the humeral head. In other embodiments, a bone density metric may be determined that indicates the lowest density bone found in the region to establish a lower bound for the bone density of the region. Conversely, the bone density metric may indicate the highest density within that region of the humeral head. The bone density metric may include multiple bone density values for corresponding portions within the humeral head. For example, the bone density metric may include a matrix of density values within a region of the humeral head, i.e., specific bone density values including corresponding voxels or a set of voxels. As such, the bone density metric may provide a higher resolution representation of the bone density within the humeral head. In any case, the bone density metric may indicate actual bone density values, image data intensities, and / or the type of implant recommended.

[0314] Processing circuit 1142 may determine the bone density metric using different techniques. In some embodiments, processing circuit 1142 may determine the bone density metric by identifying the intensities of corresponding voxels within at least a portion of the humeral head based on patient-specific image data, classifying the intensities of the corresponding voxels into one of two or more intensity levels, and determining the bone density metric based on at least one of the number of voxels classified within each of the two or more intensity levels or the location of the humeral head of the voxels classified within each of the two or more intensity levels. As such, processing circuit 1142 may be configured to classify different intensities in patient-specific image data into different intensity levels and / or the locations of these intensity levels to determine the bone density metric. For example, the location of the intensity level may be related to whether the trabecular bone is dense enough to support a stemless humeral implant. If the trabecular bone has a lower overall bone density, but the center of the humeral head is still above the threshold density required to support a stemless humeral implant, processing circuit 1142 may still determine that the bone density metric is sufficient to support a stemless humeral implant. In other embodiments, even if there is a relatively high bone density level, if a pocket of lower density trabecular bone is identified at the location where a stemless humeral implant is to be implanted, processing circuit 1142 may determine that the bone density metric indicates the need for a stemmed humeral implant.

[0315] In some embodiments, the processing circuitry 1142 may determine a bone density metric of the volume of trabecular bone within the entire humeral head. In other embodiments, the processing circuitry 1142 may determine a plane passing through the humeral head and representing the humeral incision made in the humerus to prepare the humerus for receiving a humeral implant. The humeral incision exposes the surface of the trabecular bone in which the humeral implant is to be implanted. The processing circuitry 1142 then determines a bone density metric of at least a portion of the humeral head bisected by the plane. In some embodiments, the processing circuitry 1142 may determine the bone density metric of the pixels or voxels corresponding to the plane (e.g., exposed by the plane or bisected by the plane). In other embodiments, the processing circuitry 1142 may determine a bone density metric of the volume of trabecular bone starting from the plane and extending towards the axis of the humerus. In some embodiments, the volume of trabecular bone analyzed may extend up to the cortical bone defining the outer surface of the humerus.

[0316] In some embodiments, the bone density metric may be displayed via a user interface (such as using the user interface module 1156, etc.). The processing circuitry 1142 may output such a user interface for display on the display device 1148 or a display device of another system. The user interface includes a graphical representation of the bone density metric located on a representation of at least a portion of the patient's humeral head. The graphical representation of the bone density metric may include a two-dimensional or three-dimensional graph, which may include one or more shapes or colors displayed above or instead of the trabecular bone of the humerus. In one embodiment, the bone density metric may include a heat map of multiple colors, where each color in the multiple colors represents a different range of bone density values. Thus, different colors may represent different bone density magnitudes indicating a spatial representation of the bone density variation within the volume of the trabecular bone. The graphical representation of the bone density metric may include a two-dimensional representation of the bone density variation within the plane of the humeral head. In other embodiments, the graphical representation of the bone density metric may include a three-dimensional representation of the bone density variation within at least the trabecular bone of the humeral head. In some embodiments, the display device 1148 may include a mixed reality display, and the processing circuitry 1142 may control the mixed reality display to present a user interface including a graphical representation of the bone density metric.

[0317] In some embodiments, a bone density metric can be associated with bone density data from other historical patients (e.g., image data or other data indicative of the bone structure in the humeral head) and the type of humeral implant selected by a clinician for that specific bone density data. Bone density data can be generated for these historical patients using patient-specific image data for each patient and the final type of humeral implant selected by a surgeon for each respective patient (e.g., based on a "thumb test" where a clinician uses their thumb to press on the trabecular bone in the humeral head and classifies the trabecular bone as sufficient or insufficient for a stemless humeral implant). A processing circuit 1142 can classify the bone density metric as suitable or unsuitable for a stemless humeral implant for a future patient using these implant types selected based on the thumb test. Thus, the processing circuit 1142 can associate the bone density metric with the type of humeral implant selected by a surgeon in a prior surgery performed on other subjects, where the thumb test data indicates the density range (or compression rate indicative of bone density) of manually determined trabecular bone within the corresponding humeral head of the other subjects. Based on this association, the processing circuit 1142 can determine a recommendation for the type of humeral implant for the patient. In some embodiments, the processing circuit 1142 can employ one or more neural networks to associate the previously selected implant types with the corresponding patient-specific image data to determine bone density metrics indicative of each type of implant available for a future patient. For example, the processing circuit 1142 can use the bone density metric, patient-specific image data, and the selected type of humeral implant (stemmed, stemless, and / or length of the stem) as inputs to a neural network. The output of the neural network can be these bone density metrics corresponding to the type of humeral implant.

[0318] In some embodiments, the processing circuit 1142 can generate a recommendation for a shoulder surgery for a patient using soft tissue features and the bone density metric. For example, the processing circuit 1142 can determine one or more soft tissue features (e.g., soft tissue volume, fat infiltration ratio, atrophy rate, and / or range of motion values) based on patient-specific imaging data and the bone density metric associated with the patient's humerus. As described herein, the processing circuit 1142 can generate a recommendation for the type of shoulder surgery to be performed on the patient (e.g., anatomic or reverse shoulder surgery) and a recommendation for the type of humeral implant for the patient based on the bone density metric associated with the humerus. The processing circuit 1142 can then output the recommendations for the type of shoulder surgery and the type of humeral implant for the patient for display. In some embodiments, the user interface can include a representation of one or more soft tissue features and / or the bone density metric associated with the humerus as part of a mixed reality user interface.

[0319] Figure 39Ais a flowchart showing an exemplary procedure for determining the type of humeral implant based on bone density. The processing circuit 1142 in system 1140 is described as performing Figure 39A in the embodiments, however, other devices or systems such as system 542 or virtual planning system 102 may perform one or more parts of the present technology. Further, a combination of two or more than two devices and / or systems may perform some parts of the present technology via a distributed system. The process in Figure 39A is described with reference to a three-dimensional dataset, however, in other embodiments, several two-dimensional slice data may be analyzed in a similar manner.

[0320] As Figure 39A shown in the embodiments, the processing circuit 1142 may obtain patient-specific image data such as three-dimensional CT image data (e.g., from a memory or other system) (1200). Then, the processing circuit 1142 may identify the humeral head in the patient-specific image data (1202). For example, the processing circuit 1142 may segment the bone to identify the humeral head or determine landmarks or shapes indicative of the humeral head. Using the patient-specific image data of the humeral head, the processing circuit 1142 may determine a bone density metric for at least a portion of the humeral head based on the intensity of the voxels or a set of voxels in the patient-specific image data (1204). The bone density metric may be an overall metric indicating the overall density of the trabecular bone within the humeral head, or the bone density metric may include values representing the density of each voxel in a set of voxels within a region of the humeral head.

[0321] Then, the processing circuit 1142 can determine a recommendation for a humeral implant type based on the bone density metric (1206). For example, when the bone density metric indicates or represents that the density of the trabecular bone is high enough to support a stemless humeral implant, the processing circuit 1142 can determine that the recommendation is a stemless humeral implant. The recommendation can be based on a selection algorithm (e.g., one or more tables, equations, or machine learning algorithms such as neural networks) that may be developed by the processing circuit 1142 (based on historical data related to the humeral implants previously received by the patient). For example, the historical data can include patient-specific image data (e.g., CT data) and the type of humeral implant (e.g., stemless or stemmed) selected by the surgeon for the corresponding patient (e.g., by using a thumb test to determine the trabecular bone quality or density in the humeral head). In one embodiment, the table can map voxel intensities or a set of voxel intensities to a recommendation for a stemmed or stemless implant type. In another embodiment, a first table can map voxel intensities to density values, and a second table can map the density values to a recommendation for a stemmed or stemless implant type. The system can use this mapping of image data to implant selection to inform the recommendation for the implant type for a new patient based on the image data of that patient. Then, the processing circuit 1142 can output a recommendation for the humeral implant type (1208). The recommendation can be sent for use in another recommendation or displayed to the user.

[0322] Figure 39B is a flowchart showing an exemplary procedure for applying a neural network to patient-specific image data to determine the stem size of a humeral implant. The processing circuit 1142 in the system 1140 is described as performing Figure 39B the embodiments in, however, other devices or systems such as the system 542 or the virtual planning system 102 can perform one or more parts of the present technology. Further, a combination of two or more than two devices and / or systems can perform some parts of the present technology via a distributed system. The processes in Figure 39B are described with reference to a three-dimensional data set, however, in other embodiments, several two-dimensional slice data can be analyzed in a similar manner.

[0323] As Figure 39B shown in the embodiments in, the processing circuit 1142 can obtain patient-specific image data such as three-dimensional CT image data (e.g., from a memory or other system) (1210). Then, the processing circuit 1142 selects one or more subsets of the patient-specific image data to be applied to the neural network (1212). For example, the processing circuit 1142 can select a specific portion of the three-dimensional CT image data corresponding to one or more anatomical regions of the humerus. In some embodiments, the three-dimensional CT image data can indicate bone density. Then, the processing circuit 1142 applies the neural network to the selected subset of the patient-specific image data (1214).

[0324] The neural network can include or be based on a convolutional neural network (CNN). As discussed above, a convolutional neural network can include one or more convolutional layers that perform convolutions on input data using learned filters. The filters can also be referred to as kernels. A convolutional neural network can be particularly useful for vision problems such as when the input data includes a still image, such as three-dimensional CT data or other imaging modalities such as MRI data.

[0325] Exemplary CNNs can include an ingestion network (e.g., Inception V net), a residual neural network (e.g., ResNet), or other types of networks that include transfer learning techniques applied based on previous data related to previous humerus diagnoses and / or humerus implant data. In some embodiments, a CNN can include N convolutional layers followed by M fully connected layers. Each layer can include one or more filters, and the layers can be stacked convolutional layers. In some embodiments, one or more filters can be designated for cortical bone, while other one or more filters can be designated for cancellous (e.g., trabecular) bone. The layers of the CNN can be constructed to evaluate several skeletal regions from the metaphysis to the diaphysis of the humerus.

[0326] Processing circuitry 1132 or another processor can train a CNN for humeral stem size prediction, which can include: training a model for dozens, hundreds, or thousands of test images (e.g., CT scan images). The output can be clinical data collected during or after a surgical procedure, such as stem size and fill ratio (i.e., the ratio of the humeral canal to the stem of the implanted humeral implant). The stem size can specify the length and / or cross-sectional width or area of the humeral implant. A shorter stem size can be referred to as a stemless humeral implant, while a longer stem size can be referred to as a stemmed humeral implant. The CNN can also include one or more hyperparameters. Several different types of hyperparameters can be used to reduce the average classification ratio. An exemplary hyperparameter can include a rectified linear unit (ReLU) for the non-linear part, rather than traditional slower solutions such as the Tanh or Sigmond functions. The learning rate of the CNN can depend on one or more variables of an essentially adaptive gradient descent algorithm, such as Adagrad, Adadelta, RMSprop, or Adam, where the learning rate is adaptive based on the type of data fed back into the CNN. The batch size of the data can depend on the learning rate selected for the CNN. The CNN can utilize a momentum value tested according to performance. In one embodiment, the momentum value can be selected from the range of 0.90 to 0.99. However, in other embodiments, other momentum values can be selected. Due to the relatively small training data set, the CNN can include larger weight values. However, for other types of training data sets, these weights may be different.

[0327] A CNN can output the stem size (1216) of a humeral implant based on three-dimensional patient-specific image data to which the CNN is applied. The stem size can include the stem length and / or a cross-sectional dimension (e.g., diameter, perimeter, area, or other such parameter). Thus, a determined stem size corresponding to the specific dimensions and / or bone density of the patient's trabecular bone and / or cancellous bone of the humerus can be selected. Based on the stem size, the processing circuit 1132 can output a recommendation (1218) for the type of humeral implant for the patient. Thus, the processing circuit 1132 can determine whether a stemmed or stemless humeral implant is suitable for treating the patient. In some embodiments, the CNN can be applied to a bone density metric determined from patient-specific imaging data. In other embodiments, the CNN can be used in place of the bone density metric such that the CNN can directly output the stem size of the humeral implant based on the patient-specific imaging data. In some embodiments, the bone density modeling module 1162 can include the CNN and relevant parameters.

[0328] As discussed above, the processing circuit 1132 can train a convolutional neural network and apply the convolutional neural network to patient-specific image data (e.g., 3D imaging data) to generate a recommended stem size (obtained from the patient-specific image data) for the patient's humeral implant. In one embodiment, the system can include a memory configured to store the patient-specific image data of the patient and a processing circuit (e.g., the processing circuit 1132) configured to apply the convolutional neural network to the patient-specific image data (or a subset thereof) and output a stem size for the humeral implant for the patient based on the convolutional neural network applied to the patient-specific image data. The processing circuit can also be configured to output a recommendation for the type of humeral implant including the stem size. The patient-specific image data can represent the bone density of some or all of one or more bones of the patient such as the humerus. In some embodiments, the processing circuit can also output a representation of a bone density metric representing the bone density of at least a portion of the humeral head, however, the bone density metric can be employed or not employed to generate the stem size recommendation when applying the CNN to the patient-specific image data.

[0329] Figure 39C is a flowchart showing an exemplary degree for determining a recommendation for shoulder treatment based on soft tissue structure and bone density determined from patient-specific image data. The processing circuit 1142 in the system 1140 is described as performing Figure 39C the embodiments in, however, other devices or systems such as the system 542 or the virtual planning system 102 can perform one or more parts of the present technology. Further, a combination of two or more than two devices and / or systems can perform some parts of the present technology via a distributed system. The Figure 39Cthe process in, however, in other embodiments, several two-dimensional slice data can be analyzed in a similar manner.

[0330] As Figure 39C shown in the embodiments in, the processing circuit 1142 can determine the features (1220) of one or more soft tissue structures based on patient-specific image data. For example, these features can include reference Figure 23A , Figure 23B , Figure 24 , Figure 25 , Figure 26 , and Figure 27 the volume, fat infiltration ratio, atrophy rate, and / or range of motion of one or more soft tissue structures described. Here, as described in such as Figure 39A in, the processing circuit 1142 can also determine the bone density metric of at least a part of the humeral head based on the intensity of the patient-specific image data.

[0331] The processing circuit 1142 can determine one or more recommendations for shoulder treatment (1224) based on the soft tissue features and the bone density metric. For example, the processing circuit 1142 can determine whether the shoulder replacement is a reverse or anatomic replacement based on one or more soft tissue features. In addition, the processing circuit 1142 can determine whether the type of humeral implant used in the shoulder replacement is a stemless or stemmed humeral implant type. In some embodiments, the processing circuit 1142 can determine the position of at least one of the humeral implant or the glenoid implant based on the soft tissue features and / or the bone density metric. Then, the processing circuit 1142 can output one or more recommendations (1226) determined for the treatment of the patient's shoulder. Thus, the processing circuit 1142 can use any features, metrics, or other information derived from the patient-specific image data and other patient information to provide recommendations related to shoulder treatment.

[0332] Figure 40 is a flowchart showing an exemplary program for displaying bone density information. The processing circuit 1142 in the system 1140 is described as performing Figure 40 the embodiments in, however, other devices or systems such as the system 542 or the virtual planning system 102 can perform one or more parts of the present technology. Further, a combination of two or more than two devices and / or systems can perform some parts of the present technology via a distributed system. The process in Figure 40 is described with reference to a three-dimensional data set, however, in other embodiments, several two-dimensional slice data can be analyzed in a similar manner.

[0333] As Figure 40 shown in the embodiments in, the processing circuit 1142 can be based on such as Figure 39AThe intensity of patient-specific image data such as the processes described determines a bone density metric (1230) for at least a portion of the humeral head. The processing circuitry 1142 can then determine a graphical representation (1232) of the bone density metric. These graphical representations can be similar to the graphical representations of bone density metrics described in Figure 42 and Figure 43 . The processing circuitry 1142 can then control the user interface to present a graphical representation (1234) of the bone density metric over at least a portion of the humeral head.

[0334] Figure 41 is a conceptual diagram of an exemplary user interface 1300 including the humerus 1332 and a cutting plane 1338. As shown in the embodiments of Figure 41 , the user interface 1300 includes a navigation bar 1301 and toolbars 1318 and 1320. The navigation bar 1301 can include selectable buttons that, when selected by the user, cause the user interface 1300 to change to a different function or an information view related to shoulder treatment, such as a planned shoulder replacement, etc.

[0335] The navigation bar 1301 can include a welcome button 1302 that takes the user to a welcome screen that displays information related to the patient or possible actions related to the type of treatment. A plan button 1304 can change the view of the user interface 130 to a virtual plan for shoulder surgery, which can include representations of skeletal and / or soft tissue structures, such as a view 1330 including the humerus 1332, etc. A graft button 1306 can display a view of potential bone or soft tissue grafts related to the surgery, and a humerus cutting button 1308 can display a representation of the humeral head 1332 that is cut to expose the trabecular bone therein. A mounting guide button 1310 can display a possible or recommended humeral implant. A glenoid reaming button 1314 can display a view of an exemplary reaming performed on the glenoid, and a glenoid implant button 1316 can display an exemplary possible or recommended glenoid implant for patient implantation. The toolbar 1318 can include selectable buttons that, when selected, cause the user interface 1300 to change the view, rotation, or size of the view 1330. The toolbar 1320 can include selectable buttons that, when selected, cause the user interface 1300 to transform between anatomical planes of the anatomical structures shown in the view 1330, such as a perspective or side view of the anatomical structure.

[0336] The view 1330 includes a stereogram of the humerus 1332 showing the display axis 1334 and the humeral head 1336. The cutting plane 1338 is shown indicating how the humeral head 1336 can be cut prior to implanting a humeral implant. Although the user interface 1300 can initially display a suggested position of the cutting plane 1338, however, the user interface 1300 enables the user to move the cutting plane 1338 as needed during the planning process. AsFigure 42 and ​ As shown in ​ , once the user is satisfied with the position of the cutting plane 1338, the user interface 1330 can remove the top of the humeral head 1336 to expose a representation of the trabecular bone into which the humeral implant can be implanted.

[0337] ​ FIG. ​ is a conceptual diagram of an exemplary user interface 1300 that includes a representation of a humeral head 1342 and a bone density metric 1344. As ​ shown in the embodiment of ​ , the user interface 1300 can include a view 1340 in which a humeral head 1342 is shown after removing the top of the humeral head along the ​ cutting plane 1338. The humeral head 1342 is a representation of the patient's humerus and can be derived from patient-specific image data. The bone density metric 1344 can be a graphical representation of the bone density metric generated for the trabecular bone of the humerus 1332.

[0338] The bone density metric 1344 can include different colors representing voxels whose intensities fall within corresponding ranges 1346A and 1346B of the intensity of each color. Thus, the bone density metric 1344 can include bone density values for different groups of voxels of the trabecular bone within the humeral head 1342. For example, range 1346A represents a bone density greater than 0.30 g / cm 3 and range 1346B represents a bone density between 0.15 g / cm 3 and 0.30 g / cm 3 . The bone density key 1347 indicates different colors for bone densities within the possible ranges determined from the patient-specific image data. The three ranges shown in the bone density key 1347 are only examples, and in other embodiments, one or more different numbers of ranges with different lower and upper limits can be used.

[0339] In other embodiments, the view 1340 can present the bone density metric 1344, i.e., an image representing the voxel intensity range of the patient-specific image data or an intensity representation of a single or a group of voxels. As one embodiment, the bone density metric 1344 can include only the voxel intensities of the patient-specific image data corresponding to the same cutting plane 1338. In other words, the view 1340 can include a picture of CT data in a 2D plane corresponding to the cutting plane 1338 superimposed on the representation of the exposed humerus 1332. As another embodiment, the view 1340 can include a heat map with different colors and patterns, e.g., corresponding to different ranges of Hounsfield units (e.g., CT data). Thus, although bone density metrics such as the bone density metric 1344 can be related to the representation of bone density, the actual bone density metric may not actually reflect the measurement of the bone density in that region.

[0340] Figure 43 is a conceptual diagram of an exemplary user interface 1300 that includes a representation of a humeral head 1342 and a bone density measure 1352 associated with the type of humeral implant proposed. As Figure 43 shown in the embodiments of Figure 42 similar, shows the humeral head 1342 after moving the top of the humeral head along the Figure 41 cutting plane 1338 in

[0341] The bone density measure 1352 indicates the type of humeral implant to be implanted in the trabecular bone based on the bone density determined for the humerus 1332. Thus, the bone density measure 1352 includes the bone density determined from patient-specific patient data as part of a category associated with the type of humeral implant supported by the density of the bone in the humerus 1332. The measure key 1354 indicates the color of the bone density measure 1352 corresponding to the type of humeral implant. For example, a light color indicates that a stemless humeral implant can be implanted and a dark color indicates that a stemmed humeral implant can be implanted in the humerus 1332. As Figure 43 shown in the embodiments of

[0342] The following embodiments are described herein.

[0343] Example 1: A system for modeling a patient's soft tissue structure, the system comprising: a memory configured to store patient-specific image data of a patient; and a processing circuit configured to: receive the patient-specific image data; determine a patient-specific shape representing the patient's soft tissue structure based on the intensity of the patient-specific image data; and output the patient-specific shape.

[0344] Example 2: The system according to Example 1, wherein the processing circuit is configured to: receive an initial shape; determine a plurality of surface points on the initial shape; register the initial shape with patient-specific image data; identify one or more contours in the patient-specific image data, the one or more contours representing at least partial boundaries of the patient's soft tissue structures; and iteratively move the plurality of surface points toward corresponding positions of the one or more contours to change the initial shape into a patient-specific shape representing the patient's soft tissue structures.

[0345] Example 3: The system according to Example 2, wherein the processing circuit is configured to identify one or more contours by: extending vectors from each surface point in at least one way, either outward or inward, from the corresponding surface point among the plurality of surface points; and determining, for the vectors of each surface point, corresponding positions in the patient-specific image data that exceed a threshold intensity value, wherein the corresponding positions of at least one surface point among the plurality of surface points at least partially define the one or more contours.

[0346] Example 4: The system according to any one of Examples 2 and 3, wherein the processing circuit is configured to identify one or more contours by: determining a Hessian eigenimage from the patient-specific image data, wherein the Hessian eigenimage represents a region of the patient-specific image data that includes higher intensity gradients between two or more than two voxels; identifying one or more separation regions between the soft tissue structures and adjacent soft tissue structures based on the Hessian eigenimage; and determining at least a portion of the one or more contours as passing through the one or more separation regions.

[0347] Example 5: The system according to any one of Examples 2 to 4, wherein the processing circuit is configured to determine the corresponding positions in the patient-specific image data that exceed the threshold intensity value by determining the corresponding positions in the patient-specific image data that are greater than a predetermined intensity value.

[0348] Example 6: The system according to Example 5, wherein the predetermined threshold intensity value represents bone in the patient-specific image data, and wherein, for each corresponding position in the patient-specific image data that exceeds the predetermined threshold intensity value representing bone, the processing circuit is configured to move the surface point to the corresponding position.

[0349] Example 7: The system according to any one of Examples 2 to 6, wherein the processing circuit is configured to determine the corresponding positions in the patient-specific image data that exceed the threshold intensity value by determining the corresponding positions in the patient-specific image data that are less than a predetermined intensity value.

[0350] Example 8: The system according to any one of Examples 2 to 7, wherein the processing circuit is configured to determine corresponding positions in the patient-specific image data that exceed a threshold intensity value by determining corresponding positions in the patient-specific image data that are greater than a difference threshold between the intensity associated with a corresponding surface point and the intensity at the corresponding position in the patient-specific image data.

[0351] Example 9: The system according to any one of Examples 2 to 8, wherein the processing circuit is configured to iteratively move a plurality of surface points towards corresponding positions of one or more contours by, for each iteration of moving the plurality of surface points: extending a vector from each surface point of the plurality of surface points that starts at the corresponding surface point and is orthogonal to the surface including the corresponding surface point; determining a corresponding point in the patient-specific image data that exceeds a threshold intensity value for the vector of each surface point; determining a plurality of potential positions within an envelope at the corresponding point and exceeding the threshold intensity value in the patient-specific image data, wherein the plurality of potential positions at least partially define a surface of one or more contours; for each potential position of the plurality of potential positions, determining a corresponding normal vector orthogonal to the surface; determining an angle between the corresponding normal vector and the vector of the corresponding surface point for each normal vector of the corresponding normal vectors; selecting, for each corresponding surface point, one potential position of the plurality of potential positions that includes a minimum angle between the vector of the corresponding surface point and the corresponding normal vector of each potential position of the plurality of potential positions; and for each corresponding surface point, at least partially moving the corresponding surface point towards the selected one potential position, wherein moving the corresponding surface point causes an initial shape to be modified towards a patient-specific shape.

[0352] Example 10: The system according to Example 9, wherein the processing circuit is configured to move the corresponding surface point by at least half the distance between the corresponding surface point and the selected one potential position.

[0353] Example 11: The system according to any one of Examples 9 and 10, wherein the processing circuit is configured to iteratively move a plurality of surface points towards corresponding potential positions in one or more contours by: in a first iteration, moving each surface point of the plurality of surface points from an initial shape by a first corresponding distance to generate a second shape, the first corresponding distance being within a first tolerance of a first modification distance, the first tolerance being selected to maintain smoothness of the second shape; and in a second iteration after the first iteration, moving each surface point of the plurality of surface points by a second corresponding distance to generate a third shape from the second shape, the second corresponding distance being within a second tolerance of a second modification distance, wherein the second tolerance is greater than the first tolerance.

[0354] Example 12: The system according to any one of Examples 2 to 11, wherein the processing circuit is configured to identify one or more contours by: determining a Hessian feature image from patient-specific image data, wherein the Hessian feature image represents a region of the patient-specific image data that includes a higher intensity gradient between two or more voxels; identifying one or more separation regions between the soft tissue structure and adjacent soft tissue structures based on the Hessian feature image; and determining at least a portion of the one or more contours as passing through the one or more separation regions.

[0355] Example 13: The system according to any one of Examples 2 to 12, wherein the processing circuit is configured to register the initial shape by registering a plurality of positions on the initial shape with corresponding insertion positions on one or more bones identified in the patient-specific image data.

[0356] Example 14: The system according to any one of Examples 2 to 13, wherein the initial shape and the patient-specific shape are three-dimensional shapes.

[0357] Example 15: The system according to any one of Examples 1 to 14, wherein the initial shape includes a geometric shape.

[0358] Example 16: The system according to any one of Examples 1 to 15, wherein the initial shape includes an anatomical shape representing soft tissue structures of a plurality of subjects different from the patient.

[0359] Example 17: The system according to Example 16, wherein the anatomical shape includes a statistical average shape generated from soft tissue structures imaged for a plurality of subjects.

[0360] Example 18: The system according to any one of Examples 1 to 17, wherein the patient-specific image data includes computed tomography (CT) image data generated from the patient.

[0361] Example 19: The system according to any one of Examples 1 to 18, wherein the soft tissue structure includes muscle.

[0362] Example 20: The system according to Example 19, wherein the muscle is associated with the patient's rotator cuff.

[0363] Example 21: The system according to any one of Examples 1 to 20, wherein the patient-specific shape includes a three-dimensional shape.

[0364] Example 22: The system according to any one of Examples 1 to 21, wherein the processing circuit is configured to: determine a fat volume ratio of a patient-specific shape; determine an atrophy rate of the patient-specific shape; determine a range of motion of the patient's humerus based on the fat volume ratio and atrophy rate of the patient-specific shape of the patient's soft tissue structure; and determine a type of shoulder treatment for the patient based on the range of motion of the humerus.

[0365] Example 23: The system according to Example 22, wherein the processing circuit is configured to determine the range of motion of the humerus by determining the range of motion of the humerus based on the fat volume ratio and atrophy rate of each muscle of the patient's rotator cuff.

[0366] Example 24: The system according to any one of Examples 22 and 23, wherein the type of shoulder treatment is selected from one of an anatomical shoulder replacement surgery or a reverse shoulder replacement surgery.

[0367] Example 25: The system according to any one of Examples 1 to 24, wherein the processing circuit is configured to: apply a mask to the patient-specific shape; apply a threshold to the voxels under the mask; determine a fat volume based on the voxels below the threshold; determine a fat penetration value based on the fat volume and the volume of the patient-specific shape of the soft tissue structure; and output the fat penetration value of the soft tissue structure.

[0368] Example 26: The system according to any one of Examples 1 to 25, wherein the processing circuit is configured to: determine a bone-to-muscle scale of the patient's soft tissue structure; obtain a statistical mean shape (SMS) of the soft tissue structure; deform the SMS by satisfying a threshold of an algorithm so that a deformed version of the SMS fits the bone-to-muscle scale of the soft tissue structure; determine an atrophy rate of the soft tissue structure by dividing the SMS volume by the soft tissue structure volume; and output the atrophy rate of the soft tissue structure.

[0369] Example 27: A method for modeling a patient's soft tissue structure, the method comprising: storing, by a memory, patient-specific image data of a patient; receiving, by a processing circuit, the patient-specific image data; determining, by the processing circuit, a patient-specific shape representing the patient's soft tissue structure based on the intensity of the patient-specific image data; and outputting, by the processing circuit, the patient-specific shape.

[0370] Example 28: The method according to Example 27, further comprising: receiving an initial shape; determining a plurality of surface points on the initial shape; registering the initial shape with the patient-specific image data; identifying one or more contours in the patient-specific image data, the one or more contours representing the boundaries of the patient's soft tissue structure; and iteratively moving the plurality of surface points towards the corresponding positions of the one or more contours to change the initial shape into a patient-specific shape representing the patient's soft tissue structure.

[0371] Example 29: The method according to Example 28, wherein one or more contours are identified by: extending a vector from each surface point in at least one way that is either outward or inward from the corresponding surface point among a plurality of surface points; and determining, for the vector of each surface point, a corresponding position in the patient-specific image data that exceeds a threshold intensity value, wherein the corresponding position of at least one surface point among the plurality of surface points at least partially defines one or more contours.

[0372] Example 30: The method according to any one of Examples 28 and 29, wherein identifying one or more contours includes: determining a Hessian feature image from the patient-specific image data, wherein the Hessian feature image represents a region of the patient-specific image data that includes a higher intensity gradient between two or more than two voxels;

[0373] identifying one or more separation regions between the soft tissue structure and adjacent soft tissue structures based on the Hessian feature image; and determining at least a portion of one or more contours to pass through one or more separation regions.

[0374] Example 31: The method according to any one of Examples 28 to 30, wherein determining the corresponding position in the patient-specific image data that exceeds the threshold intensity value includes: determining the corresponding position in the patient-specific image data that is greater than a predetermined intensity value.

[0375] Example 32: The method according to Example 32, wherein the predetermined threshold intensity value represents bone in the patient-specific image data, and wherein the method further includes: for each corresponding position in the patient-specific image data that exceeds the predetermined threshold intensity value representing bone, moving the surface point to the corresponding position.

[0376] Example 33: The method according to any one of Examples 28 to 32, wherein determining the corresponding position in the patient-specific image data that exceeds the threshold intensity value includes: determining the corresponding position in the patient-specific image data that is less than a predetermined intensity value.

[0377] Example 34: The method according to any one of Examples 28 to 33, wherein determining the corresponding position in the patient-specific image data that exceeds the threshold intensity value includes: determining the corresponding position in the patient-specific image data that is greater than a difference threshold between two values, the two values including the intensity associated with the corresponding surface point and the intensity at the corresponding position in the patient-specific image data.

[0378] Example 35: The method according to any one of Examples 28 to 34, wherein iteratively moving a plurality of surface points towards corresponding positions of one or more contours includes, for each iteration of moving the plurality of surface points: extending a vector from each surface point of the plurality of surface points that starts at the corresponding surface point and is orthogonal to the surface including the corresponding surface point; determining, for the vector of each surface point, a corresponding point in the patient-specific image data that exceeds a threshold intensity value; determining, for each corresponding point, a plurality of potential positions that are located within the envelope of the corresponding point and exceed the threshold intensity value in the patient-specific image data, wherein the plurality of potential positions at least partially define the surface of one or more contours; determining, for each potential position of the plurality of potential positions, a corresponding normal vector that is orthogonal to the surface; determining, for each normal vector of the corresponding normal vectors, an angle between the corresponding normal vector and the vector of the corresponding surface point; selecting, for each corresponding surface point, one potential position of the plurality of potential positions, the one potential position including the smallest angle between the vector of the corresponding surface point and the corresponding normal vector of each potential position of the plurality of potential positions; and for each corresponding surface point, moving the corresponding surface point at least partially towards the selected one potential position, wherein moving the corresponding surface point causes the initial shape to be modified towards the patient-specific shape.

[0379] Example 36: The method according to Example 35, further comprising: moving the corresponding surface point by at least half the distance between the corresponding surface point and the selected one potential position.

[0380] Example 37: The method according to any one of Examples 35 and 36, wherein iteratively moving a plurality of surface points towards corresponding potential positions of one or more contours includes: in a first iteration, moving each surface point of the plurality of surface points from the initial shape by a first corresponding distance to generate a second shape, the first corresponding distance being within a first tolerance of a first modification distance, the first tolerance being selected to maintain the smoothness of the second shape; and in a second iteration after the first iteration, moving each surface point of the plurality of surface points by a second corresponding distance to generate a third shape from the second shape, the second corresponding distance being within a second tolerance of a second modification distance, wherein the second tolerance is greater than the first tolerance.

[0381] Example 38: The method according to any one of Examples 28 to 37, wherein the processing circuit is configured to identify one or more contours by: determining a Hessian feature image from the patient-specific image data, wherein the Hessian feature image represents a region of the patient-specific image data that includes a higher intensity gradient between two or more voxels; identifying one or more separation regions between soft tissue structures and adjacent soft tissue structures based on the Hessian feature image; and determining at least a portion of one or more contours as passing through the one or more separation regions.

[0382] Example 39: The method according to any one of Examples 28 to 38, wherein registering the initial shape includes: registering a plurality of positions on the initial shape with corresponding insertion positions on one or more bones identified in the patient-specific image data.

[0383] Example 40: The method according to any one of Examples 38 to 30, wherein the initial shape and the patient-specific shape are three-dimensional shapes.

[0384] Example 41: The method according to any one of Examples 27 to 40, wherein the initial shape includes a geometric shape.

[0385] Example 42: The method according to any one of Examples 27 to 41, wherein the initial shape includes an anatomical shape representing soft tissue structures of a plurality of subjects different from the patient.

[0386] Example 43: The method according to Example 42, wherein the anatomical shape includes a statistical average shape generated from soft tissue structures imaged for a plurality of subjects.

[0387] Example 44: The method according to any one of Examples 27 to 43, wherein the patient-specific image data includes computed tomography (CT) image data generated from the patient.

[0388] Example 45: The method according to any one of Examples 27 to 44, wherein the soft tissue structure includes muscle.

[0389] Example 46: The method according to Example 45, wherein the muscle is associated with the patient's rotator cuff.

[0390] Example 47: The method according to any one of Examples 27 to 46, wherein the patient-specific shape includes a three-dimensional shape.

[0391] Example 48: The method according to any one of Examples 27 to 47, further comprising: determining a fat volume ratio of the patient-specific shape; determining an atrophy rate of the patient-specific shape; determining a range of motion of the patient's humerus based on the fat volume ratio and the atrophy rate of the patient-specific shape of the patient's soft tissue structure; and determining a type of treatment for the patient's shoulder based on the range of motion of the humerus.

[0392] Example 49: The method according to Example 48, wherein determining the range of motion of the humerus includes: determining the range of motion of the patient's humerus based on the fat volume ratio and the atrophy rate of each muscle of the patient's rotator cuff.

[0393] Example 50: The method according to any one of Examples 48 and 49, wherein the type of shoulder treatment is selected from one of anatomical shoulder replacement surgery or reverse shoulder replacement surgery.

[0394] Example 51: The method according to any one of Examples 27 to 50, further comprising: applying a mask to the patient-specific shape; applying a threshold to the voxels under the mask; determining a fat volume based on the voxels below the threshold; determining a fat infiltration value based on the fat volume and the volume of the patient-specific shape of the soft tissue structure; and outputting a fat volume ratio of the soft tissue structure.

[0395] Example 52: The method according to any one of Examples 27 to 51, further comprising: determining a bone-to-muscle scale of the soft tissue structure of the patient; obtaining a statistical mean shape (SMS) of the soft tissue structure; deforming the SMS by satisfying a threshold of an algorithm such that a deformed version of the SMS fits the bone-to-muscle scale of the soft tissue structure; determining an atrophy rate of the soft tissue structure by dividing the SMS volume by the soft tissue structure volume; and outputting the atrophy rate of the soft tissue structure.

[0396] Example 53: A computer-readable storage medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to: store patient-specific image data of a patient in a memory; receive the patient-specific image data; determine a patient-specific shape representing the soft tissue structure of the patient based on the intensity of the patient-specific image data; and output the patient-specific shape.

[0397] Example 54: A system for modeling a soft tissue structure of a patient, the system comprising: means for storing patient-specific image data of the patient; means for receiving the patient-specific image data; means for determining a patient-specific shape representing the soft tissue structure of the patient based on the intensity of the patient-specific image data; and means for outputting the patient-specific shape.

[0398] Example 101: A system for modeling a soft tissue structure of a patient, the system comprising: a memory configured to store patient-specific computed tomography (CT) data of the patient; and a processing circuit configured to: receive the patient-specific CT data; identify one or more locations within the patient-specific CT data associated with one or more bone structures; register an initial shape with the one or more locations; modify the initial shape into a patient-specific shape representing the soft tissue structure of the patient; and output the patient-specific shape.

[0399] Example 102: The system according to Example 101, wherein the one or more positions associated with the one or more bone structures include one or more insertion positions of the one or more bone structures identified in the patient-specific CT data.

[0400] Example 103: The system according to Example 102, wherein the processing circuit is configured to: identify one or more contours in the patient-specific CT data, the one or more contours representing at least partial boundaries of the patient's soft tissue structures; and modify the initial shape based on the one or more contours into a patient-specific shape representing the patient's soft tissue structures.

[0401] Example 104: The system according to Example 103, wherein the processing circuit is configured to: determine a plurality of surface points on the initial shape; and modify the initial shape by iteratively moving the plurality of surface points towards corresponding positions of the one or more contours to change the initial shape into a patient-specific shape representing the patient's soft tissue structures.

[0402] Example 105: The system according to any one of Examples 103 and 104, wherein the processing circuit is configured to modify the initial shape by: extending vectors from each surface point in at least one of an outward or inward manner from the corresponding surface point of the plurality of surface points; and determining corresponding positions in the patient-specific CT data that exceed a threshold intensity value for the vectors of each surface point, wherein the corresponding positions for at least one of the plurality of surface points at least partially define the one or more contours.

[0403] Example 106: The system according to any one of Examples 103 to 105, wherein the processing circuit is configured to identify the one or more contours by: determining a Hessian feature image from the patient-specific CT data, wherein the Hessian feature image represents a region of the patient-specific CT data that includes higher intensity gradients between two or more voxels; identifying one or more separation regions between the soft tissue structures and adjacent soft tissue structures based on the Hessian feature image; and determining at least a portion of the one or more contours as passing through the one or more separation regions.

[0404] Example 107: The system according to any one of Examples 101 to 106, wherein the processing circuit is configured to register the initial shape with one or more positions by: determining a correspondence between each of the one or more positions and a corresponding point on the initial shape; determining an intensity profile along each correspondence in the patient-specific CT data; determining a distance between a position and a corresponding point on the initial shape for each of the one or more positions and based on the intensity profile of the corresponding correspondence; and orienting the initial shape in the patient-specific CT data according to the respective distances between the one or more positions and the points on the initial shape.

[0405] Example 108: The system according to Example 107, wherein the processing circuit is configured to modify the initial shape into a patient-specific shape by scaling the initial shape such that the difference between the initial shape and the variation of the patient-specific CT data representing the soft tissue structure is minimized.

[0406] Example 109: The system according to Example 108, wherein the processing circuit is configured to determine the patient-specific shape according to the following parametric equation:

[0407]

[0408] where s’ is an initial shape representing the soft tissue structure of a plurality of subjects different from the patient, λ i is an eigenvalue and v i is an eigenvector of the covariance matrix representing the variation of the patient-specific CT data, and b i is a scaling factor for modifying the initial shape.

[0409] Example 110: The system according to any one of Examples 101 to 109, wherein the initial shape includes an anatomical shape representing the soft tissue structure of a plurality of subjects different from the patient.

[0410] Example 111: The system according to Example 110, wherein the anatomical shape includes a statistical average shape generated from the soft tissue structures imaged for a plurality of subjects.

[0411] Example 112: The system according to any one of Examples 101 to 111, wherein the soft tissue structure includes muscle.

[0412] Example 113: The system according to Example 112, wherein the muscle is associated with the patient's rotator cuff.

[0413] Example 114: The system according to any one of Examples 101 to 113, wherein the patient-specific shape includes a three-dimensional shape.

[0414] Example 115: The system according to any one of Examples 101 to 114, wherein the processing circuit is configured to: determine a fat volume ratio of a patient-specific shape; determine an atrophy rate of the patient-specific shape; determine a range of motion of the patient's humerus based on the fat volume ratio and the atrophy rate of the patient-specific shape of the patient's soft tissue structure; and determine a type of shoulder treatment for the patient based on the range of motion of the humerus.

[0415] Example 116: The system according to Example 115, wherein the type of shoulder treatment is selected from one of an anatomical shoulder replacement surgery or a reverse shoulder replacement surgery.

[0416] Example 117: A method for modeling a patient's soft tissue structure, the method comprising: storing patient-specific computed tomography (CT) data of the patient in a memory; receiving, by a processing circuit, the patient-specific CT data; identifying, by the processing circuit, one or more positions associated with one or more bone structures within the patien...

Claims

1. A system for automatically generating shoulder surgery recommendations for a patient, the system comprising: a memory configured to store patient-specific image data for the patient; and a processing circuit configured to: receive the patient-specific image data from the memory; determine, based on the patient-specific image data, one or more soft tissue features and a bone density metric associated with the patient's humerus, wherein the one or more soft tissue features include at least one of a fat infiltration value or an atrophy rate of one or more soft tissue structures of the patient; generate a recommendation for the type of shoulder surgery to be performed on the patient based on at least one of the fat infiltration value or the atrophy rate of the one or more soft tissue structures of the patient; generate a recommendation for the type of humeral implant for the patient based on the bone density metric associated with the humerus; and output, for the patient, the recommendations for the type of shoulder surgery and the type of humeral implant.

2. The system according to claim 1, wherein, The type of humeral implant includes one of a stemmed implant type or a stemless implant type.

3. The system according to claim 1, wherein The recommendation for the type of humeral implant includes a recommendation indicating the length of the stem of the humeral implant.

4. The system according to claim 1, wherein The processing circuit is configured to output a user interface for display, the user interface including a graphical representation of the bone density metric on a representation of the humerus.

5. The system according to claim 1, wherein, The one or more soft tissue features include the fat infiltration value of the one or more soft tissue structures of the patient.

6. The system according to claim 5, wherein, The processing circuit is configured to determine the fat infiltration value by: applying a mask to a patient-specific shape representing the one or more soft tissue structures; applying a threshold to the voxels under the mask; determining a fat volume based on the voxels under the threshold; and determining the fat infiltration value based on the fat volume and the volume of the patient-specific shape representing the one or more soft tissue structures.

7. The system according to claim 1, wherein The one or more soft tissue features include the atrophy rate of the one or more soft tissue structures.

8. The system according to claim 7, wherein, The processing circuit is configured to determine the atrophy rate by: determining a bone-to-muscle scale of the one or more soft tissue structures; obtaining a statistical mean shape (SMS) of the one or more soft tissue structures; deforming the SMS in a manner that meets a threshold of an algorithm to fit a deformed version of the SMS to the bone-to-muscle scale of the one or more soft tissue structures; and determining the atrophy rate of the one or more soft tissue structures by dividing the volume of the deformed SMS by the volume of the soft tissue structure.

9. The system according to claim 1, wherein The processing circuit is configured to use a neural network to determine at least one of the one or more soft tissue features and the bone density metric associated with the humerus.

10. The system according to claim 10, wherein the processing circuit is configured to determine the one or more soft tissue features by: receiving an initial shape of the one or more soft tissue structures; determining a plurality of surface points on the initial shape; registering the initial shape with the patient-specific image data; Identify one or more contours representing the boundaries of the one or more soft tissue structures in the patient-specific image data; Iteratively move the plurality of surface points towards the respective positions of the one or more contours to change the initial shape into a patient-specific shape representing the one or more soft tissue structures; And Based on the patient-specific shape, determine the one or more soft tissue characteristics of the one or more soft tissue structures.

11. The system according to claim 1, wherein, The processing circuit is configured to control a user interface to display a representation of the one or more soft tissue characteristics.

12. The system according to claim 11, wherein, The processing circuit is configured to control the user interface to display a representation of the one or more soft tissue characteristics and at least one of the bone density metrics associated with the humerus as part of a mixed reality user interface.

13. The system according to claim 1, wherein, The type of shoulder surgery includes one of anatomic shoulder replacement or reverse shoulder replacement.

14. A method for automatically generating shoulder surgery recommendations for a patient, the method comprising: Receiving patient-specific image data from a memory; Determining, by a processing circuit, one or more soft tissue characteristics and a bone density metric associated with the patient's humerus based on the patient-specific image data, wherein the one or more soft tissue characteristics include at least one of a fat infiltration value or an atrophy rate of the one or more soft tissue structures of the patient; Generating, by the processing circuit, a recommendation for the type of shoulder surgery to be performed on the patient based on at least one of the fat infiltration value or the atrophy rate of the one or more soft tissue structures; Generating, by the processing circuit, a recommendation for the type of humeral implant for the patient based on the bone density metric associated with the humerus; and Outputting, by the processing circuit, recommendations for the type of shoulder surgery and the type of humeral implant for the patient.

15. The method according to claim 14, wherein, The type of humeral implant includes one of a stemmed implant type or a stemless implant type.

16. The method according to claim 14, wherein, The recommendation for the type of humeral implant includes a recommendation indicating the length of the stem of the humeral implant.

17. The method according to claim 14, further comprising outputting a user interface for display, the user interface including a graphical representation of the bone density metric on a representation of the humerus.

18. The method according to claim 14, wherein, The one or more soft tissue characteristics include the fat infiltration value of the one or more soft tissue structures.

19. The method according to claim 18, wherein, Determining the fat infiltration value includes: Applying a mask to the patient-specific shape representing the one or more soft tissue structures; Applying a threshold to the voxels under the mask; Determining a fat volume based on the voxels under the threshold; and Determining the fat infiltration value based on the fat volume and the volume of the patient-specific shape representing the one or more soft tissue structures.

20. The method according to claim 14, wherein The one or more soft tissue characteristics include the atrophy rate of the one or more soft tissue structures.

21. The method according to claim 20, wherein, Determining the atrophy rate includes: Determining the bone-to-muscle scale of the one or more soft tissue structures; Obtaining a statistical mean shape (SMS) of the one or more soft tissue structures; Deforming the SMS in a manner that meets the threshold of the algorithm such that a deformed version of the SMS fits the bone-to-muscle scale of the one or more soft tissue structures; and The atrophy rate of the one or more soft tissue structures is determined by dividing the volume of the deformed SMS by the volume of the soft tissue structure.

22. The method according to claim 14, wherein Determining at least one of the one or more soft tissue characteristics and the bone density metric associated with the humerus includes using a neural network to determine at least one of the one or more soft tissue characteristics and the bone density metric associated with the humerus.

23. The method according to claim 14, wherein, Determining the one or more soft tissue characteristics includes: Receiving an initial shape of the one or more soft tissue structures; Determining a plurality of surface points on the initial shape; Registering the initial shape with the patient-specific image data; Identifying one or more contours in the patient-specific image data that represent the boundaries of the one or more soft tissue structures; Iteratively moving the plurality of surface points towards corresponding positions of the one or more contours to change the initial shape to a patient-specific shape representing the one or more soft tissue structures; and Based on the patient-specific shape, determining the one or more soft tissue characteristics of the one or more soft tissue structures.

24. A system for automatically generating a recommendation for a type of shoulder surgery to be performed on a patient, the system comprising: A memory configured to store patient-specific computed tomography (CT) image data generated from a patient; And A processing circuit configured to: Receive the patient-specific CT image data from the memory; Determine one or more soft tissue characteristics of the patient's soft tissue structures from the patient-specific CT image data by at least: Receiving an initial shape of the soft tissue structure; Determining a plurality of surface points on the initial shape; Registering the initial shape with the patient-specific CT image data; Identifying one or more contours in the patient-specific CT image data that represent the boundaries of the soft tissue structures; Iteratively moving the plurality of surface points towards corresponding positions of the one or more contours to change the initial shape to a patient-specific shape representing the soft tissue structures, wherein for each iteration of iteratively moving the plurality of surface points, the processing circuit is configured to perform at least the following for each corresponding surface point of the plurality of surface points: Determining a corresponding vector orthogonal to the initial shape at the corresponding surface point; Determining a plurality of potential positions associated with the one or more contours based on the corresponding vector; For each potential position of the plurality of potential positions, determining a position vector orthogonal to the one or more contours at the potential position; Calculating the angle from the corresponding vector to each position vector of the plurality of potential positions; Based on the angles from the corresponding vector to each position vector of the plurality of potential positions, selecting a single potential position from the plurality of potential positions; and Moving the corresponding surface point at least partially towards the single potential position selected for the corresponding surface point; and Determining the one or more soft tissue characteristics based on the patient-specific shape; Generating the recommendation for the type of shoulder surgery based on the one or more soft tissue characteristics; and Outputting the recommendation for the type of shoulder surgery.

25. A system for modeling a patient's soft tissue structure, the system comprising: a memory configured to store patient-specific computed tomography (CT) data of a patient; and a processing circuit configured to: receive the patient-specific CT data; identify one or more locations associated with one or more bone structures within the patient-specific CT data; register an initial shape to the one or more locations; modify the initial shape to a patient-specific shape representing the patient's soft tissue structure; and output the patient-specific shape.

Images