Algorithm-based optimization, tools and alternative simulation data for total hip replacement

Abstract

A method and system for performing hip replacement surgery includes analyzing images of a patient's hip joint at multiple locations to identify preoperative hip joint geometry. For multiple selected patient activities, a statistical patient model predicts prosthetic hip implant performance based on preoperative knee joint geometry and given prosthetic knee implant implant parameters, each selected patient activity having a predefined motion curve to calculate an optimized surgical plan for performing the procedure using a computer-assisted surgery system that can use reference markers fixed to patient tissue. Hip joint geometry can be determined by angles between landmarks in the image, including sacral tilt, pelvic incidence, pelvic femoral angle, and anteversion in an X-ray image. Implant performance criteria can include, for example, edge loads and range of motion of implant components.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Applications 62 / 801,245 (filed on February 5, 2019), 62 / 801,257 (filed on February 5, 2019), 62 / 864,663 (filed on June 21, 2019), 62 / 885,673 (filed on August 12, 2019), and 62 / 939,946 (filed on November 25, 2019), which are incorporated herein in their entirety. Technical Field

[0003] The present disclosure generally relates to methods, systems and apparatus related to computer-assisted surgery systems, which include various hardware and software components that work together to enhance surgical procedures. The disclosed technology can be applied to shoulder, hip and knee replacement surgeries, for example. Background Art

[0004] Common types of joint replacements, such as partial knee arthroplasty (PKA), total knee arthroplasty (TKA) or total hip arthroplasty (THA) utilize surgical planning to define one or more predetermined cutting planes to resect bone to accommodate the implant orientation and position (posture) of the knee or hip implant / replacement joint. By resecting the bone according to the surgical plan, the patient's bone can be shaped into a standardized, planned manner to accept a joint replacement implant with a given posture. The exact orientation and position of the joint replacement implant is typically planned according to a surgical plan developed before the start of surgery. However, the surgeon will often modify the plan in the operating room based on the information collected about the patient's joint. Various systems exist to improve surgical planning and workflow, but there is still room for improvement. Summary of the invention

[0005] In some embodiments, a method for performing a knee replacement procedure includes: receiving multiple images of a patient's musculoskeletal anatomy (including images of the patient's hip joint in a standing position and images of the patient's hip joint in a sitting position); identifying anatomical angles within the multiple images to define a preoperative hip joint geometry; and selecting one or more postoperative patient activities from a plurality of model activities to consider when optimizing prosthetic hip implant performance from a set of available postoperative patient activities. The method also includes: accessing a statistical patient model, the statistical patient model predicting prosthetic hip implant performance based on the preoperative hip joint geometry and prosthetic hip implant implant parameters for motion curves representing each of the multiple model activities; calculating a set of proposed hip implant implant parameters from the statistical patient model, the parameters providing predicted prosthetic hip implant performance that meets predetermined performance criteria for the selected one or more postoperative patient activities and the preoperative hip joint geometry. The method also includes performing the hip replacement procedure using a computer-assisted surgery system according to a surgical plan to implant a prosthetic device having the proposed hip implant implant parameters.

[0006] The anatomical angles may include at least one of sacral tilt, pelvic incidence, pelvic femoral angle, and anteversion. The predetermined performance criteria may include an estimate of implant edge load during each motion curve representing each of the selected one or more postoperative patient activities or an estimate of implant range of motion during each motion curve representing each of the selected one or more postoperative patient activities, and the multiple images may be X-ray images.

[0007] The steps may also include providing a user interface to allow a user to select the plurality of activities, and graphically displaying the expected postoperative performance of the implant for the selected plurality of activities in one or more polar coordinate plots. In addition, the step of identifying anatomical angles may be performed manually by identifying landmark anatomical feature locations in the plurality of images, or may be performed automatically by a processor by identifying landmark anatomical feature locations in the plurality of images.

[0008] In some embodiments, a computer-assisted surgery system for performing a hip replacement procedure includes the results of multiple simulations of model iliac joint performance for an iliac joint, wherein each simulation: models a hip joint having one of a plurality of hypothetical hip joint geometries undergoing a predetermined motion profile corresponding to one of a plurality of patient activities. The system also includes a statistical model computationally created from the simulation database, the statistical model predicting hip implant performance for a given hip joint geometry performing at least one of the plurality of patient activities; a first set of software instructions instructing one or more processors to receive a plurality of patient images, the plurality of patient images comprising at least an image of a patient's bony anatomy in a sitting position and an image of a patient's bony anatomy in a standing position, and identifying anatomical angles within the plurality of images to define a preoperative hip joint geometry. A second set of software instructions instructs the one or more processors to determine recommended hip implant implantation parameters such that the predicted performance of the hip implant meets predetermined postoperative performance criteria for the preoperative hip joint geometry and a set of patient activities selected by a user, and to update a surgical plan to configure the computer-assisted surgery system to assist the surgeon in resecting the patient's bone to achieve the implantation parameters.

[0009] The system may include a user interface that allows a user to select a user-selected set of patient activities and graphically displays in one or more polar plots the expected postoperative performance of the implant for the user-selected set of patient activities. The first set of software instructions may instruct the one or more processors to identify anatomical angles within the plurality of images by automatically identifying landmark anatomical feature locations in the plurality of images and facilitate user manipulation of placement of the landmark anatomical feature locations. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Various embodiments are shown by way of example in the figures of the accompanying drawings. Such embodiments are exemplary and are not intended to be exhaustive or exclusive embodiments of the inventive subject matter.

[0011] Figure 1 An exemplary computer-assisted surgery system for use in some embodiments is shown;

[0012] Figure 2A shows some examples of control instructions that may be used by a surgical computer according to some embodiments;

[0013] Figure 2B shows some examples of data that may be used by a surgical computer according to some embodiments;

[0014] Figure 2C is a system diagram illustrating an example of a cloud-based system used by a surgical computer according to some embodiments;

[0015] Figure 3A provides a high-level overview of how a surgical computer may generate recommendations according to some embodiments;

[0016] Figure 3B An exemplary implant placement interface according to some embodiments is shown;

[0017] Figure 3C shows an exemplary gap planning interface according to some embodiments;

[0018] Figure 3D shows an exemplary optimization parameterization interface according to some embodiments;

[0019] Figure 3E shows an exemplary response and justification interface according to some embodiments;

[0020] Figure 4 A system diagram is provided that illustrates how optimization of surgical parameters may be performed according to some embodiments;

[0021] Figure 5A-5F An overview of knee prediction equations according to some embodiments is provided;

[0022] Figure 6 is a flow chart illustrating a process by which system of equations optimization may be performed according to some embodiments;

[0023] Figure 7A-1 , Figure 7A-2 , Figure 7A-3 and Figure 7B An overview of an exemplary user interface for some embodiments is provided;

[0024] Figure 8 An overview of an exemplary surgical patient care system for use with some embodiments is provided;

[0025] Fig. 9 An overview of machine learning algorithms used in some embodiments is provided;

[0026] Fig.10 is a flow chart illustrating the operation of an exemplary surgical patient care system for some embodiments;

[0027] Figure 11A-11B is a flow chart illustrating the operation of an exemplary surgical patient care system for some embodiments;

[0028] Figure 11C-1 , Figure 11C-2 and Figure 11C-3 An overview of an exemplary user interface for implant placement for use in some embodiments is provided;

[0029] Figure 12A-12C providing some outputs that the anatomical modeling software for some embodiments may use to visually depict the results of modeling hip joint motion;

[0030] Fig.12D An exemplary visualization of a hip implant for use with some embodiments is provided;

[0031] Fig.13 Examples of augmented reality visualizations in some embodiments are provided;

[0032] Fig.14 is a system diagram illustrating an augmented reality visualization system for some embodiments;

[0033] Fig.15 is a flow chart illustrating the operation of an augmented reality system for some embodiments;

[0034] Fig.16 is a system diagram illustrating an augmented reality visualization system for some embodiments;

[0035] Fig.17A Examples of augmented reality visualizations in some embodiments are provided;

[0036] Fig. 17B Provides examples of three-dimensional visualization in some embodiments;

[0037] Fig.18A Provides examples of three-dimensional visualization in some embodiments;

[0038] Fig.18B Provides examples of three-dimensional visualization in some embodiments;

[0039] Fig.19 An example of a three-dimensional model for knee component visualization for some embodiments is provided;

[0040] Fig. 20 is a system diagram illustrating an exemplary computing system for use with some embodiments;

[0041] Fig.21 is a system diagram illustrating an exemplary computing system for use with some embodiments;

[0042] Fig. 22 is an anatomical diagram of a hip joint geometry that may be used in some embodiments;

[0043] Figure 23A-23B is a diagram of hip joint geometry within an x-ray image that may be used in some embodiments;

[0044] Fig.24is a flow chart illustrating an exemplary process for extracting anatomical landmarks and determining a surgical plan based on modeling performance for some embodiments;

[0045] Fig.25 is a table depicting exemplary values ​​for hip joint geometry for some embodiments;

[0046] Figure 26A-26D is an exemplary user interface for some embodiments;

[0047] Fig. 27 An exemplary combination of model results for different selected activities for a given geometry is depicted to show the aggregated results;

[0048] Fig.28 is a flow chart illustrating an exemplary method for some embodiments for creating a statistical model database based on modeling performance to assist in determining a surgical plan;

[0049] Fig.29 is a flow chart illustrating an exemplary method for some embodiments for creating a surgical plan based on modeled performance using a statistical model database;

[0050] Fig.30 is a flow chart illustrating an exemplary method for some embodiments for modifying a surgical plan based on modeled performance using a statistical model database;

[0051] Fig.31 is a pair of annotated x-ray images showing exemplary knee joint geometries that may be used in some embodiments;

[0052] Fig.32 is a system diagram of an exemplary embodiment of a surgical system for use with some embodiments;

[0053] Fig.33 is a view of a surgical scene using some of the techniques disclosed herein; and

[0054] Figure 34A-Figure 34B The process of measuring a patient's knee at different degrees of flexion is shown. DETAILED DESCRIPTION

[0055] The present disclosure is not limited to the specific systems, devices, and methods described, as these may vary. The terminology used in the description is for the purpose of describing the particular versions or embodiments only and is not intended to limit the scope.

[0056] As used in this document, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Unless otherwise defined, all scientific and technical terms used herein have the same meaning as commonly understood by those of ordinary skill in the art. Nothing in this disclosure should be construed as an admission that the embodiments described in this disclosure are not entitled to antedate the date of this disclosure due to prior invention. As used in this document, the term "including" means "including but not limited to."

[0057] The disclosed device is particularly well suited for use with surgical navigation systems (e.g. NAVIO is a registered trademark of BLUE BELT TECHNOLOGIES, Inc. of Pittsburgh, Pennsylvania, a subsidiary of SMITH & NEPHEW, Inc. of Memphis, Tennessee.

[0058] definition

[0059] For the purposes of this disclosure, the term "implant" is used to refer to a prosthetic device or structure that is manufactured to replace or enhance a biological structure. For example, in total hip replacement surgery, a prosthetic acetabular cup (implant) is used to replace or enhance a patient's worn or damaged acetabulum. Although the term "implant" is generally considered to mean an artificial structure (in contrast to a transplant), for the purposes of this specification, an implant may include a biological tissue or material that is transplanted to replace or enhance a biological structure.

[0060] For purposes of this disclosure, the term "real time" is used to refer to computations or operations that are performed immediately when an event occurs or an input is received by an operating system. However, the use of the term "real time" is not intended to exclude operations that incur some delay between input and response, as long as the delay is an unintended consequence of the performance characteristics of the machine.

[0061] Although much of the disclosure relates to surgeons or other medical specialists by specific titles or roles, nothing in the disclosure is intended to be limited to a specific title or function. A surgeon or medical specialist may include any doctor, nurse, medical specialist, or technician. Any of these terms or positions may be used interchangeably with users of the system disclosed herein unless otherwise expressly specified. For example, in some embodiments, references to a surgeon may also apply to a technician or nurse.

[0062] CASS Ecosystem Overview

[0063] Figure 1An illustration of an example computer-assisted surgery system (CASS) 100 is provided according to some embodiments. As described in further detail in the following sections, CASS uses computers, robotics, and imaging technology to assist surgeons in performing orthopedic surgical procedures, such as total knee arthroplasty (TKA) or total hip arthroplasty (THA). For example, a surgical navigation system can assist surgeons in locating a patient's anatomy, guiding surgical instruments, and implanting medical devices with high precision. Surgical navigation systems such as CASS 100 often employ various forms of computing technology to perform a wide variety of standard and minimally invasive surgical procedures and techniques. Moreover, these systems allow surgeons to more accurately plan, track, and navigate the position of instruments and implants relative to the patient's body, as well as perform preoperative and intraoperative body imaging.

[0064] The effector platform 105 positions the surgical tools relative to the patient during surgery. The exact components of the effector platform 105 will vary depending on the embodiment being employed. For example, for knee surgery, the effector platform 105 may include an end effector 105B that holds the surgical tools or instruments during their use. The end effector 105B may be a handheld device or instrument (e.g., hand piece or cutting guide or clamp), or alternatively, the end effector 105B may include a device or instrument held or positioned by the robotic arm 105A. Figure 1 One robotic arm 105A is shown in the figure, but in some embodiments, there may be multiple devices. As an example, there may be one robotic arm 105A on each side of the operating table or two devices on one side of the operating table. The robotic arm 105A can be mounted directly to the operating table, located on a floor platform (not shown) next to the operating table, mounted on a floor pole, or mounted on the wall or ceiling of the operating room. The floor platform can be fixed or movable. In a specific embodiment, the robotic arm 105A is mounted on a floor pole located between the patient's legs or feet. In some embodiments, the end effector 105B may include a suture holder or stapler to help close the wound. In addition, in the case of two robotic arms 105A, the surgical computer 150 can drive the robotic arms 105A to work together to suture the wound when closed. Alternatively, the surgical computer 150 can drive one or more robotic arms 105A to suture the wound when closed.

[0065] The actuator platform 105 may include a limb positioner 105C for positioning a patient's limb during surgery. An example of a limb positioner 105C is the SMITH AND NEPHEW SPIDER2 TM Limb positioner 105C may be manually operated by the surgeon, or alternatively, may be operated to change limb position based on instructions received from surgical computer 150 (described below). Figure 1 One limb positioner 105C is shown in the figure, but in some embodiments there may be multiple devices. As an example, there may be one limb positioner 105C on each side of the operating table or two devices on one side of the operating table. The limb positioner 105C can be mounted directly to the operating table, located on a floor platform (not shown) next to the operating table, mounted on a pole, or mounted on a wall or ceiling of the operating room. In some embodiments, the limb positioner 105C can be used in an unconventional manner, such as a retractor or a specific bone holder. As an example, the limb positioner 105C may include an ankle boot, a soft tissue clamp, a bone clamp, or a soft tissue retractor spoon, such as a hooked, curved or angled blade. In some embodiments, the limb positioner 105C may include a suture holder to assist in closing the wound.

[0066] The actuator platform 105 may include a tool such as a screwdriver, a light or laser to indicate an axis or plane, a level, a pin driver, a pin puller, a flatness checker, an indicator, a finger, or some combination thereof.

[0067] Resection device 110 ( Figure 1 The resection device 110 is a device that is used to perform bone or tissue resection using, for example, mechanical, ultrasonic, or laser techniques (not shown). Examples of the resection device 110 include drilling devices, deburring devices, oscillating sawing devices, vibrating impact devices, reamers, ultrasonic bone cutting devices, radiofrequency ablation devices, reciprocating motion devices (such as files or broaches), and laser ablation systems. In some embodiments, the resection device 110 is held and operated by the surgeon during surgery. In other embodiments, the actuator platform 105 can be used to hold the resection device 110 during use.

[0068] The actuator platform 105 may also include a cutting guide or fixture 105D for guiding a saw or drill used to remove tissue during surgery. Such a cutting guide 105D may be integrally formed as part of the actuator platform 105 or the robotic arm 105A, or the cutting guide may be an independent structure that may be cooperatively and / or removably attached to the actuator platform 105 or the robotic arm 105A. The actuator platform 105 or the robotic arm 105A may be controlled by CASS100 to position the cutting guide or fixture 105D near the patient's anatomical structure according to a surgical plan developed preoperatively or intraoperatively, so that the cutting guide or fixture will produce precise bone cutting according to the surgical plan.

[0069] Tracking system 115 uses one or more sensors to collect real-time position data of the anatomical structure and surgical instruments of the patient. For example, for a TKA procedure, tracking system 115 can provide the position and orientation of end effector 105B during the procedure. In addition to position data, data from tracking system 115 can also be used to infer the speed / acceleration of the anatomical structure / instrument, which can be used for tool control. In some embodiments, tracking system 115 can use a tracker array attached to end effector 105B to determine the position and orientation of end effector 105B. The position of end effector 105B can be inferred based on the position and orientation of tracking system 115 and the known relationship in three-dimensional space between tracking system 115 and end effector 105B. Various types of tracking systems can be used in various embodiments of the present invention, including but not limited to infrared (IR) tracking systems, electromagnetic (EM) tracking systems, video or image-based tracking systems, and ultrasound registration and tracking systems. Using the data provided by tracking system 115, surgical computer 150 can detect objects and prevent collisions. For example, surgical computer 150 can prevent robot arm 105A from colliding with soft tissue.

[0070] Any suitable tracking system can be used to track surgical objects and patient anatomy in the operating room. For example, a combination of infrared and visible light cameras can be used in an array. Various illumination sources (e.g., infrared LED light sources) can illuminate the scene so that three-dimensional imaging can be performed. In some embodiments, this can include stereoscopic, three-view, four-view, etc. imaging. In addition to the camera array fixed to the cart in some embodiments, additional cameras can be placed throughout the operating room. For example, a handheld tool or headgear worn by the operator / surgeon can include an imaging function that transmits images back to a central processor to correlate those images with images acquired by the camera array. This can provide more robust images for environments modeled using multiple perspectives. In addition, some imaging devices can have a suitable resolution on the scene or have a suitable perspective to pick up information stored in a quick response (QR) code or barcode. This helps to identify specific objects that are not manually registered with the system. In some embodiments, the camera can be mounted on the robot arm 105A.

[0071] In some embodiments, the surgeon can manually register specific objects with the system before or during surgery. For example, by interacting with the user interface, the surgeon can identify the starting position of a tool or bone structure. By tracking fiducial markers associated with the tool or bone structure, or by using other conventional image tracking methods, the processor can track the tool or bone as it moves through the environment in the three-dimensional model.

[0072] In some embodiments, certain markers, such as fiducial markers that identify individuals, important tools, or bones in the operating room, may include passive or active identification that can be picked up by a camera or camera array associated with a tracking system. For example, an infrared LED may flash a pattern that conveys a unique identification to the source of the pattern, thereby providing a dynamic identification marker. Similarly, one-dimensional or two-dimensional optical codes (barcodes, QR codes, etc.) may be fixed to objects in the operating room to provide passive identification that can occur based on image analysis. If these codes are placed asymmetrically on the object, they can also be used to determine the orientation of the object by comparing the position of the identification with the range of the object in the image. For example, a QR code may be placed in the corner of a tool tray, allowing the orientation and identification of the tray to be tracked. Other tracking methods will be described throughout the text. For example, in some embodiments, surgeons and other personnel may wear augmented reality headsets to provide additional camera angles and tracking capabilities.

[0073] In addition to optical tracking, certain features of an object can be tracked by aligning the physical properties of the object and associating them with an object that can be tracked (e.g., a fiducial marker fixed to a tool or bone). For example, a surgeon can perform a manual alignment process whereby the tracked tool and the tracked bone can be manipulated relative to each other. By striking the tip of the tool against the surface of the bone, a three-dimensional surface can be plotted for the bone, the three-dimensional surface being associated with the position and orientation of a reference system relative to the fiducial marker. By optically tracking the position and orientation (pose) of the fiducial marker associated with the bone, a model of the surface can be tracked in the environment by extrapolation.

[0074] The registration process of registering CASS100 to the relevant anatomical structure of the patient may also involve the use of anatomical landmarks, such as landmarks on bones or cartilage. For example, CASS100 may include a 3D model of the relevant bone or joint, and the surgeon may use a probe connected to CASS to collect data about the position of the bone landmarks on the patient's actual bone during surgery. Bone landmarks may include, for example, the medial and lateral malleolus, the ends of the proximal femur and the distal tibia, and the center of the hip joint. CASS100 can compare and register the position data of the bone landmarks collected by the surgeon with the position data of the same landmarks in the 3D model. Alternatively, CASS100 can construct a 3D model of a bone or joint without preoperative image data by using the position data of the bone landmarks and bone surfaces collected by the surgeon using the CASS probe or other means. The registration process may also include determining the various axes of the joint. For example, for TKA, the surgeon can use CASS100 to determine the anatomical and mechanical axes of the femur and tibia. The surgeon and CASS 100 can identify the center of the hip joint by moving the patient's leg in a spiral direction (ie, circumduction) so that CASS can determine the location of the hip joint center.

[0075] Organizational Navigation System 120 ( Figure 1 (not shown) provides the surgeon with intraoperative real-time visualization of the patient's bone, cartilage, muscle, nerve and / or vascular tissue surrounding the surgical area. Examples of systems that can be used for tissue navigation include fluorescence imaging systems and ultrasound systems.

[0076] The display 125 provides a graphical user interface (GUI) that displays images collected by the tissue navigation system 120 and other information related to the surgery. For example, in one embodiment, the display 125 overlays image information collected from various modalities (e.g., CT, MRI, X-ray, fluorescence, ultrasound, etc.) collected preoperatively or intraoperatively to provide the surgeon with various views of the patient's anatomical structure and real-time status. The display 125 may include, for example, one or more computer monitors. As an alternative or in addition to the display 125, one or more of the surgical staff may wear an augmented reality (AR) head mounted device (HMD). For example, in Figure 1 In the present invention, surgeon 111 wears AR HMD 155, which can, for example, overlay preoperative image data on the patient or provide surgical planning suggestions. Various exemplary uses of AR HMD 155 in surgical procedures are described in detail in the following sections.

[0077] The surgical computer 150 provides control instructions to the various components of CASS100, collects data from those components, and provides general processing for various data required during surgery. In some embodiments, the surgical computer 150 is a general-purpose computer. In other embodiments, the surgical computer 150 can be a parallel computing platform that uses multiple central processing units (CPUs) or graphics processing units (GPUs) to perform processing. In some embodiments, the surgical computer 150 is connected to a remote server via one or more computer networks (e.g., the Internet). The remote server can be used, for example, for the storage of data or the execution of computationally intensive processing tasks.

[0078] The surgical computer 150 may be connected to the other components of the CASS 100 using various techniques known in the art. Moreover, the computer may be connected to the surgical computer 150 using a variety of techniques. For example, the end effector 105B may be connected to the surgical computer 150 via a wired (i.e., serial) connection. The tracking system 115, the tissue navigation system 120, and the display 125 may similarly be connected to the surgical computer 150 using a wired connection. Alternatively, the tracking system 115, the tissue navigation system 120, and the display 125 may be connected to the surgical computer 150 using wireless technology, such as, but not limited to, Wi-Fi, Bluetooth, near field communication (NFC), or ZigBee.

[0079] Power impact and acetabular reamer device

[0080] The above about Figure 1 Part of the flexibility of the described CASS design is that additional or alternative devices can be added to CASS100 as needed to support specific surgical procedures. For example, in the case of hip surgery, CASS100 can include a powered impact device. The impact device is designed to repeatedly apply impact forces that surgeons can use to perform activities such as implant alignment. For example, in total hip arthroplasty (THA), surgeons typically use an impact device to insert a prosthetic acetabular cup into the acetabulum of the implanted host. Although the impact device can be manual in nature (e.g., operated by the surgeon striking the impactor with a hammer), the powered impact device is typically easier and faster to use in a surgical environment. The powered impact device can be powered, for example, using a battery attached to the device. Various attachments can be connected to the powered impact device to allow the impact force to be directed in various ways as needed during surgery. Also in the case of hip surgery, CASS100 can include a powered, robot-controlled end effector to ream the acetabulum to accommodate the acetabular cup implant.

[0081] In robot-assisted THA, the patient's anatomical structure can be registered to CASS 100 using CT or other image data, identification of anatomical landmarks, a tracker array attached to the patient's bones, and one or more cameras. The tracker array can be mounted on the iliac crest using a clamp and / or bone screws, and such a tracker array can be installed externally through the skin or internally (posterolateral or anterolateral) through an incision made to perform THA. For THA, CASS100 can use one or more femoral cortical screws inserted into the proximal femur as checkpoints to help the registration process. CASS100 can also use one or more checkpoint screws inserted into the pelvis as additional checkpoints to help the registration process. The femoral tracker array can be fixed or mounted in the femoral cortical screw. CASS100 can adopt the following steps, in which the surgeon uses a probe accurately placed on the key areas of the proximal femur and pelvis identified by the surgeon on the display 125 for verification. The tracker can be located on the robot arm 105A or the end effector 105B to register the arm and / or end effector to CASS100. The verification step may also utilize proximal and distal femoral checkpoints. CASS 100 may utilize color cues or other cues to inform the surgeon that the registration process of the bone and robotic arm 105A or end effector 105B has been verified to a certain degree of accuracy (e.g., within 1 mm).

[0082] For THA, CASS100 can include a broach tracking option using a femoral array to allow the surgeon to intraoperatively capture the broach position and orientation and calculate the patient's hip length and offset values. Based on the information provided about the patient's hip joint and the planned implant position and orientation after broach tracking is completed, the surgeon can modify or adjust the surgical plan.

[0083] For robot-assisted THA, CASS100 may include one or more power reamers connected or attached to a robot arm 105A or an end effector 105B, which prepare the pelvic bone to receive the acetabular implant according to the surgical plan. The robot arm 105A and / or the end effector 105B may notify the surgeon and / or control the power of the reamer to ensure that the acetabulum is removed (reamed) according to the surgical plan. For example, if the surgeon attempts to remove the bone outside the boundary of the bone to be removed according to the surgical plan, CASS100 may cut off the power supply of the reamer or instruct the surgeon to cut off the power supply of the reamer. CASS100 may provide the surgeon with a choice to turn off or disengage the robotic control of the reamer. Compared to surgical plans using different colors, the display 125 may show the progress of the bone being removed (reamed). The surgeon may view the display of the bone being removed (reamed) to guide the reamer to complete the reaming according to the surgical plan. CASS100 may provide the surgeon with visual or auditory prompts to warn the surgeon that a resection that does not conform to the surgical plan is being performed.

[0084] After reaming, CASS100 can use a manual or powered impactor attached to or connected to the robotic arm 105A or the end effector 105B to impact the trial implant and the final implant into the acetabulum. The robotic arm 105A and / or the end effector 105B can be used to guide the impactor to impact the trial implant and the final implant into the acetabulum according to the surgical plan. CASS100 can display the position and orientation of the trial implant and the final implant relative to the bone to inform the surgeon how to compare the orientation and position of the trial implant and the final implant with the surgical plan, and the display 125 can display the position and orientation of the implant while the surgeon manipulates the leg and hip. If the surgeon is not satisfied with the initial implant position and orientation, CASS100 can provide the surgeon with the option of replanning and redoing the reaming and implant impact by preparing a new surgical plan.

[0085] Preoperatively, CASS100 can develop a proposed surgical plan based on a three-dimensional model of the hip joint and other patient-specific information (such as the mechanical and anatomical axes of the leg bones, the epicondylar axis, the femoral neck axis, the dimensions of the femur and hip (e.g., length), the midline axis of the hip joint, the ASIS axis of the hip joint, and the location of anatomical landmarks such as the lesser trochanter landmark, the distal landmark, and the center of rotation of the hip joint). The surgical plan developed by CASS can provide recommended optimal implant size and implant location and orientation based on the three-dimensional model of the hip joint and other patient-specific information. The surgical plan developed by CASS can include recommended details regarding offset values, inclination and anteversion values, center of rotation, cup size, midline value, upper and lower fit, and femoral stem size and length.

[0086] For THA, the surgical plan developed by CASS can be viewed preoperatively and intraoperatively, and the surgeon can modify the surgical plan developed by CASS preoperatively or intraoperatively. The surgical plan developed by CASS can display the planned hip resection and superimpose the planned implant on the hip joint based on the planned resection. CASS100 can provide the surgeon with a choice of different surgical processes, which will be displayed to the surgeon based on the surgeon's preferences. For example, the surgeon can choose from different workflows based on the number and type of anatomical landmarks examined and acquired and / or the location and number of tracker arrays used in the alignment process.

[0087] According to some embodiments, the power impact device used with CASS100 can be operated with a variety of different settings. In some embodiments, the surgeon adjusts the settings by a manual switch or other physical mechanism on the power impact device. In other embodiments, a digital interface can be used, which allows, for example, setting input via a touch screen on the power impact device. Such a digital interface can allow the available settings to change based on, for example, the type of attachment connected to the power attachment device. In some embodiments, the settings can be changed by communicating with a robot or other computer system within CASS100, rather than adjusting the settings on the power impact device itself. Such a connection can be established using, for example, a Bluetooth or Wi-Fi networking module on the power impact device. In another embodiment, the impact device and the end piece can include features that allow the impact device to know what end piece (cup impactor, broach handle, etc.) is attached without the surgeon having to take any action, and adjust the settings accordingly. This can be achieved, for example, by a QR code, a barcode, an RFID tag, or other methods.

[0088] Examples of settings that can be used include cup impact settings (e.g., unidirectional, specified frequency range, specified force and / or energy range); broach impact settings (e.g., bidirectional / oscillating within a specified frequency range, specified force and / or energy range); femoral head impact settings (e.g., unidirectional / single blow at a specified force or energy); and dry impact settings (e.g., unidirectional at a specified force or energy at a specified frequency). In addition, in some embodiments, the power impact device includes settings related to acetabular liner impact (e.g., unidirectional / single blow at a specified force or energy). There may be multiple settings for each type of liner (e.g., polymer, ceramic, oxinium, or other material). In addition, the power impact device can provide settings for different bone qualities based on preoperative testing / imaging / knowledge and / or the surgeon's intraoperative assessment. In some embodiments, the power impact device can have dual functions. For example, the power impact device can not only provide reciprocating motion to provide impact force, but also provide reciprocating motion for the broach or rasp.

[0089] In some embodiments, the powered impact device includes a feedback sensor that collects data during use of the device and sends the data to a computing device, such as a controller within the device or a surgical computer 150. The computing device can then record the data for later analysis and use. Examples of data that can be collected include, but are not limited to, sound waves, predetermined resonant frequencies of each device, reaction forces or rebound energies from the patient's bones, the position of the device relative to the imaging (e.g., fluorescence, CT, ultrasound, MRI, etc.) of the registered bone anatomy, and / or external strain gauges on the bone.

[0090] Once the data is collected, the computing device can execute one or more algorithms in real time or near real time to assist the surgeon in performing the surgical procedure. For example, in some embodiments, the computing device uses the collected data to derive information such as the correct final broach size (femoral); when the stem is fully seated (femoral side); or when the cup is in place for the THA (depth and / or orientation). Once this information is known, it can be displayed for the surgeon to see, or it can be used to activate a haptic or other feedback mechanism to guide the surgical procedure.

[0091] In addition, the data derived from the aforementioned algorithm can be used to drive the operation of the device. For example, during the insertion of a prosthetic acetabular cup with a powered impact device, the device can automatically extend the impact head (e.g., end effector), move the implant to the appropriate position, or shut off the power of the device once the implant is fully in place. In one embodiment, the derived information can be used to automatically adjust the setting of bone quality, where the powered impact device should use less power to reduce femoral / acetabulum / pelvic fractures or damage to surrounding tissues.

[0092] Robotic Arm

[0093] In some embodiments, CASS100 includes a robotic arm 105A that serves as an interface for stabilizing and holding various instruments used during the surgical procedure. For example, in the case of hip surgery, these instruments may include, but are not limited to, retractors, sagittal or reciprocating saws, reamer handles, cup impactors, broach handles, and stem inserters. The robotic arm 105A may have multiple degrees of freedom (similar to a Spider device) and the ability to lock into place (e.g., by pressing a button, voice activation, the surgeon removing his hand from the robotic arm, or other methods).

[0094] In some embodiments, movement of the robotic arm 105A can be accomplished using a control panel built into the robotic arm system. For example, a display screen can include one or more input sources, such as physical buttons or a user interface with one or more icons that direct the movement of the robotic arm 105A. A surgeon or other healthcare professional can engage with one or more input sources to position the robotic arm 105A during the performance of a surgical procedure.

[0095] A tool or end effector 105B may be attached or integrated into the robotic arm 105A, which may include, but is not limited to, a deburring device, a scalpel, a cutting device, a retractor, a joint tensioning device, etc. In embodiments using an end effector 105B, the end effector 105B may be positioned at the end of the robotic arm 105A such that any motor control operations are performed within the robotic arm system. In embodiments using a tool, the tool may be fixed at the distal end of the robotic arm 105A, but the motor control operations may be located within the tool itself.

[0096] The robotic arm 105A may be motorized internally to stabilize the robotic arm, thereby preventing it from falling and striking the patient, operating table, surgical staff, etc., and allowing the surgeon to move the robotic arm without having to fully support its weight. While the surgeon is moving the robotic arm 105A, the robotic arm may provide some resistance to prevent the robotic arm from moving too fast or activating too many degrees of freedom at once. The position and locking state of the robotic arm 105A may be tracked, for example, by a controller or surgical computer 150.

[0097] In some embodiments, the robotic arm 105A can be moved by hand (e.g., by a surgeon) or with an internal motor to its ideal position and orientation for the task being performed. In some embodiments, the robotic arm 105A may be able to operate in a "free" mode, allowing the surgeon to position the arm in a desired position without restriction. In free mode, the position and orientation of the robotic arm 105A may still be tracked as described above. In one embodiment, certain degrees of freedom may be selectively released upon input from a user (e.g., a surgeon) during a specified portion of a surgical plan tracked by the surgical computer 150. Designs in which the robotic arm 105A is powered internally by hydraulics or a motor or provides resistance to external manual movement by similar means may be described as powered robotic arms, while arms that are manually manipulated without power feedback but can be manually or automatically locked in place may be described as passive robotic arms.

[0098] The robotic arm 105A or the end effector 105B may include a trigger or other device to control the power of the saw or drill. The surgeon's engagement of the trigger or other device can cause the robotic arm 105A or the end effector 105B to transition from the motorized alignment mode to the mode in which the saw or drill is engaged and powered. In addition, CASS100 may include a foot pedal (not shown) that causes the system to perform certain functions when activated. For example, the surgeon can activate the foot pedal to instruct CASS100 to place the robotic arm 105A or the end effector 105B in an automatic mode that places the robotic arm or the end effector in an appropriate position relative to the patient's anatomical structure in order to perform the necessary resection. CASS100 can also place the robotic arm 105A or the end effector 105B in a collaborative mode that allows the surgeon to manually manipulate the robotic arm or the end effector and position it in a specific position. The collaborative mode can be configured to allow the surgeon to move the robotic arm 105A or the end effector 105B medially or laterally while limiting movement in other directions. As discussed, the robotic arm 105A or end effector 105B may include a cutting device (saw, drill, and sharpener) or a cutting guide or fixture 105D to guide the cutting device. In other embodiments, the movement of the robotic arm 105A or the robotically controlled end effector 105B may be completely controlled by CASS100 without any assistance or input from the surgeon or other medical professional, or with only minimal assistance or input. In still other embodiments, the surgeon or other medical professional may remotely control the movement of the robotic arm 105A or the robotically controlled end effector 105B using a control mechanism separate from the robotic arm or robotically controlled end effector device, such as using a joystick or an interactive monitor or display control device.

[0099] The following example describes the use of a robotic device in the context of hip surgery; however, it should be understood that the robotic arm may have other applications in surgical procedures involving the knee, shoulder, etc. An example of the use of a robotic arm in the context of forming an anterior cruciate ligament (ACL) graft tunnel is described in PCT / US2019 / 048502, filed on August 28, 2019, entitled "Robotic Assisted Ligament Graft Placement and Tensioning," the entire contents of which are incorporated herein by reference.

[0100] The robotic arm 105A can be used to hold a retractor. For example, in one embodiment, the surgeon can move the robotic arm 105A to a desired position. At this point, the robotic arm 105A can lock into place. In some embodiments, the robotic arm 105A is provided with data about the patient's position so that if the patient moves, the robotic arm can adjust the retractor position accordingly. In some embodiments, multiple robotic arms can be used, thereby allowing multiple retractors to be held or more than one action to be performed simultaneously (e.g., retractor holding and reaming).

[0101] The robotic arm 105A can also be used to help stabilize the surgeon's hand when making a femoral neck incision. In this application, the control of the robotic arm 105A can impose certain restrictions to prevent soft tissue damage from occurring. For example, in one embodiment, the surgical computer 150 tracks the position of the robotic arm 105A as it operates. If the tracked position is close to an area where tissue damage is predicted, a command can be sent to the robotic arm 105A to stop it. Alternatively, in the case where the robotic arm 105A is automatically controlled by the surgical computer 150, the surgical computer can ensure that no instructions are provided to the robotic arm that cause it to enter an area where soft tissue damage may occur. The surgical computer 150 can impose certain restrictions on the surgeon to prevent the surgeon from reaming too deep into the medial wall of the acetabulum or reaming at an incorrect angle or orientation.

[0102] In some embodiments, the robotic arm 105A can be used to hold the cup impactor at a desired angle or orientation during cup impaction. When the final position has been reached, the robotic arm 105A can prevent any further positioning to prevent damage to the pelvis.

[0103] The surgeon can use the robotic arm 105A to position the broach handle in a desired position and allow the surgeon to impact the broach into the femoral canal in a desired orientation. In some embodiments, once the surgical computer 150 receives feedback that the broach is fully in place, the robotic arm 105A can limit the handle to prevent further advancement of the broach.

[0104] The robotic arm 105A can also be used for resurfacing applications. For example, the robotic arm 105A can stabilize the surgeon's hand while using traditional instruments and provide certain constraints or restrictions to allow for proper placement of implant components (e.g., guidewire placement, chamfer cutters, sleeve cutters, flat cutters, etc.). In the case of using only a bone drill, the robotic arm 105A can stabilize the surgeon's handpiece and can impose restrictions on the handpiece to prevent the surgeon from violating the surgical plan and removing undesirable bone.

[0105] The robotic arm 105A may be a passive arm. As an example, the robotic arm 105A may be a CIRQ robotic arm available from Brainlab AG. CIRQ is a registered trademark of Brainlab AG, Olof-Palme-Str. 981829, Munich, Germany. In a particular embodiment, the robotic arm 105A is an intelligent gripping arm, as disclosed in U.S. Patent Application No. 15 / 525,585 to Krinninger et al., U.S. Patent Application No. 15 / 561,042 to Nowatschin et al., U.S. Patent No. 15 / 561,048 to Nowatschin et al., and U.S. Patent No. 10,342,636 to Nowatschin et al., each of which is incorporated herein by reference in its entirety.

[0106] Generation and collection of surgical procedure data

[0107] The various services provided by medical professionals to treat clinical conditions are collectively referred to as the "care period". For a particular surgical procedure, the care period may include three phases: preoperative, intraoperative, and postoperative. During each phase, data is collected or generated that can be used to analyze the care period in order to understand various aspects of the procedure and identify patterns that can be used, for example, in training models to make decisions with minimal human intervention. The data collected during the care period can be stored as a complete data set on the surgical computer 150 or surgical data server 180 ( Figure 2C Thus, for each care period, there is one data set that includes all data collected collectively about the patient preoperatively, all data collected or stored by CASS100 intraoperatively, and any postoperative data provided by the patient or by the medical professional monitoring the patient.

[0108] As further explained in detail, the data collected during the nursing period can be used to enhance the execution of the surgical procedure or provide an overall understanding of the surgical procedure and patient results. For example, in some embodiments, the data collected during the nursing period can be used to generate a surgical plan. In one embodiment, when collecting data during surgery, the advanced preoperative plan is improved during surgery. In this way, when new data is collected by the components of CASS100, the surgical plan can be regarded as a real-time or near-real-time dynamic change. In other embodiments, preoperative images or other input data can be used to preoperatively formulate a robust plan that is simply executed during surgery. In this case, the data collected by CASS100 during surgery can be used to make suggestions to ensure that the surgeon is within the preoperative surgical plan. For example, if the surgeon is not sure how to achieve certain prescribed cutting or implant alignment, the surgical computer 150 can be queried to obtain suggestions. In other embodiments, preoperative and intraoperative planning schemes can be combined so that the perfect preoperative plan can be dynamically modified as needed or desired during the surgical procedure. In some embodiments, a biomechanically based model of the patient's anatomy contributes simulation data to be considered by CASS100 in formulating preoperative, intraoperative, and postoperative / rehabilitation procedures to optimize the patient's implant performance outcomes.

[0109] In addition to changing the surgical procedure itself, the data collected during the nursing session can also be used as input for other surgical assistance procedures. For example, in some embodiments, the nursing session data can be used to design an implant. Example data-driven techniques for designing, sizing, and fitting implants are described in U.S. patent application Ser. No. 13 / 814,531, filed on Aug. 15, 2011, entitled “Systems and Methods for Optimizing Parameters for Orthopaedic Procedures”; U.S. patent application Ser. No. 14 / 232,958, filed on Jul. 20, 2012, entitled “Systems and Methods for Optimizing Fit of an Implant to Anatomy”; and U.S. patent application Ser. No. 12 / 234,444, filed on Sep. 19, 2008, entitled “Operatively Tuning Implants for Increased Performance,” the entire contents of each of the above patents are hereby incorporated by reference into this patent application.

[0110] In addition, the data may be used for educational, training or research purposes. For example, using Figure 2C Using the web-based solution described in , other physicians or students can remotely view the surgery in an interface that allows them to selectively view data collected from various components of the CASS 100. After the surgical procedure, a similar interface can be used to "replay" the surgery for training or other educational purposes, or to identify the source of any problems or complications during the procedure.

[0111] The data acquired during the preoperative phase typically includes all information collected or generated before the operation. Thus, for example, information about the patient can be obtained from a patient entry form or an electronic medical record (EMR). Examples of patient information that can be collected include, but are not limited to, patient demographics, diagnosis, medical history, medical records, vital signs, medical history information, allergies, and laboratory test results. Preoperative data can also include images related to the anatomical region of interest. These images can be acquired, for example, using magnetic resonance imaging (MRI), computed tomography (CT), X-rays, ultrasound, or any other means known in the art. Preoperative data can also include quality of life data acquired from the patient. For example, in one embodiment, preoperative patients use a mobile application ("app") to answer a questionnaire about their current quality of life. In some embodiments, the preoperative data used by CASS 100 includes demographics, anthropometrics, culture, or other specific features about the patient, which can be consistent with activity levels and specific patient activities to customize surgical plans for patients.

[0112] Figure 2A and 2B Examples of data that can be acquired during the intraoperative phase of a care episode are provided. These examples are based on the above reference Figure 1 Various components of CASS 100 are described; however, it should be understood that other types of data may be used based on the type of equipment used during the procedure and its use.

[0113] Figure 2A 1 shows some examples of control instructions provided by the surgical computer 150 to other components of the CASS 100 according to some embodiments. It should be noted that Figure 2A The example assumes that the components of the actuator platform 105 are directly controlled by the surgical computer 150. In embodiments where the components are manually controlled by the surgeon 111, the components may be manually controlled by the surgeon 111 on the display 125 or AR HMD 155 (e.g., Figure 1 Instructions are provided on the PCB to instruct the surgeon 111 how to move the components.

[0114] The various components included in the effector platform 105 are controlled by the surgical computer 150, which provides position instructions that indicate where the components are to move within the coordinate system. In some embodiments, the surgical computer 150 provides instructions to the effector platform 105 that define how to react when components of the effector platform 105 deviate from the surgical plan. These commands are Figure 2A 105B. For example, the end effector 105B may provide a force to resist motion outside of the planned resection area. Other commands that may be used by the effector platform 105 include vibration and audio cues.

[0115] In some embodiments, the end effector 105B of the robot arm 105A is connected to the cutting guide 105D (eg, Figure 1 ) are operably connected. In response to the anatomical model of the surgical scene, the robotic arm 105A can move the end effector 105B and the cutting guide 105D to the appropriate position to match the position of the femoral or tibia cut to be performed according to the surgical plan. This can reduce the possibility of error, thereby allowing the visual system and the processor using the visual system to implement the surgical plan to place the cutting guide 105D in a precise position and orientation relative to the tibia or femur to align the cutting groove of the cutting guide with the cut to be performed according to the surgical plan. The surgeon can then use any suitable tool, such as a vibrating or rotating saw or drill, to perform the cut (or drill) with perfect placement and orientation because the tool is mechanically limited by the features of the cutting guide 105D. In some embodiments, the cutting guide 105D may include one or more nail holes, which the surgeon uses to drill and tighten or nail the cutting guide to the appropriate position before using the cutting guide to perform the resection of the patient's tissue. This can release the robotic arm 105A or ensure that the cutting guide 105D is completely fixed and does not move relative to the bone to be resected. For example, the procedure can be used to make a first distal incision of the femur during a total knee replacement. In some embodiments, where the joint replacement is a hip replacement, the cutting guide 105D can be secured to the femoral head or acetabulum for the corresponding hip replacement resection. It should be understood that any joint replacement utilizing a precision incision can use the robotic arm 105A and / or cutting guide 105D in this manner.

[0116] The ablation device 110 is provided with a variety of commands to perform bone or tissue operations. As with the actuator platform 105, position information can be provided to the ablation device 110 to specify where it should be positioned when performing ablation. Other commands provided to the ablation device 110 may depend on the type of ablation device. For example, for mechanical or ultrasonic ablation tools, the commands may specify the speed and frequency of the tool. For radiofrequency ablation (RFA) and other laser ablation tools, these commands may specify the intensity and pulse duration.

[0117] Some components of CASS 100 need not be directly controlled by the surgical computer 150; rather, the surgical computer 150 need only activate the component, which then executes software locally to specify the manner in which data is collected and provided to the surgical computer 150. Figure 2A In the example of , there are two components that operate in this manner: tracking system 115 and tissue navigation system 120.

[0118] The surgical computer 150 provides the display 125 with any visualization required by the surgeon 111 during the operation. For the monitor, the surgical computer 150 can use techniques known in the art to provide instructions for displaying images, GUIs, etc. The display 125 can include various aspects of the workflow of the surgical plan. For example, during the registration process, the display 125 can display the 3D bone model constructed before the operation, and show the position of the probe when the surgeon uses the probe to collect the position of the anatomical landmarks on the patient. The display 125 can include information about the surgical target area. For example, in conjunction with TKA, the display 125 can show the mechanical and anatomical axes of the femur and tibia. The display 125 can show the varus and valgus angles of the knee joint based on the surgical plan, and CASS100 can show how such angles will be affected if the expected corrections to the surgical plan are made. Therefore, the display 125 is an interactive interface that can dynamically update and display how changes to the surgical plan will affect the final position and orientation of the implant installed on the bone.

[0119] As the workflow proceeds to preparation for bone cutting or resection, the display 125 can show the planned or recommended bone cutting before performing any cutting. The surgeon 111 can manipulate the image display to provide different anatomical perspectives of the target area, and can have the option of changing or revising the planned bone cutting based on the patient's intraoperative assessment. The display 125 can show how the selected implant will be installed on the bone if the planned bone cutting is performed. If the surgeon 111 chooses to change the previously planned bone cutting, the display 125 can show how the revised bone cutting will change the position and orientation of the implant when installed on the bone.

[0120] The display 125 can provide the surgeon 111 with various data and information about the patient, the planned surgical procedure, and the implant. Various patient-specific information can be displayed, including real-time data about the patient's health, such as heart rate, blood pressure, etc. The display 125 can also include information about the anatomical structure of the surgical target area, including the location of the landmarks, the current state of the anatomical structure (e.g., whether any resection has been performed, the depth and angle of the planned and executed bone cuts), and the future state of the anatomical structure as the surgical plan progresses. The display 125 can also provide or show additional information about the surgical target area. For TKA, the display 125 can provide information about the gap between the femur and the tibia (e.g., gap balance) and how such a gap will be changed if the planned surgical plan is performed. For TKA, the display 125 can provide additional relevant information about the knee joint, such as data about the tension of the joint (e.g., ligament laxity) and information about the rotation and alignment of the joint. The display 125 can show how the positioning and position of the planned implant will affect the patient when the knee joint is flexed. Display 125 can show how the use of different implants or the use of the same implant of different sizes will affect the surgical plan, and preview how such implants will be positioned on the bone. CASS100 can provide such information for each planned osteotomy in TKA or THA. In TKA, CASS100 can provide robot control for one or more planned osteotomies. For example, CASS100 can only provide robot control for the initial distal femur cutting, and surgeon 111 can manually perform other cuttings (front, back and chamfer cutting) using conventional means (e.g., 4-in-1 cutting guide or fixture 105D).

[0121] The display 125 may employ different colors to inform the surgeon of the status of the surgical plan. For example, unresected bone may be displayed in a first color, resected bone may be displayed in a second color, and planned resection may be displayed in a third color. Implants may be superimposed on the bones in the display 125, and the implant colors may change or correspond to different types or sizes of implants.

[0122] The information and options shown on the display 125 can vary according to the type of surgical procedure performed. In addition, the surgeon 111 can request or select a specific surgical procedure display that matches or is consistent with his or her surgical plan preferences. For example, for the surgeon 111 who usually performs tibial cutting before femoral cutting in TKA, the display 125 and the associated workflow can be adapted to take into account this preference. The surgeon 111 can also pre-select to include or delete certain steps from the standard surgical procedure display. For example, if the surgeon 111 uses resection measurement to finalize the implantation plan, but does not analyze the ligament gap balance when finalizing the implantation plan, the surgical procedure display can be organized into modules, and the surgeon can select the modules to be displayed and the order of providing the modules according to the surgeon's preferences or the situation of a specific operation. For example, a module involving ligament and gap balance can include ligament / gap balance before and after resection, and the surgeon 111 can select which modules to include in its default surgical plan workflow according to whether such ligament and gap balance is performed before or after (or before and after) bone resection.

[0123] For more specialized display devices, such as an AR HMD, the surgical computer 150 can provide images, text, etc. using data formats supported by the device. For example, if the display 125 is a Microsoft HoloLens TM or Magic Leap One TM If the surgeon 111 is using a holographic device, the surgical computer 150 can use the HoloLens application programming interface (API) to send commands that specify the location and content of the hologram displayed in the surgeon's 111 field of view.

[0124] In some embodiments, one or more surgical planning models may be incorporated into CASS 100 and used in the development of the surgical plan provided to the surgeon 111. The term "surgical planning model" refers to software that simulates the biomechanical properties of anatomical structures in various situations to determine the best way to perform cuts and other surgical activities. For example, for a knee replacement surgery, a surgical planning model may measure parameters of functional activities, such as deep knee flexion, gait, etc., and select cutting locations on the knee to optimize implant placement. An example of a surgical planning model is the LIFEMOD® from SMITH AND NEPHEW, Inc. TM In some embodiments, surgical computer 150 includes a computing architecture (e.g., a GPU-based parallel processing environment) that allows the surgical planning model to be fully executed during surgery. In other embodiments, surgical computer 150 can be connected via a network to a remote computer that allows such execution, such as surgical data server 180 (see Figure 2C). As an alternative to full execution of the surgical planning model, in some embodiments, a set of transfer functions is derived that simplifies the mathematical operations obtained by the model into one or more prediction equations. Then, instead of performing a full simulation during surgery, the prediction equations are used. More details on the use of transfer functions are described in PCT / US2019 / 046995, entitled "Patient Specific Surgical Method and System", filed on August 19, 2019, the entire contents of which are incorporated herein by reference.

[0125] Figure 2B Examples of some types of data that can be provided to the surgical computer 150 from the various components of CASS100 are shown. In some embodiments, the components can stream data to the surgical computer 150 in real time or near real time during surgery. In other embodiments, the components can queue the data and send it to the surgical computer 150 at a set interval (e.g., every second). The data can be transmitted using any format known in the art. Therefore, in some embodiments, all components transmit data to the surgical computer 150 in a common format. In other embodiments, each component can use a different data format, and the surgical computer 150 is configured with one or more software applications capable of converting the data.

[0126] In general, the surgical computer 150 can be used as a central point for collecting CASS data. The exact content of the data will depend on the source. For example, each component of the actuator platform 105 provides a measured position to the surgical computer 150. Therefore, by comparing the measured position with the position originally specified by the surgical computer 150, the surgical computer can identify deviations that occur during surgery.

[0127] The resection device 110 can send various types of data to the surgical computer 150 depending on the type of device used. Exemplary data types that can be sent include measured torque, audio signatures, and measured displacement values. Similarly, the tracking technology 115 can provide different types of data depending on the tracking method used. Exemplary tracking data types include position values ​​of tracked items (e.g., anatomical structures, tools, etc.), ultrasound images, and surface or landmark collection points or axes. When the system is operating, the tissue navigation system 120 provides anatomical positions, shapes, etc. to the surgical computer 150.

[0128] Although the display 125 is typically used to output data for presentation to a user, it may also provide data to the surgical computer 150. For example, for embodiments where a monitor is used as part of the display 125, the surgeon 111 may interact with the GUI to provide input, which is sent to the surgical computer 150 for further processing. For AR applications, the measured position and displacement of the HMD may be sent to the surgical computer 150 so that it may update the presented view as needed.

[0129] During the postoperative phase of the nursing period, various types of data can be collected to quantify the overall improvement or deterioration of the patient's condition caused by the operation. The data can take the form of self-reported information reported by the patient through a questionnaire, for example. For example, in the case of a knee replacement surgery, the Oxford knee score questionnaire can be used to measure the functional status, and the postoperative quality of life can be measured by the EQ5D-5L questionnaire. Other examples in the case of hip replacement surgery can include the Oxford hip score, the Harris hip score, and the WOMAC (Western Ontario and McMaster University Osteoarthritis Index). Such a questionnaire can be managed, for example, by a medical professional directly in a clinical setting, or managed using a mobile application that allows the patient to answer questions directly. In some embodiments, one or more wearable devices that collect data related to the operation can be provided to the patient. For example, after performing a knee operation, a knee brace can be provided to the patient, and the knee brace includes a sensor for monitoring knee position, flexibility, etc. This information can be collected and transmitted to the patient's mobile device for the surgeon to view to evaluate the results of the operation and solve any problems. In some embodiments, one or more cameras can obtain and record the movement of the patient's body parts during the activity specified after surgery. This motion acquisition can be compared to the biomechanical model to better understand the function of the patient's joint and better predict rehabilitation progress and determine any modifications that may be needed.

[0130] The postoperative phase of the care period can continue throughout the patient's life cycle. For example, in some embodiments, the surgical computer 150 or other components including CASS100 can continue to receive and collect data related to the surgical procedure after the operation is performed. The data may include, for example, images, answers to questions, "normal" patient data (e.g., blood type, blood pressure, condition, medication, etc.), biometric data (e.g., gait, etc.), and objective and subjective data about specific problems (e.g., knee or hip pain). The data can be explicitly provided to the surgical computer 150 or other CASS components by the patient or the patient's physician. Alternatively or additionally, the surgical computer 150 or other CASS components can monitor the patient's EMR and retrieve relevant information when it is available. This longitudinal view of patient rehabilitation allows the surgical computer 150 or other CASS components to provide a more objective analysis of patient results to measure and track the success of a given procedure. For example, the conditions experienced by the patient long after the surgical procedure can be linked to the surgery by performing regression analysis on various data items collected during the care period. The analysis can be further enhanced by analyzing a group of patients with similar procedures and / or similar anatomical structures.

[0131] In some embodiments, data is collected at a central location to provide easier analysis and use. In some cases, data can be collected manually from various CASS components. For example, a portable storage device (e.g., a USB stick) can be attached to surgical computer 150 to retrieve data collected during surgery. The data can then be transferred to a centralized storage device, such as via a desktop computer. Alternatively, in some embodiments, surgical computer 150 is directly connected to a centralized storage device via network 175, such as Figure 2C as shown in .

[0132] Figure 2C A "cloud-based" embodiment is shown, in which the surgical computer 150 is connected to a surgical data server 180 via a network 175. The network 175 may be, for example, a private intranet or the Internet. In addition to data from the surgical computer 150, other sources may also transmit relevant data to the surgical data server 180. Figure 2CThree additional data sources are shown: patient 160, healthcare professional 165, and EMR database 170. Thus, patient 160 can send pre-operative and post-operative data to surgical data server 180, for example, using a mobile application. Healthcare professionals 165 include the surgeon and his or her staff and any other professionals working with patient 160 (e.g., personal physician, rehabilitation specialist, etc.). It should also be noted that EMR database 170 can be used for both pre-operative and post-operative data. For example, assuming that patient 160 has given adequate permission, surgical data server 180 can collect the patient's pre-operative EMR. The surgical data server 180 can then continue to monitor the EMR for any updates after surgery.

[0133] At the surgical data server 180, the nursing period database 185 is used to store various data collected during the patient's nursing period. The nursing period database 185 can be implemented using any technology known in the art. For example, in some embodiments, a SQL-based database can be used, in which all various data items are structured in a way that allows them to be easily incorporated into two SQL sets of rows and columns. However, in other embodiments, a No-SQL database can be used to allow unstructured data while providing the ability to quickly process and respond to queries. As understood in the art, the term "No-SQL" is used to define a class of databases that are not related in their design. Various types of No-SQL databases can generally be grouped according to their underlying data models. These groups can include databases using column-based data models (e.g., Cassandra), document-based data models (e.g., MongoDB), key-value-based data models (e.g., Redis), and / or graph-based data models (e.g., Allego). Any type of No-SQL database can be used to implement the various embodiments described herein, and in some embodiments, different types of databases can support the nursing period database 185.

[0134] Data may be transmitted between the various data sources and the surgical data server 180 using any data format and transmission technology known in the art. Figure 2C The architecture shown in allows for the transmission of data from data sources to the surgical data server 180, and the retrieval of data from the surgical data server 180 by the data sources. For example, as explained in detail below, in some embodiments, the surgical computer 150 can use data from past surgeries, machine learning models, etc. to help guide the surgical procedure.

[0135] In some embodiments, the surgical computer 150 or surgical data server 180 may perform a de-identification process to ensure that the data stored in the care session database 185 meets Health Insurance Portability and Accountability Act (HIPAA) standards or other requirements imposed by law. HIPAA provides a list of certain identifiers that must be removed from the data during de-identification. The aforementioned de-identification process can scan these identifiers in the data transmitted to the care session database 185 for storage. For example, in one embodiment, the surgical computer 150 performs a de-identification process before initially transmitting a particular data item or group of data items to the surgical data server 180. In some embodiments, a unique identifier is assigned to the data from a particular care session to allow for re-identification of the data when necessary.

[0136] although Figure 2A-2C Data collection in the context of a single care session has been discussed, but it should be understood that the general concept can be extended to data collection for multiple care sessions. For example, surgical data can be collected throughout the care session each time a surgery is performed using CASS100, and stored at the surgical computer 150 or surgical data server 180. As further explained in detail below, a robust database of care session data allows the generation of optimized values, measurements, distances or other parameters, and other suggestions related to surgical procedures. In some embodiments, various data sets are indexed in a database or other storage medium in a manner that allows rapid retrieval of relevant information during a surgical procedure. For example, in one embodiment, a patient-centric set of indexes can be used so that data for a group of patients similar to a specific patient or a specific patient can be easily extracted. This concept can be similarly applied to surgeons, implant characteristics, CASS component types, etc.

[0137] Further details for managing episode of care data are described in PCT / US2019 / 067845, filed on December 20, 2019, entitled “Methods and Systems for Providing an Episode of Care,” the entire contents of which are incorporated herein by reference.

[0138] Open vs. Closed Digital Ecosystems

[0139] In some embodiments, CASS is designed to be used as a standalone or "closed" digital ecosystem. Each component of CASS is specifically designed to be used in a closed ecosystem, and devices outside the digital ecosystem generally cannot access the data. For example, in some embodiments, each component includes software or firmware that implements proprietary protocols for activities such as communication, storage, security, etc. The concept of a closed digital ecosystem may be ideal for companies that want to control all components of CASS to ensure that certain compatibility, security, and reliability standards are met. For example, CASS can be designed so that new components cannot be used with CASS unless they are certified by the company.

[0140] In other embodiments, CASS is designed to function as an "open" digital ecosystem. In these embodiments, components can be produced by a variety of different companies, and the components implement standards for activities such as communication, storage, and security. Thus, by using these standards, any company is free to build independent, compliant components of the CASS platform. Data can be transferred between components using publicly available application programming interfaces (APIs) and open, shareable data formats.

[0141] CASS Query and CASS Recommendation

[0142] Simple joints, such as ball and socket joints (e.g., hip and shoulder joints) or pivot joints (e.g., elbow joints), or more complex joints, such as condylar joints (e.g., knee joints), are incredibly complex systems whose performance can be significantly affected by a variety of factors. Procedures for replacing, resurfacing, or otherwise repairing these joints are common, such as in response to damage or other degeneration of the joints. For example, TKA, which replaces the articular surfaces of the femur, tibia, and / or patella with an artificial implant, is a common procedure for patients suffering from knee degeneration or trauma.

[0143] Selecting the best parameters for performing joint surgery is challenging. To continue with the example of knee replacement surgery, the surgeon may place a first prosthesis on the distal end of the femur and a second prosthesis at the proximal end of the tibia, or the surgeon may install the prostheses in the reverse order. The surgeon seeks to optimally place the prostheses with respect to various parameters, such as the gap between the prostheses throughout the range of motion. Misplacement of an implant may negatively impact the quality of life of a patient after surgery. For example, if the gap between the tibia and femur is too small at any time during the range of motion, the patient may experience painful joints. On the other hand, if the gap is too large, the knee joint is too loose and may become unstable.

[0144] In some embodiments, CASS (or preoperative planning application) 100 is configured to generate recommendations based on queries received from surgeons or surgical staff. Examples of recommendations that can be provided by CASS100 include, but are not limited to, optimizing one or more surgical parameters, optimizing implant positions and orientations relative to one or more reference points such as anatomical or mechanical axes, modifications of surgical plans, or descriptions of how to achieve specific results. As described above, the various components of CASS100 generate various types of data that jointly define the state of the system. In addition, CASS100 can access various types of preoperative data (e.g., patient demographic data, preoperative images, etc.), historical data (e.g., from other surgeries performed by the same or different surgeons), and simulation results. Based on all of these data, CASS100 can operate in a dynamic manner and allow surgeons to modify surgical plans intelligently and instantly as needed. In some embodiments, these modifications are performed before surgery (e.g., before printing cutting guides). In some embodiments, without using custom cutting guides (e.g., non-patient-specific cutting guides that can be selected and placed by CASS can be selected), they can be modified by CASS during surgery. Thus, for example, in some embodiments, CASS 100 notifies the surgeon via display 125 of a modified surgical plan or optimization based on conditions that were not detected prior to surgery.

[0145] In some embodiments, a surgical plan can be created before surgery using preoperative images and data. These images may include X-ray, CT, MRI, and ultrasound images. The data may include characteristics of the patient, including joint geometry, age, weight, activity level, etc., as well as data about the prosthesis to be implanted. The plan can then be modified based on additional information collected during surgery. For example, additional medical images can be taken during the surgical procedure, and the additional medical images can be used to modify the surgical plan based on additional physiological details collected from such images. In some embodiments, the surgical plan is based on patient information without the need to capture three-dimensional images, such as through a CT or MRI scan of the patient. Additional optical or X-ray images can be taken during surgery to provide additional details and change the surgical plan, thereby allowing the surgical plan to be formulated and modified without the need for expensive medical imaging before surgery.

[0146] The processor of CASS100 can recommend any aspect of the surgical plan and modify this recommendation based on new data collected during the operation. For example, the processor of CASS100 can optimize the anteversion and abduction angle of hip cup placement (in hip replacement) or the depth and orientation of the distal and posterior femoral cutting planes and patellar configuration (in PKA / TKA) in response to images captured before or during the operation. Once the initial default plan is generated, the surgeon can request recommendations for specific aspects of the operation and can deviate from the initial surgical plan. The request for recommendation may generate a new plan, partially deviate from the initial or default plan, or confirm and approve the initial plan. Therefore, by using the processor of CASS100, using a data-driven approach, the surgical plan can be updated and optimized when the operation occurs. As explained throughout, these optimizations and recommendations can be generated by the processor before or during the operation based on a statistical model or transfer function of the patient's anatomical structure from multiple simulations, and the statistical model or transfer function is informed by the simulation of the specific details of the patient's anatomical structure receiving the operation. Thus, any additional data collected about the patient's anatomy can be used to update the statistical model for that patient to optimize implant characteristics to maximize the performance criteria for the expected outcome of the surgery in the surgical plan.

[0147] Figure 3A A high-level overview of how recommendations can be generated is provided. This workflow begins at 305, with the surgical staff executing the surgical plan. The surgical plan can be an original plan generated based on preoperative or intraoperative imaging and data, or the plan can be a modification of the original surgical plan. At 310, the surgeon or a member of the surgical staff requests a recommendation on how to solve one or more problems in the surgical procedure. For example, the surgeon may request a recommendation on how to best align an implant based on intraoperative data (e.g., acquired using a point probe or new images). In some embodiments, a request can be made by manually entering a specific request into the GUI or voice interface of CASS 100. In other embodiments, CASS 100 includes one or more microphones that collect verbal requests or inquiries from the surgeon, which are translated into formal requests (e.g., using natural language processing techniques).

[0148] Continue to refer Figure 3A, at 315, a recommendation is provided to the surgeon. Various techniques may be used to provide the recommendation. For example, when the recommendation provides a recommended cut to be made or a recommended implant orientation alignment, a graphical representation of the recommendation may be depicted on the display 125 of CASS100. In one embodiment, the recommendation may be overlaid on the patient's anatomical structure in the AR HMD 155. The resulting performance characteristics may be presented (e.g., for PKA / TKA, various flexions and bends for ligament tension showing range of motion, medial and lateral condylar gaps and patellar groove tracking, or for THA, a drawing of range of motion and pressure center and edge load stress between the femoral head and acetabulum). In some embodiments, information may be transmitted to the user via the display 125 (which may include the HMD 155) in the form of a drawing, a number, or by changing a color or indicator. For example, in PKA / TKA, when planned changes or patient data indicate through statistical models that the patella will encounter tracking issues relative to the patellar groove or overstrained patellar ligament without additional changes, an image of the patella (e.g., overlaid on the patient image) may emit a red light or flash. For THA, a portion of the acetabular cup may illuminate to indicate where edge loads or risk of dislocation are increased. In some embodiments, the interface may then invite the user to click to obtain a recommended solution, such as patellar filling, ligament release, or posture changes of the patellar implant or femoral implant (PKA / TKA) or acetabular cup anteversion and abduction angle (THA) to optimize performance.

[0149] In addition to the recommendation, in some embodiments, the CASS / planning system 100 can also provide a reason for the recommendation. For example, for the suggested alignment or orientation of the implant, CASS100 can provide a list of patient-specific features or activities that affect the recommendation. The alignment or orientation of the CASS recommendation of the implant can further refer to a reference system or reference point, such as an anatomical or mechanical axis or a distance from a bone or bone landmark. In addition, as shown in 320, CASS can simulate how selecting a specific recommendation will affect the rest of the surgical procedure. For example, before surgery, a default cutting guide can be generated based on a preoperative three-dimensional CT or MRI scan. During surgery, once an incision is made, the surgeon can obtain high-resolution data of the patient's anatomical structure from an MRI or using a point probe or optical camera. CASS100 can use such data to create a new or updated plan or recommendation about using a resection tool or cutting guide to remove bone tissue. At step 320, the impact of this revised plan or recommendation can be presented in the form of revised alignment instructions, etc. It is also possible to present multiple recommendations to the surgeon and observe the impact of each recommendation on the surgical plan. For example, two feasible recommended bone resection plans or recommendations can be generated, and the surgeon can decide which one to execute based on the impact of each recommendation on the subsequent steps of the operation. Animations of range of motion or dynamic activities (such as walking or climbing stairs, etc.) can also be presented to the surgeon, which are the functional results of each recommendation to allow the surgeon to better understand how the recommendation will affect the patient's movement characteristics. In some embodiments, the surgeon is able to select a single data item or parameter (such as alignment, tension, and flexion gap, etc.) for optimizing the recommendation. Finally, once the surgeon selects a specific recommendation, it is performed at step 325. For example, in an embodiment where a custom cutting guide is manufactured before surgery, step 325 can be performed by printing a cutting guide and delivering it to the surgeon for use during surgery. In an embodiment where the robotic arm holds the cutting guide at a specific predetermined position, a command to place the cutting guide can be sent to the robotic arm as part of the CASS workflow. In an embodiment where a cutting guide is not used, CASS can receive instructions to help the surgeon remove the femoral component and tibia according to the recommendation.

[0150] CASS100 can present recommendations to the surgeon or surgical staff at any time before or during surgery. In some cases, the surgeon may explicitly request Figure 3ARecommendations discussed. In other embodiments, CASS100 may be configured to execute the recommendation algorithm as a background process while surgery is being performed based on available data. When a new recommendation is generated, CASS100 may notify the surgeon using one or more notification mechanisms. For example, a visual indicator may be presented on the display 125 of CASS100. Ideally, the notification mechanism should be relatively unobtrusive so that it does not interfere with the surgery. For example, in one embodiment, different text colors may be used to indicate that a recommendation is available as the surgeon navigates through the surgical plan. In some embodiments, when an AR or VR headset 155 is used, the recommendation may be provided to the user's field of view and highlighted to draw the user's attention to the recommendation or the availability of the recommendation. The user (e.g., a surgeon or technician) may then interact with the recommendation or solicitation of the recommendation in the AR or VR user interface using any of the means described below. In these embodiments, CASS100 may treat the user's headset 155 as an additional display and use any conventional means for communicating with the display to convey information to be displayed thereon.

[0151] To illustrate one type of recommendation that can be performed with CASS100, a technique for optimizing surgical parameters is disclosed below. The term "optimization" in this article refers to selecting the best parameters based on certain specified criteria. In extreme cases, optimization can refer to selecting the best parameters based on data from the entire care period (including any preoperative data, the state of the CASS data at a given time point, and the postoperative goals). Moreover, optimization can be performed using historical data, such as data generated during past surgeries involving, for example, the same surgeon, past patients with similar physical characteristics to the current patient, etc.

[0152] The optimized parameters can depend on the part of the patient's anatomy to be operated on. For example, for knee surgery, the surgical parameters can include positioning information of the femoral and tibial components, including but not limited to rotational alignment (e.g., varus / valgus rotation, external rotation, flexion rotation of the femoral component, posterior tilt of the tibial component), resection depth (e.g., varus knee, valgus knee), and the type, size and position of the implant. The positioning information can also include surgical parameters for combined implants, such as overall limb alignment, combined tibiofemoral hyperextension and combined tibiofemoral resection. Other examples of parameters that CASS100 can optimize for a given TKA femoral implant include the following:

[0153]

[0154] Additional examples of parameters that CASS can optimize for a given TKA tibial implant include the following:

[0155]

[0156] For hip surgery, surgical parameters may include femoral neck resection location and angle, cup inclination, cup anteversion, cup depth, femoral stem design, femoral stem size, fit of the femoral stem in the canal, femoral offset, leg length, and femoral pattern of the implant.

[0157] Shoulder parameters may include, but are not limited to, humeral resection depth / angle, humeral shaft type, humeral offset, glenoid type and inclination, and reverse shoulder parameters such as humeral resection depth / angle, humeral shaft type, glenoid inclination / type, glenosphere orientation, glenosphere offset, and offset direction.

[0158] There are various conventional techniques for optimizing surgical parameters. However, these techniques are often computationally intensive and, therefore, often require that the parameters be determined preoperatively. As a result, the surgeon's ability to modify the optimization parameters based on problems that may arise during surgery is limited. Moreover, conventional optimization techniques often operate in a "black box" manner, with little or no explanation regarding the recommended parameter values. Therefore, if the surgeon decides to deviate from the recommended parameter values, the surgeon often does so without fully understanding the impact of the deviation on the rest of the surgical process or the impact of the deviation on the patient's quality of life after surgery.

[0159] In order to address these and other shortcomings of conventional optimization techniques, in some embodiments, optimization can be performed using buttons or other components in the GUI presented to the surgeon during the surgical workflow (e.g., on display 125 or AR HMD155). For the purposes of the following discussion, the surgeon or other healthcare professional can use any means, such as oral request / command or manual input (e.g., using a touch screen or button), to call a request for recommendation or input from CASS100. For the purposes of this application, other feedback provided to the surgeon or medical professional for these types of inquiries or requests for recommended course of action, recommended parameter optimization, or responses to such inquiries or requests is referred to as CASS recommendation request or "CASSRR". The CASS recommendation request can be called or activated at any time during surgery by the surgeon or healthcare professional. For example, for TKA, CASSRR can be called during the femoral implant planning stage, tibial implant planning stage, and / or gap planning stage. In THA surgery, CASSRR can be used during femoral neck resection, acetabular implant placement, femoral implant placement, and implant selection (e.g., size, offset, bearing type, etc.). CASSRR may be invoked, for example, by pressing a button or by speaking a specific command (eg, "optimize clearance"). As described above, the recommendation system may be configured to provide or prompt the surgeon to seek recommendations or optimizations at any time during surgery.

[0160] Figure 3B – Figure 3EIt is shown that the CASS / Planning app can be used during the surgical workflow, e.g. Figure 1 Examples of GUIs for apps depicted in FIG. These GUIs may be displayed, for example, on the display 125 of the CASS 100 or on a workstation during the planning phase of an upcoming surgery. In each example, an AI-driven recommendation request may be invoked using a button presented as a visual component of the interface. Specifically, Figure 3B An exemplary implant placement interface 330 is shown, wherein a recommendation button 335 (labeled CASSRR in this example) is displayed in the lower left corner. Figure 3C An exemplary gap planning interface 340 is shown having a button 335 .

[0161] Calling CASSRR can make it possible for the parameterized interface 370 to be optimized when the user requests a recommendation. Figure 3D The display shown in . Figure 3D In the example of the present invention, the implant placement interface 330 (such as Figure 3B ) invokes a CASSRR / recommendation request by clicking an activated button on the implant (as shown in ). The optimization parameterization interface 370 includes degree of freedom (DoF) buttons 345 related to the movement of the implant (e.g., translation, rotation, etc.). During the optimization analysis, activation of any DoF button 345 "locks" the corresponding degree of freedom. For example, if the surgeon is satisfied with the anterior or posterior positioning, the surgeon can activate the "A" button or the "P" button to lock the anterior position or the posterior position, respectively. In some embodiments, the lock button changes color or provides a different type of visual indication when switched to the locked position. Figure 3DIn the example of , the "S" button (corresponding to the upper positioning) has been locked, as shown by the graphical depiction of the lock to the right of the button. It should be noted that the use of a button for locking a position is merely one example of how a surgeon may interface with CASS100; for example, in other embodiments, the surgeon may verbally request to lock a particular position (e.g., "lock the upper positioning"), or in a VR environment, a gesture may be employed. The values ​​that may be fixed may depend on the available optimization factors and the surgeon's comfort level in optimizing different aspects of the surgery. For example, so far, it has been assumed that the optimization is performed from a fully functional kinematic perspective. Therefore, the optimization system does not really understand how the implant interacts with the bone. This can be addressed, for example, by adding image-based analysis to the optimization. However, for embodiments that do not perform such analysis, the surgeon may wish to constrain the problem based on how he / she views the bone fit. For example, with respect to the femoral component of an implant in a TKA procedure, the anterior overhang may not be known when the optimization is performed. This means that the surgeon may wish to adjust the AP position of the femoral component because he or she is viewing the implant in terms of bone fit, rather than viewing the implant in terms of kinematic performance. Thus, the surgeon can determine AP position, rotation, and possibly joint line for determining final implant location, orientation, and position. In this way, the surgeon can supplement the knowledge provided from the computer-generated optimization run, thereby allowing the surgeon to implement a surgical plan that deviates from the computational recommendations.

[0162] The buttons can also be used to provide bounds control for a given parameter used for optimization. Figure 3D In the example of , there are two boundary control buttons 355 for the castor angle. These parameters can be used to set the minimum or maximum value of the relevant parameters that will be bound to the parameters in the optimization. If the right boundary control button 355 is locked, the optimization can be configured to produce a value higher than the current value specified for the castor angle. On the contrary, if only the left boundary control button 355 is locked, the optimization can be configured to produce a value lower than the current value specified for the castor angle. If both boundary control buttons 355 are locked, the optimization is configured so that it does not change the specified castor angle value. On the other hand, if the boundary control buttons 355 are not locked, the optimization can freely change the value (within the device specification range).

[0163] The optimization parameterization interface 370 includes an optimization button 350, which, when activated, optimizes the implant placement parameters using any of the data-driven / AI methods described herein. It should be noted that this general concept is not limited to implant placement; in fact, in general, any surgical parameter or set of parameters can be optimized using similar interfaces and techniques. This optimization process is further detailed below. After optimization, the surgeon can return to the implant placement interface 330 (e.g., Figure 3B The surgery was continued with the optimized parameters.

[0164] Switch button 365 allows surgeon to switch between any two views or aspects of surgical procedure or surgical procedure plan. For example, switch button 365 can provide current bone condition and future bone condition to surgeon based on partial or complete execution of surgical plan or current planned implantation position and alternative (e.g., recommended) implantation position. Switch button 365 can also provide alternative future condition of bone and / or implant to surgeon according to whether surgeon chooses to take an action course rather than alternative action course. The activation of this button makes various images and data presented on optimization parameterization interface 370 updated with current or previous alignment information. Therefore, surgeon can quickly view the impact of any change. For example, the switching feature can allow surgeon to visualize the prosthesis positioning change suggested by optimizer relative to its previous concept of appropriate implant placement. In one embodiment, during the initial use of the system, the user can choose to plan the case without optimization, and wish to visualize the impact of automation. Similarly, the user may wish to visualize the impact of various aspects of "locking" planning.

[0165] If the surgeon wishes to understand the rationale behind the optimization, the response and rationale button 360 on the optimization parameterization interface 370 may be activated to display Figure 3E 375. This interface 375 includes an animation screen 380 that provides an animation of the anatomical structure of interest during a performance measurement activity (e.g., deep knee flexion). This animation can be used as an output of an anatomical modeling software that performs optimization (e.g., LIFEMOD TM ), or alternatively, separate software can be used to generate animations based on the output of the optimization software. For example, an animated GIF file or a small video file can be used to depict the animation. In an embodiment using an AR HMD 155, an animated hologram can be provided over the relevant anatomical structure to provide further contextualization of the simulated behavior.

[0166] The response and rationale interface 375 also includes a response screen 385 that displays plots of various performance or condition metrics (e.g., measuring the v-angle of flexion). A set of performance measure selection buttons 390 on the right hand side of the response and rationale interface 375 allows the surgeon to select various relevant performance measures and update the plots shown in the response screen 385. Figure 3E In the example of screen 385, these performance measures include internal-external (IE) rotation, medial and lateral rollback, MCL and LCL strain, iliotibial band (ITB) strain, bone stress, varus-valgus (VV) rotation, medial-lateral (ML) patellar shear, quadriceps force, and bone interface force. The example shown in screen 385 depicts lateral and medial gaps throughout the flexion range, which is a traditional estimate of TKA performance. Traditionally, lateral and medial gaps are only considered at two degrees of flexion.

[0167] To support the various interfaces described above, the algorithm supporting the CASSRR / recommend button should preferably be executed as quickly as possible to ensure that the surgical workflow is not disrupted. However, the calculations involved in performing the optimization can be computationally intensive. Therefore, in some embodiments, in order to simplify the required processing, a set of prediction equations can be generated based on the training data set and simulated performance measurements. These prediction equations provide a simplified form of the parameter space that can be optimized in near real time. These processes can be executed locally or on a remote server, such as in the cloud.

[0168] Figure 4 A system diagram is provided that illustrates how optimization of surgical parameters may be performed according to some embodiments. Such optimization may be performed during the preoperative planning phase, such as in embodiments where a custom cutting guide is created prior to surgery, or during surgery, such as in embodiments where CASS may adjust the exact posture of the resection plane robotically or by other practical means such as tactile feedback. In brief, the surgeon 111 provides certain patient-specific parameters and parameters related to the implant to the surgical computer 150 (via a GUI presented on the display 125). The surgeon 111 requests that optimization be performed. The surgical computer 150 uses the parameters to retrieve a set of predictive equations from an equation database 410 stored on the surgical data server 180. In embodiments that do not use a cloud-based architecture, this equation database 410 may be stored directly on the surgical computer 150. Optimization of the set of equations provides the desired optimization (e.g., optimal implant alignment and positioning). This information may then be presented to the surgeon 111 via the display 125 of the CASS 100 (see, e.g., Figure 3D and 3E ).

[0169] As explained in more detail below, each equation data set provides kinematic and dynamic responses for a set of parameters. In some embodiments, an equation data set derived by a simulation computer 405 based on a set of training data is used to fill the equation database 410. The training data set includes a surgical data set previously collected by CASS100 or another surgical system. Each surgical data set may include, for example, information about the patient's geometry, how the implant is positioned and aligned during surgery, ligament tension, etc. Any technology known in the art can be used to collect data. For example, for ligament tension, a robot-assisted technique can be used, as described in PCT / US2019 / 067848, entitled "Actuated Retractor with Tension Feedback", filed on December 20, 2019, the entire contents of which are incorporated herein by reference. Another example is provided in PCT / US2019 / 045551 and PCT / US2019 / 045564, entitled “Force-Indicating Retractor Device and Methods of Use,” filed on August 7, 2019, which are incorporated herein by reference in their entirety.

[0170] For each surgical data set, the simulation computer 405 performs an anatomical simulation on the surgical data set to determine a set of kinematic and dynamic responses. Non-limiting examples of suitable anatomical modeling tools that may be used include LIFEMOD TM or KNEESIM TM(Both available from LIFEMODELER, INC. of San Clemente, California, a subsidiary of SMITH AND NEPHEW, INC.). Additional examples of the use of biomechanical modeling during surgery are described in the following documents: U.S. Patent No. 8,794,977, entitled "Implant Training System"; U.S. Patent No. 8,712,933, entitled "Systems and methods for determining muscle force through dynamic gain optimization of a muscle PID controller for designing a replacement prosthetic joint"; U.S. Patent No. 8,412,669, entitled "Systems and methods for determining muscle force through dynamic gain optimization of a muscle PID controller," the entire contents of which are incorporated herein by reference.

[0171] In addition to determining the response of the surgical data set, the simulation computer 405 can also be used to supplement the real-world surgical data set with an artificially generated surgical data set, which fills any gaps in the training data set. For example, in one embodiment, a small permutation of various factors in the training set is used to perform Monte Carlo analysis to see how they affect the response. Therefore, a relatively small real-world surgical data set (e.g., 1,000 data sets) can be essentially extrapolated to produce an exponentially larger data set covering various patient anatomy, implant geometry, etc. Once the data set has been filled, one or more equation fitting techniques commonly known in the art can be used to derive the equation data set stored in the equation database 410.

[0172] In order to determine the kinematic and dynamic responses for each equation data set, the simulation performed by the simulation computer 405 can model and simulate various activities that cause stress to the anatomical structure of interest. For example, in the context of TKA or other knee surgery, a weighted deep knee flexion can be used. During deep knee flexion, the knee is flexed downward at various angles (e.g., 120°, 130°, etc.) under a certain load and returns to an upright position. During deep knee flexion, loads appear on the leg extensors (i.e., quadriceps), leg flexors (i.e., hamstrings), passive ligaments in the knee, etc. Therefore, deep knee flexion stresses the anterior cruciate ligament (ACL), the posterior cruciate ligament (PCL), the lateral collateral ligament (LCL), and the medial collateral ligament (MCL). In addition, deep knee flexion allows measurement of various kinematics (e.g., how the patella moves relative to the femoral component, how the femur moves relative to the tibia, etc.). It should be noted that this is an example of a performance measurement that can be applied, and various other measurements can be used as a supplement or alternative to deep knee flexion. Knee kinematics can also be simulated using the knee model to perform approximate real-world motions associated with dynamic activities, such as walking up and down stairs or swinging a golf club. Other joints in the body can be simulated as simple ideal components, while individual ligaments and implant components of interest that perform motion can be simulated in detail under exemplary loads associated with each activity considered.

[0173] Figure 5A-5F Exemplary joint prediction equations that can be used in the equation data set in some embodiments are described. Although these figures will be described with respect to knee replacements, the concepts are equally applicable to other joint replacements, such as the hip. Regardless of the type of joint replacement being performed, the basic categories can be the same, with specific data in each category being relevant to the surgery being performed. Figure 5A An overview of the knee prediction equation is provided. As will be explained below, the terms of this equation are simplified to allow understanding of the individual terms. Therefore, it should be understood that the exact mathematical construction may differ from those shown in the accompanying figures.

[0174] In these prediction equations, the terms on the left hand side are referred to as "factors", and the terms on the right hand side are referred to as "responses". Responses and factors can be associated with specific numerical values, but in at least some embodiments, at least some can be expressed as probability distributions (e.g., bell curves) or another way of reflecting uncertainty about the actual values ​​of factors or responses. Therefore, the equations can take into account the uncertainty of certain aspects of this process. For example, in at least some embodiments, it may be difficult to identify soft tissue attachment locations with certainty, and therefore, uncertainty information can be used to reflect the probability distribution of such soft tissue attachment locations that are actually located based on the estimated positions identified during image processing. Similarly, in at least some embodiments, rather than determining the exact optimal position and orientation of an orthopedic implant, it may be desirable to determine the optimal position and orientation in the context of a possible variable location where the implant will actually be positioned and oriented (e.g., to take into account tolerances in manufacturing custom cutting guide instruments, variability in the surgeon's surgical technique, etc.).

[0175] Figure 5B Patient-specific parameters for the knee prediction equation are shown. These parameters can be measured by the surgical staff based on preoperative or intraoperative data. Figure 5B As shown in the example of , the X-ray measurement tool can be used to measure various anatomical features in the image. Examples of patient-specific parameters that can be used include load-bearing access (LBA), pelvic width, femoral ML width, tibial ML width, femoral length, etc.

[0176] Figure 5C Soft tissue balance parameters included in the knee prediction equation are shown. Soft tissue balance parameters can be derived from multiple sources. For example, by default, parameters can be derived from a set of landmarks based on the patient's bone geometry. This can be supplemented with results from anterior drawer tests, varus-valgus stability measurements, tissue attachment estimates, and tissue condition measurements (e.g., stiffness). The data can also be supplemented with data obtained intraoperatively, such as joint distraction tests, instrumented tibial inserts, force sensing gloves, etc.

[0177] Figure 5D Implant geometry parameters for the knee prediction equation are shown. These parameters may include, for example, the femoral and tibial gaps, the distal or posterior radius, the patellar geometry and alignment or fill, and the femoral anterior-posterior / lateral-medial placement of the implant. It should be noted that for a given patient, there may be many possible implants (e.g., models, sizes, etc.). Therefore, different knee prediction equations can be designed with the same patient-specific and tissue-balanced parameters, but different implant geometry parameters. For example, a range of sizes can be represented by a set of key prediction equations. In some embodiments, a method such as LIFEMOD can be used. TMThe anatomical modeling software of the invention programmatically determines the implant geometry. Exemplary techniques for optimizing parameters related to the anatomical and biomechanical fit of an implant or implant system implanted into a patient's joint are described in U.S. Patent Application No. 13 / 814,531, previously incorporated herein by reference.

[0178] Figure 5E Implant alignment and positioning parameters that can be used in the knee prediction equations are shown. As described above, these can be used as variables during optimization. Figure 5E As shown in , example parameters include femoral superior-inferior (SI) position, femoral anterior-posterior (AP) position, femoral varus-valgus (VV) position, femoral internal-external (IE) position, tibial slope, tibial VV position, tibial IE position, and tibial depth, as determined by the extension gap.

[0179] Fig. 5F The response portion of the knee prediction equation is shown. The response can include a data set that includes kinematic data and kinetic data related to the knee. The kinematic data provides a measure of how the kinematics of a particular patient and component are set up compared to a specified target. For example, the kinematics can measure the internal-external rotation of the femoral component relative to the tibia and what this signature looks like over the flexion history of a deep knee flexion event. This can then be compared to a target measure of internal-external rotation. This concept can be extended to the patella and other anatomical structures related to knee motion. The kinetic data provides a measure of the load on the various components of the knee (e.g., LCL, MCL, etc.). As Fig. 5F As shown in , simulations derived this data for several different degrees of knee flexion (e.g., 30, 60, 90, 120, etc.). Thus, for a given set of parameters on the left hand side of the equation, a set of equations with different response values ​​can be specified.

[0180] Figure 6 The process of optimizing the set of equations according to some embodiments is shown. Beginning at step 605, patient-specific parameters and soft tissue balance parameters are input by the surgical staff. As described above, patient-specific parameters can be derived based on any combination of preoperative and intraoperative data. The tissue balance parameters are measured by the surgeon (or default values ​​can be used). At step 610, implant geometry or implant manufacturer, model, and manufacturer and product ID are input by the surgical staff. In some embodiments, the surgical staff can manually input each implant geometry parameter. In other embodiments, the surgical staff can specify a specific implant manufacturer, model, and / or size (e.g., SMITH&NEPHEW, II left femoral implant, size 6), and the appropriate implant parameters (e.g., geometry, size, or other implant characteristics) can be retrieved from a local or remote database.

[0181] Continue to refer Figure 6 , at step 615, a set of equations is selected based on patient-specific parameters, soft tissue parameters and implant geometry. The set of equations includes one or more prediction equations, each of which provides different response values. Next, at step 620, the set of equations is optimized to obtain implant alignment and position recommendations. In at least some embodiments, it may not be possible to completely solve all equations because these factors can affect various responses in different ways. Therefore, in some embodiments, responses can be associated with weighted values ​​so that the optimization process gives certain responses greater weights than other responses. These weighted values ​​can serve as desirable factors or functions that quantify the relative importance of various responses. For example, in some embodiments, optimization is performed using a goal programming (GP) algorithm, wherein the weights of the response variables are obtained by a group decision making (GDM) process. Finally, at step 625, implant alignment and position recommendations are intuitively depicted, for example, on the display 125 of CASS100.

[0182] In some embodiments, the relationship between factor and response can be defined by a set of trained neural networks rather than a series of equations. Statistical and modeling tools similar to those described above can be used to define and train neural networks and the factors used therein. In some embodiments, tools available from NEURODIMENSIONS, INC. in Gainesville, Florida, such as NEUROSOLUTIONS 6.0, can further promote the development and training of neural networks. In some embodiments, information databases collected from previous orthopedic procedures or studies can be used to train neural networks, and as additional data is collected over time, neural networks can be further improved to enhance the optimization process described herein. In some embodiments, kernel methods can be used to explore the relationship between factors and responses. Kernel-based learning algorithms can be used to solve complex computational problems by clustering, classification, etc., to detect and utilize complex patterns in data.

[0183] In some embodiments, the relationship between the factors and the response can be defined by one or more trained support vector machines. As with some neural networks, support vector machines can be trained to recognize patterns in existing data, such as data collected from previous orthopedic procedures or studies, and once trained, used to predict the response of an orthopedic procedure for a particular patient based on the settings of specific factors.

[0184] Although the above discussion relates to recommendations in the context of knee surgery, the factors and responses used in the prediction equations may be modified as needed based on the anatomy that is the subject of the surgical procedure. For example, surgery to repair a torn or damaged anterior cruciate ligament ("ACL") may benefit from the use of the CASS and CASSRR concepts described above. The application of a robotic surgical system to ACL surgery is described in PCT / US2019 / 048502, entitled "Robotic Assisted Ligament Graft Placement and Stressing," filed on August 28, 2019, which application was previously incorporated by reference in its entirety.

[0185] Another surgical intervention that can benefit from the use of the above-mentioned CASS and CASSRR concepts is a high tibial osteotomy ("HTO") procedure. In an HTO procedure, an incision in the tibia is prepared, and bone wedges can be removed or added from the incision in the tibia to better align the tibia and femur in the knee joint. For example, CASSRR can be used to optimize the amount of bone added or removed to achieve a desired dynamic response (i.e., removing the affected compartment to delay further cartilage damage). In addition, changes in tibial inclination that are difficult to plan can be simulated and implemented robotically.

[0186] In the context of THA procedures, the above discussion on Figure 5E The "implant alignment / position" factor discussed can be replaced with parameters such as cup tilt, cup anteversion, cup size, cup depth, bearing type (traditional, ceramic on ceramic, dual mobility, surface resurfacing, etc.), femoral stem design, femoral stem form, combined anteversion, femoral stem size, femoral stem offset (STD, HIGH) and femoral head offset. Another component of implant alignment / position can be screw placement. The software can make recommendations on how many screws should be used, the length and trajectory of each screw, and the problem areas (soft tissue, blood vessels, etc.) to be avoided. For hip revision surgery, the "implant geometry" factor can be modified to make recommendations for which type of acetabular or femoral component will best fill the missing anatomical structure.

[0187] In addition, although the recommendation system is discussed above about the generation of intraoperative recommendations, it should be noted that recommendations can also be applied in the preoperative and postoperative stages of the nursing period. For example, based on preoperative data and historical data, a recommended surgical plan can be formulated. Similarly, if a surgical plan has been generated, recommendations can be generated based on the changes that occur after the preoperative data are generated. After surgery, data collected in the earlier stages of the nursing period can be used to generate recommended postoperative recovery programs (e.g., goals, exercises, etc.). Examples of data that can affect repair programs include but are not limited to implant models and sizes, operating time, hemostasis time, tissue release, and intraoperative flexion. In addition to the activities of the recovery program, the device for repair and recovery can also be customized based on the nursing period data. For example, in one embodiment, the nursing period data is used to generate a design of a custom insole that can be 3D printed for a patient. In addition to generating postoperative recommendations for patients, postoperative nursing period data can also be used as a feedback mechanism in CASSRR to further improve the machine learning model for providing recommendations for performing surgical operations on other patients.

[0188] Slider interface for providing interactive anatomical modeling data

[0189] In some embodiments, as an alternative or in addition to the interfaces described above, dynamic sliders can be used to depict various measurements, such as Figure 7A-3 Although Figure 7A-3 Continuing with the example of knee implant alignment, it will be appreciated that Figure 7A-3 The general concepts shown in can be applied to various types of measurements performed during surgery or in the preoperative phase. Figure 7A-1 , the surgeon manually aligns the implant (as depicted in image 705), or manipulates the implant on a display screen using the above-described optimization process (as shown in image 710) to produce a response 715. In this case, the response 715 is shown as a plurality of sliders with settings corresponding to a plurality of flexion angles (30, 60, 90, and 120 degrees). These sliders are "dynamic" in the sense that they are updated in real time when the surgeon changes the alignment. Therefore, for example, if the surgeon manually moves the implant 705, the response 715 will be recalculated, and each slider will be updated accordingly.

[0190] Figure 7B A further illustration of the slider contents is provided showing the sagittal alignment response of the tibial implant. Figure 7B, the desired or preferred alignment configuration is 5 degrees of flexion, but the current flexion measurement is only 4 degrees. Instructions, prompts or notes can be provided to indicate that the flexion should be changed (e.g., reduced or increased) to avoid cutting the fibula. The current implant alignment is depicted with a marker on the lower portion of the slider, and the surgeon's 5 degree target is displayed as a marker on the upper portion of the slider. FDA 510 (k) limits (which define the allowable parameters that meet FDA regulations) are shown with an outer bar overlaid on the slider.

[0191] exist Figure 7B In an example of , the slider also includes an internal bar overlaid on the slider that shows the surgeon's restrictions on the response value. These restrictions can be derived or determined by reviewing and analyzing certain historical data (i.e., restrictions from past surgeries) that can be obtained from CASS, or the surgical staff or other technicians can enter this information before the operation. In some embodiments, each restriction is not explicitly provided; instead, the restrictions are derived from a set of general instructions provided by the surgeon. For example, in one embodiment, the surgeon or the surgeon's staff provides a text description of the surgeon's preferences. Then, a natural language processing algorithm is applied to extract relevant information from the text. Based on the extracted text, rules for generating text are generated. The following table provides an example set of rules generated based on the input text shown.

[0192]

[0193] Surgical Patient Care Systems

[0194] The general concept of optimization can be extended to the entire care period using a surgical patient care system 820 that uses surgical data as well as other data from the patient 805 and healthcare professionals 830 to optimize outcomes and patient satisfaction, such as Figure 8 as shown in .

[0195] Conventionally, preoperative diagnosis, preoperative surgical planning, intraoperative execution of the established plan, and postoperative management of total joint replacements are based on personal experience, published literature, and the surgeon's training knowledge base (ultimately, the tribal knowledge of the individual surgeon and his or her peer "network" and journal publications) and their instinct for accurate intraoperative tactile discrimination of "balance" and accurate manual execution of planar resections using guidance and visual cues. This existing knowledge base and execution style is limited in terms of optimizing outcomes for patients who need care. For example, there are limitations in: accurately diagnosing patients for appropriate, minimally invasive established care; aligning dynamic patient, medical economic, and surgeon preferences with patient desired outcomes; executing surgical plans to properly align and balance bones, etc.; and receiving data from disconnected sources with different biases that are difficult to reconcile into a holistic patient framework. Therefore, data-driven tools that more accurately model anatomical responses and guide surgical planning can improve upon existing approaches.

[0196] The surgical patient care system 820 is designed to utilize patient-specific data, surgeon data, medical institution data, and historical outcome data to develop algorithms that suggest or recommend the best overall treatment plan for the patient's entire care period (preoperative, intraoperative, and postoperative) based on the desired clinical outcomes. For example, in one embodiment, the surgical patient care system 820 tracks compliance with the suggested or recommended plan and adjusts the plan based on patient / care provider performance. Once the surgical treatment plan is completed, the surgical patient care system 820 records the collected data in a historical database. The database can be accessed by future patients and used to develop future treatment plans. In addition to utilizing statistical and mathematical models, simulation tools (e.g., ) simulates outcomes, alignment, kinematics, etc. based on a preliminary or proposed surgical plan, and reconfigures the preliminary or proposed plan to achieve a desired or optimal outcome based on the patient's profile or the surgeon's preferences. The surgical patient care system 820 ensures that each patient is receiving personalized surgical and rehabilitation care, thereby improving the chances of a successful clinical outcome and reducing the financial burden on the facility associated with near-term revisions.

[0197] In some embodiments, the surgical patient care system 820 employs a data collection and management approach to provide a detailed surgical case plan with different steps that are monitored and / or executed using CASS 100. The user's execution is calculated as each step is completed and used to suggest changes to subsequent steps of the case plan. The generation of the case plan relies on a series of input data stored in a local or cloud storage database. The input data can be related to the patient currently being treated or historical data from patients who have received similar treatments.

[0198] The patient 805 provides inputs such as current patient data 810 and historical patient data 815 to the surgical patient care system 820. Various methods generally known in the art can be used to collect such inputs from the patient 805. For example, in some embodiments, the patient 805 fills out a paper or digital survey that the surgical patient care system 820 parses to extract patient data. In other embodiments, the surgical patient care system 820 can extract patient data from existing information sources such as electronic medical records (EMRs), health history files, and payer / provider history files. In still other embodiments, the surgical patient care system 820 can provide an application program interface (API) that allows external data sources to push data to the surgical patient care system. For example, the patient 805 may have a mobile phone, wearable device, or other mobile device that collects data (e.g., heart rate, pain or discomfort level, movement or activity level, or patient-submitted responses to any number of preoperative planning standards or condition compliance) and provides the data to the surgical patient care system 820. Similarly, the patient 805 may have a digital application on his or her mobile or wearable device that can collect data and transmit it to the surgical patient care system 820.

[0199] Current patient data 810 may include, but is not limited to: activity level, past conditions, comorbidities, pre-rehabilitation performance, health and fitness level, pre-operative expectation level (related to hospital, surgery, and rehabilitation), metropolitan statistical area (MSA) driver score, genetic background, previous injuries (sports, trauma, etc.), previous joint replacements, previous trauma surgeries, previous sports medicine surgeries, treatment of contralateral joints or limbs, gait or biomechanical information (dorsal and ankle tissue), pain or discomfort level, care infrastructure information (payer coverage type, home medical infrastructure level, etc.), and an indication of the expected ideal outcome of the surgery.

[0200] Historical patient data 815 may include, but is not limited to: activity level, past conditions, comorbidities, pre-rehabilitation performance, health and fitness level, pre-operative expectation level (related to hospital, surgery, and rehabilitation), MSA driver score, genetic background, previous injuries (sports, trauma, etc.), previous joint replacements, previous trauma surgeries, previous sports medicine surgeries, treatment of contralateral joints or limbs, gait or biomechanical information (dorsal and ankle tissues), pain or discomfort level, care infrastructure information (payer coverage type, home medical infrastructure level, etc.), expected ideal outcome of the surgery, actual outcome of the surgery (patient reported outcomes [PRO], implant survival, pain level, activity level, etc.), size of implant used, position / orientation / alignment of implant used, soft tissue balance achieved, etc.

[0201] The healthcare professional 830 performing the surgery or treatment can provide various types of data 825 to the surgical patient care system 820. For example, the healthcare professional data 825 can include a description of known or preferred surgical techniques (e.g., cruciate retention (CR) vs. posterior stabilization (PS), size increase vs. size reduction, with tourniquet vs. without tourniquet, femoral stem style, preferred options for THA, etc.), the healthcare professional's 830 training level (e.g., years in practice, position trained, place trained, technique they modeled), previous success levels including historical data (outcomes, patient satisfaction), and expected ideal results with respect to range of motion, recovery days, and device survival. The healthcare professional data 825 can be obtained, for example, through a paper or digital survey provided to the healthcare professional 830, via the healthcare professional's input to a mobile application, or by extracting relevant data from an EMR. In addition, CASS100 can provide data such as profile data (e.g., patient-specific knee device profile) or historical records describing the use of CASS during surgery.

[0202] Information about the facility where the surgery or treatment is to be performed may be included in the input data. This data may include, but is not limited to, the following: ambulatory surgical center (ASC) vs. hospital, facility trauma level, comprehensive care plan (CJR) or bundle candidacy for joint replacement, MSA driver score, community vs. metropolitan, academic vs. non-academic, postoperative network access (skilled nursing facility [SNF] only, home health, etc.), availability of medical professionals, availability of implants, and availability of surgical equipment.

[0203] These facility inputs may be obtained, for example, but not limited to, via surveys (paper / digital), surgical planning tools (e.g., apps, websites, electronic medical records [EMR], etc.), hospital information databases (on the Internet), etc. Input data related to associated healthcare economics may also be obtained, including, but not limited to, the patient's socioeconomic profile, the expected level of reimbursement the patient will receive, and whether the treatment is patient-specific.

[0204] These healthcare economic inputs can be obtained, for example but not limited to, through surveys (paper / digital), direct payer information, socioeconomic status databases (zip codes available on the Internet), etc. Finally, data derived from simulations of the procedure are obtained. Simulation inputs include implant size, location, and orientation. Custom or commercially available anatomical modeling software programs (such as It should be noted that the above data inputs may not be available for every patient and the available data will be used to generate the treatment plan.

[0205] Prior to surgery, patient data 810, 815 and healthcare professional data 825 may be acquired and stored in a cloud-based or online database (e.g., Figure 2C The surgical data server 180 shown in ). Information related to the procedure is provided to the computing system manually via wireless data transmission or using portable media storage. The computing system is configured to generate a case plan for CASS100. The generation of the case plan will be described below. It should be noted that the system can access historical data of previously treated patients, including implant sizes, positions and orientations automatically generated by a computer-aided patient-specific knee instrument (PSKI) selection system or CASS100 itself. To this end, a surgical sales representative or case engineer uploads case log data to a historical database using an online portal. In some embodiments, data transmission to the online database is wireless and automated.

[0206] Historical data sets from online databases are used as input to machine learning models (e.g., recurrent neural networks (RNNs) or other forms of artificial neural networks). As is generally understood in the art, artificial neural networks function similarly to biological neural networks and consist of a series of nodes and connections. The machine learning model is trained to predict one or more values ​​based on the input data. For the following sections, it is assumed that the machine learning model is trained to generate prediction equations. These prediction equations can be optimized to determine the optimal size, position, and orientation of the implant to achieve the best results or satisfaction.

[0207] Fig. 9 An example of determining a prediction equation from machine learning using seed data according to some embodiments is shown. An input signal including information from an online database (described previously) is introduced into the system as an input node. Each input node is connected to a series of downstream nodes for calculations in the hidden layer. Each node is usually represented by a real number (usually between 0 and 1), but the connection between the nodes also has a weighted value that changes as the system "learns". In order to train the system, a set of seeds or training data are provided with associated known output values. The seed data is iteratively passed to the system, and the inter-node weighted values ​​are changed until the system provides results that match the known outputs. In this embodiment, the weighted values ​​in the hidden layer of the network are captured in a weighted matrix that can be used to characterize surgical performance. The weighted matrix values ​​are used as coefficients in the prediction equation, which associates the database input with the results and satisfaction. Initially, the RNN will be trained with seed data developed through clinical research and registration data. Once a sufficient number of cases have been established in the database, the system will use historical data for system improvement and maintenance. Note that the use of RNN will act as a filter to determine which input data has a greater impact on the output. The system operator may select a sensitivity threshold so that input data that has no significant effect on the output may be ignored and no longer captured for analysis.

[0208] Fig.10 An embodiment 1000 of the manner in which a surgical patient care system 820 may be used during surgery is shown in accordance with some embodiments. Beginning at step 1005, the surgical staff begins surgery with CASS. CASS may be an image-based or image-free system, as is generally understood in the art. Regardless of which type of system is employed, at step 1010, the surgical staff may access or acquire a 3D representation of the patient's relevant body anatomy (traditional probe drawing, 3D imaging with reference mapping, visual edge detection, etc.). In many cases, the 3D representation of the anatomical structure may be mathematically achieved by capturing a series of Cartesian coordinates representing the surface of the tissue. Example file formats include, but are not limited to, .stl, .stp, .sur, .igs, .wrl, .xyz, etc. The 3D representation of the patient's relevant body anatomy may be generated preoperatively based on, for example, image data, or the 3D representation may be generated using CASS during surgery.

[0209] Certain input data of the current patient can be loaded on the computing system, for example, by wireless data transmission or using a portable storage medium. The input file is read into the neural network, and the resulting prediction equation is generated at step 1015. Next, at step 1020, a global optimization of the prediction equation is performed (e.g., using direct Monte Carlo sampling, random tunneling, parallel tempering, etc.) to determine the optimal size, position and orientation of the implant, so as to achieve the best result or satisfaction level, and the corresponding resection to be performed is determined based on the implant size, position and orientation. During the optimization phase, the system operator can choose to ignore aspects of the equation. For example, if the clinician believes that the input related to the patient's economic status is irrelevant, the coefficients related to these inputs can be deleted from the equation (e.g., based on the input provided by the GUI of CASS100).

[0210] In some embodiments, rather than using an RNN to calculate the prediction equation, a Design of Experiments (DOE) approach is used. The DOE will provide sensitivity values ​​that relate each input value to an output value. The valid inputs are combined in the mathematical formula previously described as the prediction equation.

[0211] No matter how to configure or determine the prediction equation, the optimization of this equation can provide recommendation, preferred or optimized implant positioning, for example in the form of a homogeneous transformation matrix. The transformation determines the implant component size and the directional implant component mathematically relative to the patient's anatomical structure. The Boolean intersection of implant geometry and the patient's anatomical structure produces a volume representation of the bone to be removed. This volume is defined as a "cutting envelope". In many commercially available orthopedic robotic surgery systems, bone removal tools (with optical tracking and other methods) are tracked relative to the patient's anatomical structure. Position feedback control is used to adjust the speed or depth of the cutting tool based on the tool position in the cutting envelope (that is, when the position of the tool end is in the cutting envelope, the cutting tool will rotate, and when its position is outside the cutting envelope, the cutting tool stops or retracts).

[0212] Once the procedure is complete, all patient data and available outcome data, including implant size, position, and orientation determined by CASS, are collected and stored in a historical database at step 1025. Any subsequent calculation of the objective equation by the RNN will include data from previous patients in this manner, allowing for continuous improvement of the system.

[0213] In addition to or as an alternative to determining implant positioning, in some embodiments, prediction equations and associated optimization can be used to generate resection planes for use with the PSKI system. When used with the PSKI system, the calculation and optimization of the prediction equations are completed before surgery. The patient's anatomical structure is estimated using medical image data (X-ray, CT, MRI). Global optimization of the prediction equations can provide the ideal size and position of the implant components. The Boolean intersection of the implant components and the patient's anatomical structure is defined as the resection volume. PSKI can be generated to remove the optimized resection envelope. In this embodiment, the surgeon cannot change the surgical plan intraoperatively.

[0214] The surgeon may choose to change the surgical case plan at any time before or during surgery. If the surgeon chooses to deviate from the surgical case plan, the sizes, positions, and / or orientations of the altered components are locked, and the global optimization (using the previously described techniques) is refreshed to find new ideal positions for the other components based on the new sizes, positions, and / or orientations of the components, and the corresponding resections that need to be performed to achieve the new optimized sizes, positions, and / or orientations of the components. For example, if the surgeon determines that the size, position, and / or orientation of the femoral implant in a TKA needs to be updated or modified intraoperatively, the position of the femoral implant will be locked relative to the anatomy, and a new optimal position for the tibia will be calculated (via global optimization) by taking into account the surgeon's changes to the size, position, and / or orientation of the femoral implant. Additionally, if the surgical system used to implement the case plan is robotically assisted (e.g., using a robotically assisted system), the surgeon may also be able to perform a global optimization to find new optimal positions for the other components and the corresponding resections that need to be performed to achieve the new optimized sizes, positions, and / or orientations of the components. If, for example, a 3D-printed surgical system such as the MAKO or MAKO Rio is used, bone removal and bone morphology during surgery can be monitored in real time. If the resection performed during the procedure deviates from the surgical plan, the processor can optimize the subsequent placement of additional components taking into account the actual resection that has been performed.

[0215] Fig.11A A surgical patient care system 820 ( Figure 8 ) can be adapted to perform a case plan matching service. In this example, data 810 related to the current patient 805 is obtained and compared with all or part of a historical database of patient data and related results 815. For example, the surgeon may choose to compare the plan of the current patient with a subset of the historical database. The data in the historical database may be filtered to include, for example, only data sets with good results, data sets corresponding to historical surgeries of patients with profiles that are the same or similar to the current patient profile, data sets corresponding to specific surgeons, data sets corresponding to specific aspects of the surgical plan (e.g., surgery that only preserves specific ligaments), or any other criteria selected by the surgeon or medical professional. For example, if the current patient data matches or correlates with data from a previous patient who experienced good results, the case plan of the previous patient may be accessed and adapted or adopted for the current patient. The prediction equation may be used in conjunction with an intraoperative algorithm that identifies or determines actions associated with the case plan. Based on relevant information from the historical database and / or pre-selected information, the intraoperative algorithm determines a series of recommended actions for the surgeon to perform. Each execution of the algorithm generates the next action in the case plan. If the surgeon performs the action, the result is evaluated. The results of the actions performed by the surgeon are used to refine and update the inputs to the intraoperative algorithm, which is used to generate the next step in the case plan. Once the case plan has been fully executed, all data related to the case plan (including any deviations from the surgeon's execution of the suggested actions) are stored in a database of historical data. In some embodiments, the system uses preoperative, intraoperative, or postoperative modules in a segmented manner, rather than across the entire continuum of care. In other words, the caregiver can prescribe any permutation or combination of treatment modules, including the use of a single module. These concepts are described in detail in the literature. Fig. 11B and can be applied to any type of surgery using CASS100.

[0216] The surgical process is shown

[0217] As mentioned above about Figure 1-2CAs described, the various components of CASS100 generate detailed data records during surgery. CASS100 can track and record the various actions and activities of the surgeon during each step of the surgery, and compare the actual activities with the preoperative or intraoperative surgical plan. In some embodiments, the data can be processed into a format that can effectively "play back" the surgery using software tools. For example, in one embodiment, one or more GUIs can be used, which show all the information presented on the display 125 during surgery. This can be supplemented with graphics and images showing data collected by different tools. For example, a GUI that provides a visual illustration of the knee during tissue resection can provide a measured torque and displacement of the resection device adjacent to the visual illustration to better provide an understanding of any deviation from the planned resection area that occurs. The ability to view the playback of the surgical plan or switch between different aspects of the actual surgery and the surgical plan can benefit surgeons and / or surgical staff, allowing such personnel to identify any deficiencies or challenging aspects of the surgery, which can be modified in future surgeries. Similarly, in an academic environment, the above-mentioned GUI can be used as a teaching tool for training future surgeons and / or surgical staff. Additionally, because the dataset effectively records many aspects of a surgeon's activities, it can also be used as evidence of whether a specific surgical procedure was performed correctly or incorrectly for other reasons, such as legal or compliance reasons.

[0218] Over time, as more and more surgical data is collected, a rich database may be acquired describing surgical procedures performed by different surgeons for different patients for various types of anatomies (knee, shoulder, hip, etc.). Moreover, aspects such as implant type and size, patient demographics, etc. can be further used to enhance the overall dataset. Once the dataset has been established, it can be used to train a machine learning model (e.g., RNN) to predict how the surgery will proceed based on the current state of CASS100.

[0219] Training of the machine learning model can be performed as follows. During surgery, the overall state of CASS100 can be sampled over multiple time periods. The machine learning model can then be trained to transform the current state of the first time period into a future state for a different time period. By analyzing the entire state of CASS100 rather than individual data items, any causal effects of interactions between different components of CASS100 can be obtained. In some embodiments, multiple machine learning models can be used instead of a single model. In some embodiments, the machine learning model can be trained not only with the state of CASS100, but also with patient data (e.g., obtained from EMR) and the identity of the surgical staff. This allows the model to make predictions with greater specificity. Moreover, if necessary, it allows surgeons to selectively make predictions based solely on their own surgical experience.

[0220] In some embodiments, the predictions or recommendations made by the aforementioned machine learning models can be integrated directly into the surgical process. For example, in some embodiments, the surgical computer 150 can execute a machine learning model in the background to make predictions or recommendations for upcoming actions or surgical conditions. Therefore, multiple states can be predicted or recommended for each period. For example, the surgical computer 150 can predict or recommend the state of the next 5 minutes in 30-second increments. Using this information, the surgeon can utilize a "process display" view of the surgery to allow visualization of future states. For example, Figure 11C-1 to Figure 11C-3 A series of images that can be displayed to the surgeon are shown, showing an implant placement interface. The surgeon can, for example, traverse these images by entering a specific time in the display 125 of CASS100 or instructing the system to use tactile, verbal or other instructions to advance or rewind the display in specific time increments. In one embodiment, the process display can be presented in the upper part of the surgeon's field of view in the AR HMD. In some embodiments, the process display can be updated in real time. For example, when the surgeon moves the resection tool around the planned resection area, the process display can be updated so that the surgeon can see how his or her actions affect other aspects of the operation.

[0221] In some embodiments, rather than simply using the current state of CASS100 as an input to a machine learning model, the input to the model may include a planned future state. For example, a surgeon may indicate that he or she is planning to perform a specific bone resection of the knee joint. The indication may be manually entered into the surgical computer 150, or the surgeon may provide the indication verbally. The surgical computer 150 may then generate a film showing the expected effect of the incision on the surgery. Such a film may show at a specific time increment how the surgery will be affected if the expected course of action is to be performed, including, for example, changes in the patient's anatomy, changes in implant position and orientation, and changes in related surgical procedures and instruments. A surgeon or medical professional may call or request this type of film at any time during the surgery to preview how the expected course of action will affect the surgical plan if the expected action is to be performed.

[0222] It should be further noted that, using a fully trained machine learning model and robotic CASS, various aspects of the surgery can be automated so that the surgeon only needs to be minimally involved, for example, only needing to provide approval for each step of the surgery. For example, over time, robotic control using arms or other means can be gradually integrated into the surgical process, and the manual interaction between the surgeon and the robotic operation gradually becomes less and less. In this case, the machine learning model can learn which robot commands are needed to achieve certain states of the CASS implementation plan. Ultimately, the machine learning model can be used to generate a film or similar view or display that can predict and preview the entire operation from an initial state. For example, an initial state including patient information, surgical plan, implant characteristics, and surgeon preferences can be defined. Based on this information, the surgeon can preview the entire operation to confirm that the plan recommended by CASS meets the surgeon's expectations and / or requirements. Moreover, since the output of the machine learning model is the state of CASS100 itself, commands can be derived to control the components of CASS to achieve each predicted state. Therefore, in extreme cases, the entire operation can be automated based only on the initial state information.

[0223] Preoperative planning using anatomical modeling software

[0224] In some embodiments, anatomical modeling software, such as LIFEMOD TM , can be used to develop preoperative or intraoperative plans to guide surgery. For example, in the context of hip surgery, if the anatomical modeling software has knowledge of the relationship between the spine and pelvis during multiple functional activities, the software can better predict the optimal implant position. Studies have shown that individuals with limited or abnormal spinal-pelvic mobility are at higher risk for misalignment. For these patients, surgeons recommend taking lateral radiographs in several positions (e.g., standing, sitting, flexed standing) in order to understand how the spine and pelvis interact during various activities. These images can be fed into a 3D biomechanical simulation to better predict the optimal implant position and orientation. In addition, as an alternative to the manual process of taking radiographs, anatomical modeling software can also be used to simulate the position of the lumbar spine and pelvis in a range of activities. In the context of knee surgery, if the anatomical modeling software understands the relationship between the mechanical axis of the joint, the condylar axis and the central axis of the femur and tibia, and the existing flexion and extension gaps, the software can better determine how changes in the size and posture (position and orientation) of the implant components can affect the mechanics of the replacement knee. More specifically, if the software incorporates the relationship between these variables throughout the range of motion and exemplary forces for a given patient activity, implant performance can be modeled.

[0225] Figure 12A-12CSome outputs of the anatomical modeling software are provided that can be used to visually depict the results of modeling hip joint motion using the anatomical modeling software. Fig. 12A and Fig. 12B A hip range of motion (ROM) graph is shown. In this case, anatomical modeling software can be used to perform a ROM "test" by placing the patient in various positions and modeling the movement of the hip joint as the patient's legs move in different positions. For example, the software can virtually simulate the position and orientation of the implant relative to the bone anatomy in various activities that the patient may experience postoperatively. These can include standard stability checks performed during a total hip procedure, or can even include activities that are high risk for impingement and dislocation (crossing legs when sitting, deep flexion when sitting, hyperextension when standing, etc.). After performing the test, the collected ROM data can be presented in a graphical user interface such as a video or video. Fig. 12A In addition, the anatomical modeling software can identify any impacted ROMs where there are abnormal contacts and wear contacts between the patient's anatomy and the implant components. Fig. 12B As shown in , after the unimpacted ROM is determined, it can be graphically overlaid on a 3D model of the patient's anatomy.

[0226] Fig. 12C A 2D graphic is shown demonstrating recommendations for a range of desired or "safe" positions for seating a hip implant in the acetabulum. In this case, anatomical modeling software may be used to identify a safe range of placement positions by modeling functional activity to failure points. For example, initially, the anatomical modeling software may assume that the implant may be placed anywhere within a large bounding box surrounding the anatomy of interest. Then, for each possible implant position, the anatomical modeling software may be used to test whether the position causes an anatomical or implant failure under normal functional activity. If so, the position is discarded. Once all possible points have been evaluated, the remaining points are deemed "safe" for implant placement. As Fig. 12B As shown in , for hip surgery, a safe position can be indicated by a graph of abduction relative to anteversion. In this case, the Lewinnek safety zone is superimposed on the graph. As is generally understood in the art, the Lewinnek safety zone is based on the clinical observation that if the acetabular cup is placed within 30-50 degrees of abduction and 5-25 degrees of anteversion, then dislocation is unlikely to occur. However, in this case, some of the patient's "safe" positions (depicted in "red") are outside the Lewinnek safety zone. Therefore, based on the characteristics of the patient's anatomical structure, the surgeon is given more flexibility to deviate from the standard recommendations.

[0227] The additional output of anatomical modeling software can be the 3D rendering of the final implant component displayed relative to bone.In certain embodiments, 3D rendering can be presented in the interface that allows the surgeon to rotate around the whole image and observe rendering from different viewing angles.This interface can allow the surgeon to articulate joints to visualize how implants will perform and identify the position where impact, misalignment, excessive strain or other problems may occur.The interface can include allowing the surgeon to hide some implant components or anatomical features so that the function of some areas of the best visual patient anatomical structure.For example, for knee implants, the interface can allow the surgeon to hide femoral components, and only display bearing surfaces and tibial components in visualization. Fig.12D An exemplary visualization is provided in the context of a hip implant.

[0228] In some embodiments, the anatomical modeling software can provide an animation that illustrates the position of the implant and bone when the patient performs different physical activities. For example, in the context of knee surgery, deep knee flexion can be presented. For hip surgery, the spine and pelvis can be shown during other activities that may represent a challenge to the implant, such as walking, sitting, standing, and the like.

[0229] Revision hip or knee replacement surgery involves removing one or more existing hip / knee implants and replacing the removed implants with new implants in a single surgical procedure. In some embodiments, the preoperative planning stage can anticipate the handling of bone spurs or other tasks other than the plane resection required to receive the new implant. For example, in some embodiments, for revision surgery, the output of the anatomical modeling software can be a 3D image or bone map showing the placement of each component and problem areas that may require special preparation by the surgeon. The software in the surgical plan can highlight the areas that will prevent the implant from being fully in place to show the surgeon which areas need to remove the bone. The software can be interactive, allowing the surgeon to virtually "expand" or "grind" the bone to best prepare the bone to receive the implant. This can be performed, for example, by allowing the surgeon / engineer to selectively remove individual pixels or groups of pixels from an X-ray image or 3D representation by touching the area or using the "virtual drill" component of the interface. In some embodiments, images in mixed modes can be registered and overlaid to provide views of the structures of anatomical interest areas, anatomical structures, etc. In some embodiments, the anatomical modeling software can provide recommended areas for removing bone burrs. For example, a machine learning model can be trained based on how a surgeon (or other surgeons) have performed deburring in the past to identify areas for deburring. Then, using an X-ray image or other patient measurements as input, the machine learning model can input recommended deburring areas for the surgeon to review during the virtual deburring procedure. In some embodiments, the virtual deburring process can be performed interactively with other aspects of anatomical modeling discussed above. This deburring process can also be used with primary joint replacements.

[0230] Using a point probe to obtain high resolution of critical areas during hip surgery

[0231] The use of a point probe is described in U.S. Patent Application No. 14 / 955,742, entitled "Systems and Methods for Planning and Performing Image Free Implant Revision Surgery," the entire contents of which are incorporated herein by reference. In short, an optically tracked point probe can be used to map the actual surface of a target bone that requires a new implant. Mapping is performed after a defective or worn implant is removed, and after any diseased or otherwise unwanted bone is removed. By brushing or scraping the entire remaining bone with the tip of the point probe, multiple points can be collected on the bone surface. This is called tracking or "mapping" the bone. The collected points are used to create a three-dimensional model or surface map of the bone surface in a computer planning system. The created 3D model of the remaining bone is then used as the basis for planning surgery and the necessary implant size. An alternative technique for determining a 3D model using X-rays is described in U.S. patent application Ser. No. 16 / 387,151, filed on April 17, 2019, and entitled “Three Dimensional Guide with Selective Bone Matching,” the entire contents of which are incorporated herein by reference.

[0232] For hip applications, point probe mapping can be used to obtain high-resolution data of key areas such as the acetabular rim and acetabular fossa. This can enable the surgeon to obtain a detailed view before starting reaming. For example, in one embodiment, a point probe can be used to identify the bottom (fossa) of the acetabulum. As is well known in the art, in hip surgery, it is important to ensure that the bottom of the acetabulum is not damaged during reaming to avoid damaging the medial wall. If the medial wall is inadvertently damaged, the surgery will require additional bone transplantation steps. With this in mind, during the surgical procedure, information from the point probe can be used to provide operational guidance for the acetabular reamer. For example, the acetabular reamer can be configured to provide tactile feedback to the surgeon when the surgeon reaches the bottom or otherwise deviates from the surgical plan. Alternatively, CASS100 can automatically stop the reamer when the bottom is reached or when the reamer is within a threshold distance.

[0233] As an additional safeguard, the thickness of the area between the acetabulum and the medial wall can be estimated. For example, once the acetabular rim and acetabular socket are mapped and registered to the preoperative 3D model, the thickness can be easily estimated by comparing the position of the acetabular surface to the position of the medial wall. Using this knowledge, CASS100 can provide an alarm or other response in the event of any surgical activity predicted to protrude through the acetabular wall when reaming.

[0234] The point probe can also be used to collect high-resolution data of common reference points used when orienting the 3D model to the patient. For example, for pelvic plane landmarks like the ASIS and pubic symphysis, the surgeon can use a point probe to draw the bones to represent the true pelvic plane. Knowing a more complete view of these landmarks, the registration software will have more information to orient the 3D model.

[0235] Point probes can also be used to collect high-resolution data describing proximal femoral reference points that can be used to improve the accuracy of implant placement. For example, the relationship between the tip of the greater trochanter (GT) and the center of the femoral head is often used as a reference point for aligning femoral components during hip replacement. Alignment is highly dependent on the correct position of the GT; therefore, in some embodiments, a point probe is used to draw the GT to provide a high-resolution view of the area. Similarly, in some embodiments, a high-resolution view with the lesser trochanter (LT) may be useful. For example, during hip replacement, the Dorr classification helps to select a stem that will maximize the ability to achieve a press fit during surgery, thereby preventing micromotion of the femoral component after surgery and ensuring optimal bone ingrowth. As is generally understood in the art, the Dorr classification measures the ratio between the tube width at the LT and the tube width 10 cm below the LT. The accuracy of the classification is highly dependent on the correct position of the relevant anatomical structure. Therefore, it may be advantageous to draw the LT to provide a high-resolution view of the area.

[0236] In some embodiments, a point probe is used to map the femoral neck to provide high-resolution data, allowing the surgeon to better understand where to make the neck incision. The navigation system can then guide the surgeon as they make the neck cut. For example, as understood in the art, the femoral neck angle is measured by placing a line below the center of the femoral stem and a second line below the center of the femoral neck. Therefore, a high-resolution view of the femoral neck (and possibly the femoral stem) will provide a more accurate calculation of the femoral neck angle.

[0237] High-resolution femoral head and neck data can also be used to navigate resurfacing procedures, where software / hardware helps the surgeon prepare the proximal femur and place the femoral component. As is generally understood in the art, during hip resurfacing, the femoral head and neck are not removed; rather, the head is trimmed and covered with a smooth metal covering. In this case, it would be advantageous for the surgeon to draw the femur and cap so that an accurate assessment of their respective geometries can be understood and used to guide trimming and placement of the femoral component.

[0238] Use point probes to register preoperative data to patient anatomy

[0239] As described above, in some embodiments, a 3D model is developed based on a 2D or 3D image of an anatomical region of interest during the preoperative phase. In such embodiments, registration between the 3D model and the surgical site is performed prior to the surgical procedure. The registered 3D model can be used to track and measure the patient's anatomical structures and surgical tools during surgery.

[0240] During the surgical procedure, landmarks are acquired to facilitate registration of the preoperative 3D model to the patient's anatomy. For knee surgery, these points may include the center of the femoral head, distal femoral axis, medial and lateral epicondyles, medial and lateral malleolus, proximal tibial mechanical axis, and tibial A / P orientation. For hip surgery, these points may include the anterior superior iliac spine (ASIS), pubic symphysis, points along the acetabular rim and within the hemisphere, greater trochanter (GT), and lesser trochanter (LT).

[0241] In revision surgery, the surgeon may draw certain areas containing anatomical defects in order to better visualize and navigate implant insertion. These defects can be identified based on analysis of preoperative images. For example, in one embodiment, each preoperative image is compared to an image library showing "healthy" anatomical structures (i.e., without defects). Any significant deviation between the patient image and the healthy image can be marked as a potential defect. Then, during surgery, the surgeon can be warned of possible defects by a visual alarm on the display 125 of CASS100. The surgeon can then draw areas to provide more detailed information about potential defects to the surgical computer 150.

[0242] In some embodiments, the surgeon can use a non-contact method to align the incisions within the bone anatomy. For example, in one embodiment, laser scanning is used for alignment. A laser bar is projected onto the anatomical region of interest, and changes in height of the region are detected as changes in the line. Other non-contact optical methods, such as white light interferometry or ultrasound, may also be used alternatively for surface height measurement or alignment of anatomical structures. For example, where there is soft tissue between the alignment point and the bone being aligned (e.g., ASIS, pubic symphysis in hip surgery), ultrasound technology may be beneficial, providing a more precise definition of the anatomical plane.

[0243] Surgical navigation with mixed reality visualization

[0244] In some embodiments, the surgical navigation system utilizes an augmented reality (AR) or mixed reality (MR) visualization system to further assist the surgeon during robotic-assisted surgery. Conventional surgical navigation can be augmented with AR by using graphical and information overlays (e.g., holographic or head-up display / HUD) to guide surgical performance. The exemplary system allows for the implementation of multiple headsets to share the same mixed or different reality experience in real time. In multi-user use cases, multiple user profiles may be implemented for selective AR display. This may allow the headsets to work together or independently, displaying a different subset of information to each user.

[0245] Embodiments utilizing AR / MR include surgical systems for operating in a surgical environment to enhance surgery through visual tracking, including a head-mounted display (HMD) worn by one or more individuals performing surgical functions. For example, a surgeon may have an HMD, and some or all nurses or laboratory technicians assisting the surgeon (or residents, other surgeons, etc.) may have their own HMD. By using the HMD, the surgeon can view information related to the surgery, including information traditionally associated with robotic surgery enhancement, without the surgeon having to move his or her field of view away from the patient. This can make surgery faster because the surgeon does not need to switch backgrounds between the display and the patient. In some embodiments, a virtual holographic monitor can be selectively presented to the surgeon, which reflects the display of a conventional LCD screen mounted on a cart during surgery. The HMD interface can allow the surgeon to move the holographic monitor to appear to be fixed in space at any position in the space of her choice, such as exposed patient tissue immediately in front of the surgical curtain.

[0246] In some embodiments, various types of HMDs can be used. Typically, the HMD includes a headgear and a communication interface worn on the user's head. The communication interface can be wired, such as USB, serial port, SATA or a proprietary communication interface, or preferably wireless, such as Wi-Fi or Bluetooth (however, timing and bandwidth limitations can limit the practical choice of some faster conventional communication interfaces, such as Wi-Fi or USB 3.0). The exemplary HMD also has a power supply (e.g., a battery or hard-wired power connector), an onboard computer (including a processor, GPU, RAM and non-volatile data and instruction memory), and one or more displays for superimposing information into the user's field of view. The exemplary HMD may also include a camera array (which may include optical sensors and IR sensors and lighting sources) that captures a 3-D image of the environment. In some embodiments, the HMD or an external processor can create a model of the user's environment using an image processing algorithm that identifies important features of the environment and creates a 3D model of the environment by processing stereo or IR data. By calibrating the HMD's display to the user's field of view, information displayed on a holographic display (e.g., using direct retinal or semi-reflective projection) can be reliably superimposed on the user's field of view to enhance the user's observation of the environment.

[0247] The HMD worn by the surgical staff may include a commercially available off-the-shelf HMD, such as the Oculus Rift TM , Microsoft HoloLens TM , Google Glass TM 、Magic Leap One TM Or custom designed hardware for surgical environments. In some embodiments, supplemental hardware is added to commercially available HMDs to enhance their use in surgical environments using off-the-shelf HMD components and custom hardware. In some embodiments, the HMD hardware can be integrated into traditional surgical masks and face shields, allowing the HMD to act as a personal protective device and allowing information to be displayed by reflecting light from the mask that the surgeon is already familiar with wearing. There are many ways that HMD technology on the market can provide users with a mixed reality environment. For example, virtual reality headsets such as the Oculus Rift TM 、HTC Vive TM 、Sony PlayStation VR TM or Samsung Gear VR TM, blocking the user's natural vision and replacing the user's entire field of view with a stereoscopic screen, thus creating a 3D environment. These systems can use one or more cameras to recreate an augmented form of the three-dimensional environment to display to the user. This allows the natural environment to be captured and re-displayed to the user with mixed reality components. Other AR headsets, such as Google Glass TM and Microsoft HoloLens TM , augments reality by providing supplementary information to the user, which is displayed as a hologram within the user's field of view. Since the user observes the environment directly or through a clear lens, the additional displayed information is semi-transparent to the user, either projected from a reflective surface in front of the user's eyes or directly onto the user's retina.

[0248] Commercially available HMDs typically include one or more outward-facing cameras to collect information from the environment. These cameras may include visible light cameras and IR cameras. The HMD may include a lighting source to assist the camera in collecting data from the environment. Most HMDs include some form of user interface that a user can use to interact with the processor via the display of the HMD. In some systems, the user interface may be a handheld remote controller that is used to select and engage displayed menus. This may not be ideal for a surgical environment due to sterilization issues and fluid-covered hands. Other systems, such as the Microsoft HoloLens TM and Google Glass TM , using one or more cameras to track gestures or MEMS accelerometers to detect movement of the user's head. The user can then use gestures to interact with the virtual interface. For example, a virtual keyboard can be displayed holographically, and the camera can track the user's finger movements to allow the user to type on the virtual floating keyboard. Some HMDs may also have a voice interface. In addition to the display, the head mount may provide tactile feedback through actuators or provide audio signals to the user through the head mount or speakers. Other onboard sensors of the HMD may include a gyroscope, a magnetometer, a laser, or an optical proximity sensor. In some embodiments, the HMD may also have a laser or other projection device that allows information to be projected into the environment rather than onto the user in a holographic manner.

[0249] While AR headsets can provide a more natural feel to the user than VR because much of the imagery the user sees is natural, it can be difficult to properly overlay and align the displayed information with the user's viewpoint. There are already a number of software initiatives in the industry that address this issue, which have been pioneered by AR headset manufacturers. As a result, AR and VR headsets typically come with the software tools necessary to overlay information onto the user's field of view so that the information is aligned with what the user sees in the environment.

[0250] In some embodiments, information similar to that displayed on a conventional cart-mounted display is provided to the surgeon as part of a robotic-assisted surgical system, such as the NAVIO Surgical System. In some embodiments, different HMDs worn by different people in the operating room may display different information at any time. For example, the surgeon may see information related to what is in his current field of view, while the HMD worn by the surgical resident may display what the attending surgeon sees and any enhanced camera views seen by the attending surgeon. In some embodiments, the resident may see additional patient information that may be helpful to the resident in learning or conveying to the surgeon, such as preoperative imaging, patient files, information from tool or medical device manufacturers, etc.

[0251] The HMD includes one or more cameras that capture the wearer's field of view (or a wider or narrower version thereof) and provide images to a processor, thereby allowing the processor to parse the two-dimensional image captured by the HMD and its relationship to the 3D model of the operating room. For example, one or more tracking cameras of the HMD can capture the presence of fiducial markers of the patient's bones and / or tools to determine how the wearer's angles relate to the 3D models of the patient and / or tools. This can allow the image processor to extract features, such as bones / tissues and tools, from the captured images and use this information to display enhanced information on the image that the wearer is viewing. For example, when a surgeon is performing surgery on a knee, a camera on the surgeon's HMD can capture what the surgeon sees, allowing the image processing software to determine where the surgeon is looking in three-dimensional space and determine the specific patient features that the surgeon is looking for. These specific patient features can be located in the two-dimensional image by the image processing software using pattern matching or machine learning techniques, which are told where the surgeon is looking in the surgical scene.

[0252] For example, the surgeon can observe the tibial plateau and femoral condyle. The camera on the HMD will capture the image in real time (including actual processing and communication delays) and send the image to the image processing software. The image processing software can identify the observed object as the tibial plateau and femoral condyle (or can receive prompts based on the viewing angle), and look for patterns in the image to identify the range of these features in the surgeon's field of view. Then, the information about the tibial plateau and femoral condyle can be overlaid with the image seen by the surgeon. If a fiducial marker is available in the image (or the most recent image), or the surgeon's HMD includes a fiducial marker captured by a robot vision system or by a camera of another HMD in the room, to allow the surgeon's HMD camera to be calculated, the software can accurately position the two-dimensional image relative to the three-dimensional model of the surgical scene.

[0253] In some embodiments, information is holographically overlaid in the user's field of view. In some embodiments, information can be digitally projected from the HMD into the environment. For example, a laser array MEMS mirror coupled to the headgear can project an image directly onto a surface in the environment. Because such a projection comes from approximately the same position as the user's eyes, such a projection can be easily juxtaposed with the user's field of view, overlaying information onto the environment in a more robust manner than presenting a floating hologram to the user. For example, Fig.13 As shown in , the HMD 1100 projects a computer-generated image of a cutting envelope onto a portion of the patient's knee 1102 to indicate to the wearer exactly where she should cut on the bone in the knee without distracting the wearer or interfering with her peripheral vision.

[0254] Surgical systems that use optical tracking modalities, such as the NAVIO system, are well suited for use with HMDs. Including one or more cameras in the HMD makes it particularly convenient to adapt the HMD for use in an operating room. Fig.14 As shown in , one or more cameras 1110 mounted to a cart or fixed in a surgical environment use optical and IR tracking to capture the positions of the fiducial markers 1112, 1114 mounted to the tool and the patient's bone. Adding one or more HMDs 1100 to this environment can supplement this tracking system by providing additional perspectives for optical or IR tracking. In some embodiments, multiple HMDs 1100 can be used simultaneously in an operating room, thereby providing a variety of perspectives to help track the fiducial markers of the tool and the patient's anatomical structure. In some embodiments, the use of multiple HMDs provides a more robust tracking mode because additional perspectives such as comparison, weighting, association, etc. can be used in software to verify and refine the 3D model of the environment. In some embodiments, cameras permanently installed in the operating room (mounted to a wall, cart, or surgical light) can utilize higher quality IR components and optical components than those in the HMD. When there is a gap in the quality of the components, the 3D model refinement using multiple perspectives can assign heuristic weights to different camera sources, allowing refinement based on higher quality components or more reliable perspectives.

[0255] Given the rapid development of optical sensor technology driven by the mobile device market, HMD optical sensor technology is developing rapidly. In some embodiments, a camera array mounted on a cart is unnecessary for an optical tracking modality. Optical sensors and IR sensors on HMDs worn by surgeons, residents, and nurses can provide sufficient viewing angles to track fiducial markers on patients and tools without the need for a separate cart. This can reduce the cost of adding an HMD to a conventional tracking modality or reduce overall system cost. The price of HMDs with existing components is rapidly declining as they are accepted by the consumer market. In some embodiments, a camera mounted on a cart or a wall-mounted camera can be added to a system that uses optical sensors and IR sensors of lower quality than traditional cart-mounted tracking systems to supplement embodiments that otherwise rely entirely on IR sensors and optical sensors of the HMD. (Optical sensors include IR sensors / cameras and optical sensors / cameras, and may be generally described as cameras, but for clarity, these components may be listed separately; embodiments may include any subset of available optical sensors.)

[0256] like Fig.14 As shown in , each camera array (of the tracking system 1110 mounted on the cart and each HMD 1100) has a camera pose that defines the reference frame of the camera system. In the case of cart-mounted or wall-mounted cameras, the camera pose may be fixed throughout the operation. In the case of HMD cameras, the camera pose will typically change during surgery as the user moves their head. Therefore, in some embodiments where the HMD works with a cart-mounted or wall-mounted tracking system to supplement the tracking information, fiducial markers are used to at least identify the location of the other camera systems in the environment. In some embodiments, fiducial markers may be rigidly applied to these camera systems so that the other camera systems can identify the location and pose of each other camera system. This can be used to calculate the camera viewing angle to determine a robust 3D model of the surgical environment. It should be understood that when the user moves in the HMD, the field of view may ignore the other camera systems, but the fiducial markers on the system allow the camera system to quickly identify the other cameras and their poses after moving back into the field of view.

[0257] Once a camera has identified the position and / or pose of other cameras in the environment, the camera can identify the position and orientation of fiducial markers affixed to the patient's bones or tools. When two cameras have the same fiducial markers in their field of view, a central processor or peer process can correlate the position and orientation of those markers relative to each camera to produce a model of the 3D environment that includes the position and pose of each camera and the position and pose of each operable bone of the patient. Fig.14In the example shown, the fiducial markers 1112 and 1114 mounted to each patient's bone include four reflective points of known geometry. Each of these fiducial markers 1112 and 1114 has a pose defined in three-dimensional space. Since this geometry is known, this pose can be associated through a registration process to define a reference frame for each bone that is a transfer function of the pose of the corresponding fiducial marker. Therefore, when the pose can be determined from the camera information by the processor, the pose of each bone 1116 and 1118 relative to the field of view of each camera can be calculated. In this example, the headgear 1100 and the tracking system 1110 mounted on the cart wirelessly communicate with each other or with a central processor.

[0258] The robustness of the 3D model is improved by the number of fiducial markers observed by the multiple cameras. Because cameras capture the analog world using digital signals and are limited by the quality of the optical components, the accuracy with which each camera can locate a fiducial marker in space includes a certain degree of error. Using multiple camera systems or HMDs can reduce this error, resulting in a more robust 3D model with a level of accuracy that cannot be achieved with a single cart-mounted tracking model. Any method known in the art for computing a 3D model of an environment from pose information captured by multiple cameras can be applied to a multi-HMD operating room environment.

[0259] Fig.15An exemplary method 1200 for using an AR headset in a surgical environment using fiducial markers on patients and tools is shown. Each camera system in the operating room may include a camera on the HMD and a fixed camera, such as a camera mounted to a cart or wall, which can perform steps 1202 to 1212. Each camera system can be initialized at step 1202. This may include any startup procedures and calibrations required to prepare the camera for operation during surgery. At step 1204, the camera system captures images and applies any techniques to prepare these images for processing, including eliminating distortion or any image preprocessing. At step 1206, each camera system can optionally estimate its own pose. This may include a gyroscope sensor, an accelerometer sensor, a magnetic sensor, or a compass and a model of a previous pose updated based on image information. At step 1208, each camera system attempts to identify any other camera systems in its field of view. This can be used to determine the geometric relationship between various camera systems. In some embodiments, this recognition step looks for certain markers in the visible field that may include fiducial markers placed on cameras or IR beacons, etc. At step 1210, each camera system identifies a fiducial marker in the captured image. The fiducial marker may be a physical marker placed on or fixed to a patient's anatomy, a tool, another camera, or a landmark in the environment. A processor associated with each camera system may determine pose information based on these fiducial markers. This pose information may include the pose of the camera system and / or the pose of the object in the environment. For example, a tool may have multiple IR / UV reflective or unique color or patterned markers placed thereon so that the position and orientation of the tool can be identified from one or more images. Similarly, multiple fiducial markers may be placed on a camera to allow the position and orientation of the camera to be calculated by other camera systems. At step 1212, this information about the position and orientation of other objects in the environment may be reported to a central processor of the management system wirelessly or through a hard-wired network. The above steps (1204-1212) may be repeated as each camera system continuously captures images and prepares them for analysis.

[0260] At step 1214, the central processor receives image data and pose or fiducial marker information from each camera system. At step 1216, the central processor creates or updates models of all camera systems in the environment and their poses relative to the environmental reference system. At step 1218, the central processor identifies the fiducial markers in the received image to associate the markers captured by multiple camera systems. By capturing such markers in multiple fields of view, different perspectives of the object can be used to optimize the determination of the position and orientation of the object in the environment. Once related, at step 1220, the central processor can calculate the position and orientation of each object with a fiducial marker, such as a tool, environmental marker, other HMD or camera system, and patient anatomical structure. At step 1222, this information is used to update the 3D model of the environment, calculate the position and orientation of the object relative to the fixed reference system, and identify the pose of the reference system defined by each object. For example, the patient's femur has a given pose based on the way the patient is lying in the operating room. The femur also has its own reference system, which can be used to identify the part of the femur that may need to be removed during surgery when the preoperative imaging is associated with the structure of the patient's femur. At step 1224, the central processor can use the updated 3D model to send information about the environment to the surgical robotic system and any HMDs in the operating room. The central processor continues to receive images and update the environment model. In some embodiments, this process can be enhanced for any object in the operating room via other sensors, such as accelerometers, compasses, etc. Once this 3D model is generated, the central processor is free to interact with the various HMDs according to the many software applications described herein.

[0261] Fig.161300 is a system diagram of an augmented reality system 1300 used during surgery. In this example, fiducial markers 1302A-G are placed on each HMD and camera system and the patient's femur and tibia. In this example, a partial or total knee replacement is being performed. A camera system 1110 (e.g., a camera system that can be used for a NAVIO surgical system) mounted on a cart is placed to have a view of the surgical scene. Fiducial markers 1302E-F with multiple fiducial markers (e.g., three or more spherical IR reflective markers) are temporarily mechanically fixed to the patient's femur and tibia. This allows the camera system 1110 mounted on the cart to track the posture and movement of the tibia and femur. This can be used to determine the pivot center between these bones, as well as other information about the patient's anatomical structure during movement. The surgical robot 1306 can also use this information to determine the ideal placement of the cut and replacement knee portion. The camera system 1110 mounted on the cart communicates via a local area network (e.g., a secure Wi-Fi network) with a central processor 1304 that includes software and memory sufficient to process the images from the camera mounted on the cart to determine a model of the environment and compute any information that may be useful to the surgical robot 1306 to assist the surgeon in performing the procedure. In some embodiments, the central processor 1304 is part of the cart that supports the camera system. In these embodiments, the camera system 1110 mounted on the cart can communicate directly with the central processor 1304. The central processor 1304 then communicates with any robotic system 1306 used during the procedure.

[0262] In addition to this conventional robotic surgery system, the physician and house staff wear multiple HMDs 1100, 1100A, and 1100B during the procedure. Each HMD has an identifier that allows the HMD to be identified in the visual plane of the camera system 1110 mounted on the cart. In some embodiments, the identifier includes an IR transmitter that transmits a binary modulated code that identifies a particular HMD or wearer.

[0263] In this embodiment, each HMD 1100-1100B has a camera array (IR and / or visible spectrum) that captures the user's field of view (or a sub-portion thereof or a wider angle) from multiple positions, thereby producing a stereoscopic (or higher order) view of the scene for image processing. This allows each headpiece to capture image data sufficient to provide three-dimensional information about the scene. Each HMD captures a different foreground of the surgical scene. Fig.15As explained in , the camera array of each HMD captures an image of the scene, estimates the current pose of the HMD (which can also be estimated by non-optical sensors), and identifies other cameras and any fiducial markers in the field of view. In some embodiments, the HMD has an IR transmitter to illuminate reflective fiducial markers. These captured images can then be processed on the HMD or sent to a central processor for more powerful image processing. In some embodiments, a pre-processing step is performed on each HMD to identify fiducial markers and other cameras and estimate the pose of the fiducial markers and other cameras. Each HMD can communicate with other HMDs or with a central processor 1304 via a local network 1310, such as a secure Wi-Fi network. The central processor 1304 receives image information from each HMD and further optimizes the 3D model of the surgical space.

[0264] Based on the images captured from each HMD, the central processor 1304 can determine which objects in the three-dimensional model of the operating room correspond to reference features in the two-dimensional image plane of each HMD. This information can then be transmitted back to the HMD via the network 1310. In some embodiments, each HMD periodically receives information about the three-dimensional model, but uses on-board image processing to track objects based on the three-dimensional model information received from the central processor. This allows the headset to accurately track objects within the field of view in real time without delays caused by network communications.

[0265] In some embodiments, the HMD 1100 receives information about the object from the central processor 1304 and processes the current and recent images to identify salient features of the object corresponding to the three-dimensional model information received from the central processor. The HMD 1100 uses local image processing to track those features (and associated objects) in the visual plane. This allows the HMD 1100 to overlay reference pixel locations with information related to the object.

[0266] For example, the surgeon's HMD 1100 can capture reference markers 1302E and 1302F related to the tibia and femur. Based on the three-dimensional model of the tibia and femur that is identified by communication with the central processor 1304, the surgeon's HMD 1100 can identify which features in the two-dimensional image correspond to the features of the tibial plateau and the femoral condyle. By tracking the reference markers on these bones or by tracking the image features identified as the tibial plateau or the femoral condyle, the surgeon's HMD 1100 can track these features of the bones in real time (taking into account any processing and memory delays) without having to worry about network delays. If the software running on the HMD 1100 wishes to overlay a visual indication of the cutting position on the tibial plateau, the HMD can track the tibial plateau when the surgeon's head moves or the patient's leg moves. The surgeon's HMD 1100 can accurately estimate where the reference point of the bone feature is and where to display the enhanced holographic feature.

[0267] In some embodiments, HMDs 1100-1100B can communicate directly to assist each other in updating any location models. Using peer-to-peer communication, HMDs 1100-1100B can expose their environment models to other HMDs, allowing them to correct their own models using arbitration rules.

[0268] In some embodiments, objects within the surgical space may include QR codes or other visual indications that can be captured by a camera on the HMD 1100 to convey information about the object directly to the HMD. For example, a surgical tray 1308 with tools for a given surgery may have a QR code indicating the identity of the tray. By consulting a database, the HMD 1100 that captures the QR code can accurately identify the objects and surgical tools on the tray before the surgery begins. This can allow the HMD 1100 to identify important objects related to the surgery, such as various knives, cutting tools, sponges, or implantable devices. The HMD 1100 can then track these objects within the surgical space while taking into account the last known location of those objects. In some embodiments, HMDs 1100-1100B can share information about the last known location of objects within the operating room to enable each head-mounted piece to more quickly or easily identify objects that enter the field of view of each HMD. This can help surgeons quickly determine the tools or objects to be grabbed in the next step of the surgery. For example, a tray may include a series of cutting guides for various surgeries. However, each patient may only need to use a single cutting guide on his femur. The HMD 1100 can identify the cutting guides in the tray based on the QR code and initial layout of the tray and track individual cutting guides. The holographic display of the HMD 1100 can overlay instructions to the surgeon for the cutting guide to use for a given step in the procedure.

[0269] In various embodiments, various information may be displayed to a user of the HMD. Fig.17A , the user is presented with a holographic overlay 1315 of the resection area on the femoral condyle to indicate where the surgeon should remove tissue for placement of the replacement knee joint portion that will be bonded and secured to the femoral head. Fig.17A The view that the surgeon sees includes the natural scene and superimposed shapes placed holographically on the surface of the bone. In some embodiments, rather than using holograms as part of a conventional AR display, a laser or projector mounted to the HMD can project this information directly onto the surface of the bone. A camera on the HMD can provide a feedback loop to ensure that the protrusions are placed appropriately and consistently on the bone.

[0270] exist Fig. 17B, an exemplary display of a three-dimensional model is shown. Conventionally, this display may appear on a computer screen of a robotic surgical system. It indicates on the three-dimensional model exactly where the surgeon's tools should go and where the tools should cut the bone. This information may be adapted for an AR display, whereby a portion of the three-dimensional model of the bone is holographically overlaid on the bone, allowing the resection area to be displayed to a user viewing the real-world bone. In some embodiments, only the three-dimensional model of the resection area is displayed, while the three-dimensional model of the bone is compared to the field of view of the HMD to ensure that the holographically displayed three-dimensional model of the resection area is correctly placed. Additionally, Fig. 17B Any menus or additional information shown in the AR user interface may be displayed as holographic menus to the user of the HMD, allowing the user to select certain user interface menus through the standard AR user interface to learn more information or view.

[0271] Fig.18A Additional three-dimensional model views that may be displayed to the user are shown. Fig.18A Depicts the points identified by the point probe during femoral head mapping. Such information can be displayed to the surgeon via a holographic image of the portion of the bone that has been detected to identify the shape of the femoral head. In some embodiments, this information can be displayed on the resident's HMD. Fig.18A The images shown in can be displayed on a conventional computer display in the operating room. Individual parts of the display, such as a three-dimensional model of the femoral head, can be displayed holographically onto the bone using an AR display. Again, any of these menus shown can be displayed to the user's head-mounted device, or can be displayed on a two-dimensional LCD type display on a cart in the room. These menus can be selected by the user of the HMD to change the view or obtain additional information. The user of the AR HMD can select menus using any conventional AR selection means, such as an air click or head / hand gesture that can be detected by a sensor or camera in the HMD. In some embodiments, where a robotic cutting tool is used, this display can indicate where the robot is to cut, allowing the surgeon to visually confirm before cutting begins.

[0272] Fig.18B Another 3D model of the femur is shown, illustrating the proper placement of a femoral cutting guide. This can be displayed to the surgeon on a hologram to indicate exactly where the femoral cutting guide should be placed. During alignment of the cutting guide, the surgeon can consult this holographic overlay to ensure that the placement is approximately correct. A robotic vision system in the room can provide final confirmation before installing the cutting guide. In hip surgery, this same technology can be used to help the surgeon place a customized femoral neck cutting guide or an acetabular jig to aid in the placement of a cup. In some embodiments, Fig.18B The display can be obtained from Fig.18AA menu selection in allows the surgeon to switch between a history of steps and a model of the next steps that can be completed. This can allow the user to effectively rewind and fast forward the procedure, seeing future steps to be performed and steps that have already been performed.

[0273] In some embodiments, the user may also select and modify a proposed cutting angle, thereby allowing the processor to calculate how changes in the cutting angle may affect the geometry of the replacement knee using an anatomical model, such as the LIFEMOD manufactured by SMITH AND NEPHEW, INC. TM The information displayed can include the static and dynamic forces that will be generated on the patient's ligaments and tissues if the cutting geometry is altered. This can allow the surgeon to modify the replacement knee procedure on the fly to ensure proper placement of the replacement knee components during the procedure to optimize the patient's outcome.

[0274] Fig.19 A three-dimensional model of a fully replaced knee system including a model of ligament and tendon forces is shown. If the surgeon wishes to see how cutting decisions during surgery affect the hinge geometry and the stresses on the ligaments and tendons, a portion of the model can be holographically displayed to the surgeon during surgery. Exemplary modeling of implant placement and other parameters for arthroplasty procedures is described in U.S. Patent Application No. 13 / 814,531, which is previously incorporated herein by reference. The surgeon can change the parameters of the arthroplasty via an AR interface to generate hypothetical results that can be displayed by a headgear, such as whether parameter changes during total knee arthroplasty (TKA) cause the patient's gait to become more varus or valgus, or how the patella is tracked. Optimized parameters can also be modeled and the procedure displayed to the surgeon holographically.

[0275] In some embodiments, the wearer of the HMD can selectively request the display of patient medical history information, including preoperative scans of patient tissue. In some embodiments, the display of the scan can be holographically overlaid onto existing patient tissue observed in the scene by aligning imaged features with features found in the patient. This can be used to guide the surgeon in determining appropriate resection areas during the procedure.

[0276] In some embodiments, a video of the three-dimensional data model can be recorded, logged, and time-stamped. Playback of the video can be performed after the procedure, or the video can be called up on the HMD display during the procedure to view one or more steps. This can be a valuable teaching tool for residents or surgeons who want to see when a certain cut or step is performed. Playback can be a useful tool for creating surgical plan changes during the procedure based on events during the procedure. The head or hand gestures of the HMD operator can rewind or advance the virtual viewing of the video or 3-D model information.

[0277] Example use cases

[0278] By adding AR to the operating room using an HMD, many improvements to surgical workflow in various surgeries can be achieved. The following are some examples of ways AR can be used in various surgical procedures.

[0279] The software can utilize the HMD camera and display to determine (with the assistance of the preoperative plan and processor) the ideal starting location for the incision. Lines defining the range of incision locations can be displayed to the surgeon observing the patient via a holographic overlay (or by projecting directly onto the patient's skin). The exact location can take into account the specific patient anatomy and intraoperative point registration, where the user more accurately registers the patient geometry to the system.

[0280] Soft tissue dissection can utilize the HMD's built-in camera and a model of the patient to highlight certain muscle groups, ligaments of a joint, such as the hip capsule or knee joint, nerves, vascular structures, or other soft tissues to help the surgeon reach the hip or knee joint during dissection. The augmented display can show the surgeon an indication of the location of these soft tissues, such as by displaying a holographic 3D model or preoperative imaging.

[0281] During hip replacement or repair, the software can overlay a line indicating an ideal neck cut on the proximal femur based on the preoperative plan. This can define the precise position where the bone is cut to place the femoral head prosthesis that replaces it. Similar to the above example, during acetabular re-reaming, the head-up display can show the amount of bone that needs to be removed by covering different colors on the patient's acetabulum. This allows the surgeon to know when he / she is approaching the bottom surface of the acetabulum. In some embodiments, the range of the resection area for reaming the acetabulum can be superimposed on the patient's bone. For example, green, yellow and red color indications can indicate the depth of reaming relative to the predetermined resection area to the surgeon (based on the position of his / her tool). This provides a simple indication to avoid removing too much bone during surgery. In some embodiments, the head-up display can overlay an image of a reamer handle (or other tool) to indicate the appropriate tilt and anteversion from the preoperative plan (or from the surgeon's input). It can also display the actual tilt / anteversion of the reamer / tool ​​handle to allow the surgeon to correct its approach angle.

[0282] During the cup impaction step for hip replacement, similar to reaming, inclination and anteversion values ​​can be displayed on the heads-up unit along with the values ​​determined from preoperative planning. A superimposed image of the cup impactor (or long axis) can be displayed to assist the surgeon in positioning the implant. The HMD can also display an indication of how close the cup is to being fully seated. For example, superimposed cup or impactor measurements or color changes can be used to indicate whether the cup is fully seated, or a model can be overlaid that highlights the final ideal position of the cup that differs from what the surgeon currently sees.

[0283] The HMD can highlight the screw hole that should be used to secure any hardware to the bone. When using a drill and drill guide, the heads-up display can overlay the "ideal" axis for screw insertion to help position the screw within the screw hole.

[0284] During femoral canal preparation, as the surgeon inserts the broach into the femoral canal, the HMD can superimpose an image of the broach in the proper orientation (corresponding to the preoperative plan). Additionally, instructions can be superimposed onto the scene to provide the surgeon with information regarding the final broach setting. This can be presented by changing the color around the broach or by giving the surgeon a percentage or number indicating whether the broach is fully seated and whether the size is the correct "final" component size.

[0285] The surgeon performing the trial reduction can be provided with the option of displaying the combination of leg length and offset increase based on the implant set being used (e.g., in the form of a chart). For example, the chart can list the combination of leg length and offset changes for each implant combination. Alternatively, there can be an option to superimpose the proposed / changed components on the patient's anatomical structure to show the surgeon what the resulting implant positioning will be if the neck offset / femoral head length is changed (e.g., from STD to high offset neck, +0 to +4 femoral head). The surgeon can select the appropriate implant from the first trial and perform the implantation step. Multiple trial steps, i.e., conventional standard of care, may be unnecessary.

[0286] Resurfacing techniques can also be improved through AR. When performing a resurfacing procedure, the HMD can superimpose an axis indicating the ideal position and orientation of the guidewire. In some embodiments, the software allows the surgeon to adjust this axis, and the HMD can superimpose a cross-section or other view of the femoral neck to show the surgeon how thick the bone will be when inserting the implant in a certain position (one of the most common complications from resurfacing surgery is inserting a component in varus, which can cause a femoral neck fracture). Giving the surgeon the ability to adjust this axis can enable optimization of the performance of the implant in the body.

[0287] In some embodiments, this traditional femoral resurfacing technique can be replaced by a burr-only technique. In this exemplary technique, the surgeon prepares the proximal femur entirely by drilling. A bone map can be superimposed on the bone to indicate how much bone remains to be removed. This can also be indicated by color on the map. A variety of cutting instruments can be provided to the surgeon to reduce the total amount of time required to cut the bone into the desired shape.

[0288] Any suitable tracking system can be used to track surgical objects and patient anatomy in the operating room. For example, a combination of infrared and visible light cameras can be used in an array. Various illumination sources (e.g., infrared LED light sources) can illuminate the scene so that three-dimensional imaging can be performed. In some embodiments, this can include stereoscopic, three-view, four-view, etc. imaging. In addition to the camera array fixed to the cart in some embodiments, additional cameras can be placed throughout the operating room. For example, a handheld tool or headgear worn by the operator / surgeon can include imaging capabilities that can transmit images back to a central processor to correlate those images with images acquired by the camera array. This can provide more robust images for environments modeled using multiple perspectives. In addition, some imaging devices can have a suitable resolution on the scene or have a suitable perspective to pick up information stored in a QR code or barcode. This helps to identify specific objects that are not manually registered with the system.

[0289] In some embodiments, the surgeon can manually register specific objects with the system before or during surgery. For example, by interacting with the user interface, the surgeon can identify the starting position of a tool or bone structure. By tracking fiducial markers associated with the tool or bone structure, or by using other conventional image tracking methods, the processor can track the tool or bone as it moves through the environment in the three-dimensional model.

[0290] In some embodiments, certain markers, such as fiducial markers that identify individuals, important tools, or bones in the operating room, may include passive or active identification that can be picked up by a camera or camera array associated with a tracking system. For example, an infrared LED may flash a pattern that conveys a unique identification to the source of the pattern, thereby providing a dynamic identification marker. Similarly, one-dimensional or two-dimensional optical codes (barcodes, QR codes, etc.) may be fixed to objects in the operating room to provide passive identification that can occur based on image analysis. If these codes are placed asymmetrically on the object, they can also be used to determine the orientation of the object by comparing the position of the identifier with the range of the object in the image. For example, a QR code may be placed in the corner of a tool tray, allowing the orientation and identification of the tray to be tracked. Other tracking methods will be described throughout the text. For example, in some embodiments, surgeons and other personnel may wear augmented reality headsets to provide additional camera angles and tracking capabilities.

[0291] In addition to optical tracking, certain features of an object can be tracked by aligning the physical properties of the object and associating it with an object that can be tracked (e.g., a fiducial marker fixed to a tool or bone). For example, a surgeon can perform a manual alignment process whereby the tracked tool and the tracked bone can be manipulated relative to each other. By striking the tip of the tool against the surface of the bone, a three-dimensional surface can be plotted for the bone, the three-dimensional surface being associated with the position and orientation of a reference frame relative to the fiducial marker. By optically tracking the position and orientation (pose) of the fiducial marker associated with the bone, a model of the surface can be tracked in the environment by extrapolation.

[0292] Fig. 20 An example of a parallel processing platform 2000 that can be used to implement a surgical computer 150, a surgical data server 180, or other computing system used in accordance with the present invention is provided. The platform 2000 can be used, for example, in embodiments where machine learning and other processing-intensive operations benefit from parallel processing tasks. The platform 2000 can be used, for example, with NVIDIA CUDA TM or similar parallel computing platform implementation. The architecture includes a host computing unit ("host") 2005 and a graphics processing unit (GPU) device ("device") 2010 connected via a bus 2015 (e.g., a PCIe bus). The host 2005 includes a central processing unit or "CPU" ( Fig. 20 2010 includes a graphics processing unit (GPU) and its associated memory 2020, which is referred to herein as device memory. Device memory 2020 may include various types of memory, each optimized for different memory usage. For example, in some embodiments, device memory includes global memory, constant memory, and texture memory.

[0293] The parallel portions of the big data platform and / or the big simulation platform can be executed on the platform 2000 as "device kernels" or simply as "kernels". Kernels include parameterized code configured to perform specific functions. The parallel computing platform is configured to execute these kernels on the platform 2000 in an optimal manner based on parameters, settings, and other selections provided by the user. In addition, in some embodiments, the parallel computing platform may include additional functionality to allow kernels to be automatically processed in an optimal manner with minimal input provided by the user.

[0294] The processing required by each kernel is performed by a grid of thread blocks (described in more detail below). Using concurrent kernel execution, streaming, and synchronization with lightweight events, the platform 2000 (or a similar architecture) can be used to parallelize portions of machine learning-based operations performed in training or to utilize the intelligent editing processes discussed herein. For example, the parallel processing platform 2000 can be used to execute multiple instances of a machine learning model in parallel.

[0295] The device 2010 includes one or more thread blocks 2030 representing the computing units of the device 2010. The term thread block refers to a group of threads that can cooperate and synchronize their execution through shared memory to coordinate memory access. For example, threads 2040, 2045 and 2050 operate in thread block 2030 and access shared memory 2035. Depending on the parallel computing platform used, the thread blocks can be organized in a grid structure. Then the calculation or a series of calculations can be mapped to this grid. For example, in an embodiment using CUDA, the calculation can be mapped on a one-dimensional, two-dimensional or three-dimensional grid. Each grid contains multiple thread blocks, and each thread block contains multiple threads. For example, thread block 2030 can be organized in a two-dimensional grid structure of m+1 rows and n+1 columns. Typically, threads in different thread blocks of the same grid cannot communicate or synchronize with each other. However, thread blocks in the same grid can run on the same multiprocessor in the GPU at the same time. The number of threads in each thread block may be limited by hardware or software constraints.

[0296] Continue to refer Fig. 20 , registers 2055, 2060, and 2065 represent fast memory available to thread block 2030. Each register can only be accessed by a single thread. Therefore, for example, register 2055 can only be accessed by thread 2040. In contrast, shared memory is allocated per thread block, so all threads in the block can access the same shared memory. Therefore, shared memory 2035 is designed to be accessed in parallel by each thread 2040, 2045, and 2050 in thread block 2030. Threads can access data in shared memory 2035 loaded from device memory 2020 by other threads in the same thread block (e.g., thread block 2030). Device memory 2020 is accessed by all blocks of the grid and can be implemented using, for example, dynamic random access memory (DRAM).

[0297] Each thread can have one or more memory access levels. For example, in platform 2000, each thread can have three memory access levels. First, each thread 2040, 2045, 2050 can read and write its corresponding registers 2055, 2060 and 2065. Registers provide the fastest memory access to threads because there is no synchronization problem, and registers are usually located near the multiprocessor that executes the thread. Secondly, each thread 2040, 2045, 2050 in thread block 2030 can read and write data to the shared memory 2035 corresponding to the block 2030. Usually, due to the need to synchronize access between all threads in the thread block, the time required for the thread to access the shared memory exceeds the time required to access the register. However, like the registers in the thread block, the shared memory is usually located near the multiprocessor that executes the thread. The third level of memory access allows all threads on the device 2010 to read and / or write the device memory 2020. The device memory requires the longest access time because the access must be synchronized between the thread blocks operating on the device.

[0298] The embodiments of the present disclosure may be implemented in any combination of hardware and software. Fig. 20 In addition to the parallel processing architecture presented in the present invention, standard computing platforms (e.g., servers, desktop computers, etc.) can be specifically configured to perform the techniques discussed herein. In addition, embodiments of the present disclosure can be included in an article of manufacture (e.g., one or more computer program products) having, for example, a computer-readable non-transitory medium. The medium may contain computer-readable program code therein for providing and facilitating the mechanisms of embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0299] Apply statistical models to optimize pre-operative or intraoperative planning based on patient activity

[0300] Simple and processor-efficient planning tools are needed for surgical staff to perform patient-specific preoperative or intraoperative planning. Due to the volume of surgery and the time constraints on engineers and surgeons, the preoperative planning stage of joint replacement should be computationally and labor-efficient, while the intraoperative planning stage imposes more computational constraints on any simulation data because there is no time to wait for simulation in the operating room. With the rise of cheap tablets or mobile devices (generally speaking, lower-power computing systems such as cart workbenches in operating rooms), there is an opportunity to provide low-processor overhead applications that can assist surgical staff in collecting data and planning surgical procedures, which utilize more powerful systems connected to the network of data storage devices that maintain simulated or real-world data from past patient cases. These back-end systems can maintain, manipulate, create and mine large amounts of data, allowing lower-power devices in the operating room or utilizing this treasure trove of information. The ideal planning application should assist surgical staff in collecting data and planning surgical procedures and accessing data storage devices for simulated or real-world data from past patient cases.

[0301] For example, in some embodiments, operating room devices or mobile devices with intuitive touch screen interfaces, wireless network capabilities, and cameras make them particularly suitable for helping to create or modify surgical plans that can be used with CASS during the preoperative stage or during surgery. Its interface can be useful in collecting and interacting with new information that can be used to improve surgical plans (e.g., imaging or force characteristics of joints captured during surgery). Devices such as tablets, laptops, or cart-based computers generally lack powerful processors, which can be improved by utilizing servers or databases of past simulations or clinical results, thereby providing the advantages of patient-specific simulations to devices in the operating room without the need to run simulations locally or on demand. Some embodiments of surgical planning tools utilize a network interface to allow a remote database or server processor to offload some data storage or processing from an operating room device or mobile device. By looking up or learning similar simulations in the past, these databases provide opportunities for efficient and instant estimation of simulation results for given patient data, and these databases are particularly suitable for low processor overhead applications.

[0302] Some embodiments recognize that the patient goals for surgery are often unique and personalized. One patient may just want to return to a pain-free, more sedentary life, while another patient may want to return to an active life of golfing, biking, or running. By utilizing a big data storage of similar patient simulations or past results, preoperative data can help surgeons optimize surgical plans toward these specific activity-based goals. Each of various common activities (e.g., walking, climbing stairs, squatting, leaning over, playing golf, etc.) can be characterized by a motion curve that takes into account the actual motion that the joint will experience during the exemplary repeated motion associated with the activity and the expected stress on the implant and soft tissue. Repeated motion curve simulations can be performed for each activity for a variety of patient joint geometries to fill in a database of simulation results. CASS or an application on a user computing device can then request to select the activity for which the patient's surgical plan is optimized. The surgical plan can then focus on the simulation results associated with those motion curves, while ignoring or weighting the less severe simulation results associated with unselected activities.

[0303] An exemplary embodiment of a surgical application or CASS utilizing past simulation results is a knee replacement planning tool. It should be noted that these techniques can also be applied to other resection surgeries, such as partial knee replacement, hip replacement, shoulder replacement, ankle replacement, spinal resection, etc., or any program in which the patient's geometry affects the performance of a prosthetic implant. In the example of a knee replacement planning tool, the planning tool can assist the surgeon in selecting the appropriate size, proper position and orientation of the implant, and the type of implant to be used when implanting the prosthesis, to maximize mobility, and minimize the possibility of failure due to premature wear, impact, dislocation or unnecessary load of the implant or ligament during the expected activity.

[0304] In the context of THA, current guidelines that utilize the thumb range rule for abduction and anteversion angles for acetabular cup placement (e.g., guidelines using the Lewinnek "safety zone") may not be adequate to minimize the risk of hip dislocation as the patient recovers. For example, recent studies have shown that more than 50% of postoperative dislocations occur in implants installed within the Lewinnek "safety zone." Studies have also shown that the patient's spinopelvic mobility can directly impact appropriate acetabular cup placement, which may not be considered in traditional guidelines. Therefore, appropriate implant position and orientation may benefit from planning tools that utilize simulation results to account for patient-specific risk factors.

[0305] One embodiment of a surgical planning tool can be an application running on a desktop, server, laptop, tablet, mobile phone, or cart-based workstation. The exemplary application mainly considers geometric data within X-rays (or other medical image data, such as CT, MRI, ultrasound, etc.) that can minimize the effects of X-ray / image distortion. In some embodiments, the image processing software can estimate the location and distance of prominent points within the image, or a touch screen (or other) interface can allow the user to easily manipulate these points and measurements by dragging them on the image until the surgeon is satisfied with the accuracy of the placement relative to the patient's anatomy. For example, in PKA / TKA, the anatomical axes of the femur and tibia can be extracted from the image to determine the varus / valgus angle; the center points and radii of the distal and posterior ends of the medial and lateral condyles can be extracted; the medial and lateral gaps between the tibial plateau and the corresponding condyles can be measured from images of various degrees of flexion. The shape of the patellar groove can be determined from the anterior / posterior images of various degrees of flexion, and the depth can be determined from lateral images or MRI. The surgeon can also estimate the tension or relaxation of the ligament by applying force to the knee at a predetermined degree of flexion to supplement the image data. Some embodiments use a combination of estimates of the locations of these points (which may be learned from past interactions with the surgeon user as the surgeon user places these points on the image) done by automatic image processing (guided by searching for salient features learned from past human selected training sets) and refined by the user using a touch screen or computer. In some embodiments, images of two or more positions of extension and flexion (e.g., lateral and AP X-rays or MRIs) are considered. In some embodiments, an X-ray (or other image) of the patient standing and an X-ray (or other image, such as an MRI) of the patient sitting are considered. This can provide the system with an estimate of the change in pelvic tilt between standing and sitting, which can be used to estimate patient mobility issues. In other embodiments, X-rays of patients in challenging positions, such as a flexed sitting position or hyperextension when standing, may be considered. Some embodiments also utilize a motion capture system to provide information about the patient's existing joint mobility, such as pelvic mobility limitations.

[0306] Once the image is marked to identify the geometric features and relationships of the patient's anatomical structure (automatically through image analysis or manually through touch screen / UI manipulation), the simulation model can also include any additional patient conditions, such as spinal or hip motion problems. (For example, in the case of THA, the conditions may include a specific range of motion in the sagittal plane and a measure of stiffness. This is important for positioning the implant device in the patient to reduce the incidence of edge loads and misalignment for the patient's expected activity level.) Then, the planning application can perform a lookup of previously performed anatomical simulation results or perform calculations based on transfer functions extracted from multiple past anatomical simulation results (of various patient geometries and attributes) to generate curves for ligament impingement risk, ligament stress, and condylar compartment gap, and patella tracking (for PKA / TKA) or during repeated motion curves for each of the various selected activities, the pressure center and range of motion between the femoral head and acetabular cup (THA) throughout the range of motion. In some embodiments, the various activities to be considered can be guided by the patient's lifestyle and activity level and the overall results of the simulation of each activity. Then, simple user interface options can be presented to the user, to change the position and orientation of the distal incision and the rear incision, and patellar attachment point / filling (for PKA / TKA) or abduction and anteversion (THA) to observe how the position and orientation of the implant are changed to affect these features of the resulting joint. Different implant options that affect the performance of the resulting joint can also be presented to the user, such as target gap size, artificial cartilage thickness, implant size and model, tibia and femur implant depth, relaxation, etc. For THA, these can include but are not limited to: femoral head diameter, liner type (neutral, capped, anteversion, ceramic, constraint, dual mobility or surface resurfacing), standard / high offset, femoral head offset, implant style / series (that is, flat tapered wedge relative to fit and fill), implant depth in bone, implant size, etc. Since the results can be quickly generated algorithmically by table lookup or using the transfer function from past simulation results, the user obtains seemingly instantaneous feedback in the interface about the effect of his or her selection in implant type and position / orientation. This is particularly useful in the operating room when new information is gathered about the patient (e.g., force loading of the joint, soft tissue laxity, etc.) or when the surgeon requests a change to an existing preoperative plan, and provides rapid feedback on how the change or new information affects the expected outcome and joint performance.

[0307] In certain embodiments, a button is also presented to the user, which allows the system to automatically suggest the optimized distal and rear cutting planes, tibial implant depth, patellar posture and filling and implant type and size, to minimize the deviation of the tibia-condylar gap, and the ligament tension and laxity throughout the range of motion, while also minimizing the required ligament release during surgery to achieve acceptable balance (for TKA / PKA). These selections or optimized results can then be easily added to the surgical plan of CASS, or the clear information about range of motion and pressure center is shared between colleagues. Any suitable algorithm can be used to optimize, such as by gradually adjusting angle or implant type and rerunning database search or transfer function calculation, until reaching local or global maximum. In some embodiments, an optimization is performed in advance for each possible combination of patient geometries (or each combination within a reasonable range), searching for a distal and posterior cutting posture or implant type for that combination to minimize deviation in compartment gapping or ligament strain / laxity (PKA / TKA), or an anteversion and abduction angle or implant type for that combination to minimize the risk of edge loading or dislocation (for THA).

[0308] In some embodiments, all reasonable combinations of patient geometry or subsets of all combinations are simulated to create a simulation database for subsequent real-world patient implantation plans. Once enough simulations are performed, the algorithm can search for transfer functions that approximate the results of these simulations, thereby allowing simulation graphs to be interpolated for different combinations of implant types and positions / orientations without actively simulating each combination. Because it may take several minutes to perform each anatomical simulation, determining the transfer function from all or past simulated subsets can speed up the process of determining a specific patient's plan. In some embodiments, a subset of implant orientations can be simulated, and the implant angle is optimized to fill the result database. Then, a suitable machine learning algorithm can be applied to create a learning result model, which can be applied to the results of additional combinations of estimated parameters. This allows a good estimate of flexion and extension gaps and ligament tension (PKA / TKA) or edge loads or misalignment stresses (THA) for various activities and implant orientations to be dynamically generated by the result model, without using an anatomical model (which is processor-intensive each time) to fully simulate motion for each combination. Exemplary algorithms that can be applied alone or in combination to develop and train the learned result model or optimize angle selection include, for example, linear or logistic regression, neural networks, decision trees, random forests, K-means clustering, K-nearest neighbors, or a suitable deep learning open source library. The transfer function can be identified by regression analysis, neural network creation, or any suitable AI / analytic algorithm that can estimate the parameters of the transfer function.

[0309] By running many different simulations for different implant orientations and types for each activity, and simulating the loads, stresses, and range of motion in the joint, the database or model created by analyzing these results can act as a statistical model that describes the output of the multibody system for a given output combination for each activity. This can be achieved by simulating all possible inputs to a model with only a few degrees of freedom as a transfer function for fitting a multidimensional expression that closely estimates the response of the system after hundreds or thousands of simulations that mine a subset of possible parameter combinations, or by applying a conventional machine learning software system to create an AI-driven model of the system response that approximates a multibody model of the joint based on a large number of simulations. In various embodiments, any of these statistical models can be used to model joint behavior in a manner that approaches the accuracy of an on-demand simulation of a patient-specific model for a given activity, without the impracticality of on-demand simulation of the performance of a detailed multibody model of the patient's joint.

[0310] These same concepts can be applied to selecting the position and orientation of other implantable devices, such as total or partial knee replacements, allowing the user to easily mark the X-ray / image data to manipulate the implant position and orientation of the hardware, and quickly get at-a-glance feedback on how those positions and orientations affect risk factors associated with the implant based on the patient's activity level, and easily create or modify surgical plans. Each anatomical simulation uses a specific patient anatomical geometry with a specific implant position and orientation that performs a given repetitive motion associated with a given activity, such as climbing stairs. Each simulation creates a data point in a database that can be accessed when trying to optimize an implant plan for a similar patient later. By repeating this simulation for hundreds to thousands of other geometry combinations and activities, the database can be used for a wide combination of native patient geometries, implant postures, and optional activities.

[0311] Fig.21It is a system diagram of system 2100 of an exemplary embodiment of a surgical planning tool, which can be used in an independent system or as a part of CASS to produce or modify a surgical plan. In some embodiments, a low computational overhead client-server method using pre-filled data storage of complex simulation results allows surgeons to perform instant optimization and adjustment, thereby allowing selection of various postoperative patient activities and changes to implant factors. In this case of hip replacement, this may include acetabular cup anteversion and abduction angle, bearing type (polyethylene, hard material, dual mobility, constraint, surface resurfacing), femoral head size and offset, femoral stem offset (standard, high, valgus), femoral stem style, femoral stem depth and orientation (form). In the case of knee replacement, implant factors include anterior-posterior / lateral-medial placement or cutting angle, distal depth, the distal incision of the condyle and the specific orientation of the posterior incision to ensure that the compartment gap is consistent in the range of motion that occurs during the selected activity, and the ligament is neither overstrained nor too loose, as well as the geometry of the patella relative to the femoral features (PKA / TKA). The user device 2102 can be a personal computing device, such as a mobile device, a tablet computer, a surgical workstation / cart or a laptop / desktop computer. For the purpose of this example, the user device 2102 can be described as a tablet device. In general, it is assumed that the user device 2102 has a low processing power relative to a computer, which can be used to run the simulation of the patient's anatomical structure. System 2100 is particularly suitable for a system in which it is computationally impractical to perform an instant simulation of a given patient's geometry and implant orientation and position for each selected activity. An application 2103 guiding the surgical planning process resides in the memory of the user device 2102. The application 2103 includes a user interface (UI) 2104 and data and an input storage area 2106 for user data. The application 2103 requests certain inputs about a given patient from the user. In some embodiments, the application 2103 can communicate with a medical record server, which includes patient records to provide some of this information.

[0312] Exemplary information requested by the application 2103 via the UI 2104 and stored in the database 2106 includes capturing a picture of one or more images of an X-ray / medical image of the patient using the camera of the user device 2102 (or providing a device by which the user can upload previously captured X-ray / CT / MRI images from medical records or currently captured images during surgery if the application is used during surgery), entering important information about the patient, such as height, weight, age, physical development, and activity level. In some embodiments, X-rays are the primary medical images, but in some embodiments MRI, CT, or ultrasound images may be used. The UI also allows the user to select information about the prosthesis to be implanted, and to select various activities that the patient wishes to participate in postoperatively (e.g., running, golfing, climbing stairs, etc.). In some embodiments, the ability of the user to enter this information using the touch screen of the UI and the device camera simplifies the application so that it does not need to communicate with electronic medical records, which may bring additional regulatory issues and require additional security and software modules, etc.

[0313] The user device 2102 communicates with the server or cloud service 2110 via the Internet 2108. The server 2110 provides backend processing and resources for the user device 2102, allowing the application 2103 to be a lightweight application, and the user device 2102 is almost any available user device, such as a tablet computer or an existing surgical workstation. The server 2110 maintains a model database 2112, which contains simulation results of various implant patient geometries that perform a predetermined motion curve associated with each patient activity. The database 2112 can be pre-filled or continuously updated with additional simulations by a suitable processor, such as a multi-core processor of the server 2110. In some embodiments, additional computers or clusters (not shown) fill this model database, allowing the server 2110 to handle incoming requests from multiple user devices.

[0314] In addition to the simulation results, the model database 2112 may also include a guide to assist users in understanding the appropriate range for selecting appropriate implant posture. In the case of TKA / PKA, this may include selecting appropriate filling and other implant features for patella. For THA, this may include various spinal pelvic motions or sacral tilt angles to assist users in selecting appropriate anteversion and abduction angles, and other implant features. The model database 2112 may also include the best recommendation for implantation information, such as the best implant posture, best implant size or type for a given patient geometry and activity or the best range that will work with a given patient anatomical geometry for performing a given activity. In certain embodiments, the simulation results also include the simulation of the patient with a given geometry and additional obstacles, the orthopedics or neurological pathology that the additional obstacles are not fully captured when viewing the sitting and standing images, for example.

[0315] Once the user has uploaded X-ray images (or other medical images) and manipulated those images to identify certain points and angles in those images (or image processing software has automatically identified or estimated these angles from the image), patient characteristics, expected patient activities and optional starting point implant features (for example, starting posture, implant size, bearing type etc.), server 2110 can consult model database 2112, to find the entry that best meets user input and medical imaging geometry. In certain embodiments, a large amount of independently adjustable variables can make the complete database of all possible combinations become impractical. In these embodiments, the database can include the subset of the possible combination of patient characteristics, X-ray / imaging geometry and implant features, and server 2110 can interpolate specific results from the surrounding entries that best match specific user selection. In certain embodiments, the closest matching can be provided as a result, without interpolation.

[0316] In some embodiments, once various simulations have been performed for various combinations of patient features and geometric shapes, the processor can optimize the transfer function to closely match the results of the simulation using any conventional means as described above. By fitting the transfer function to the results of many simulations, the transfer function can be provided to the server 2110 to quickly calculate the results of various activities for given user inputs of x-ray and implant features and patient features, regardless of whether the specific combination has been simulated previously. This can enable the server 2110 to quickly process the requests of multiple users without having to run potential tens of thousands or more combinations in the simulation to fill the model database 2112. The model database 2112 can be used to store transfer functions, allowing the server 2110 to calculate the results instead of searching the model database 2112. In some embodiments, a learning algorithm is used to train a patient response model for each activity of a given geometry to allow rapid estimation of the response and determination of the optimized implant position and orientation at the server, similar to the use of a transfer function.

[0317] Exemplary simulations can utilize various simulation tools, including LIFEMOD available from LIFEMODELER INC., a subsidiary of SMITH ANDNEPHEW, INC. of Memphis, Tennessee. TMAnatomical modeling software. Exemplary simulations are explained in concurrently owned U.S. Patent Application No. 12 / 234,444 to Otto et al., which is incorporated herein by reference. Anatomical modeling software can utilize a multi-body physical model of human anatomy including bone and soft tissue elements that accurately simulate human joints of a given geometry. Specific physics-based biomechanical models can be customized for patient-specific information, such as height, weight, age, gender, bone segment length, range of motion and stiffness curves for each joint, balance, posture, previous surgeries, lifestyle expectations, etc. Test designs are created to simulate various anatomical geometries and implant angles to perform various predetermined motions, each of which is associated with a different optional activity.

[0318] Fig. 22 Various theoretical angles that can be extracted from X-ray imaging using hip joint geometry models are shown. Model 2120 is a model of the geometry of the hip joint in a standing position, while model 2122 is a model of the hip joint in a sitting position. Various angles that can be extracted from the geometry of these models include dynamic sacral tilt or inclination (ST, standing 45°, sitting 20°), static pelvic incidence (PI, standing and sitting 55°), dynamic pelvic femoral angle (PFA, standing 180°, sitting 125°), dynamic anteversion (AI, standing 35°, sitting 60°). Fig. 22 Theoretical angles that can be extracted and used, not shown in FIG. 1 , may include a combined sagittal index, which is defined as the sum of the anteversion angle and the pelvic femoral angle. The static sacroacetabular angle (SAA) at the intersection of the lines that produce the ST and AI angles is also not shown in this model, as explained below.

[0319] Fig.23A are various points, lines, and angles of x-ray that can be extracted from an x-ray image 2130 of a standing person's hip. The user can manipulate the points 2151 to 2156 using a user interface / touch screen, or the points can be automatically generated by an image processing algorithm that can be improved through multiple iterations of machine learning on user input as the application is developed and used by real-world users. Note that in some embodiments, other image types can be used, including MRI, CT, or ultrasound image data. Once the points 2151 to 2156 are placed in the image, the lines 2132 to 2144 can be automatically placed on the image, allowing for Fig. 22 The various angles described are automatically calculated by the processor of the user device. Points include the upper / posterior S1 endplate 2151, the lower / anterior S1 endplate 2152, the center point between the hip joint center 2153, the posterior acetabulum 2154, the internal acetabulum 2155 and the femoral axis point 2156.

[0320] Lines include horizontal line 2132 originating from point 2151, line 2136 (which runs between superior / posterior S1 endplate point 2151 and inferior / anterior S1 endplate point 2152), line 2134 (which automatically generates line 2136 perpendicular to the bisector of points 2152 and 2151), line 2138 (defined by the locations of points 2153 and 2155), line 2140 (defined by the intersection between lines 2134 and 2136 and point 2155), and line 2142 (defined by points 2155 and 2156), and horizontal line 2144 (from point 2154). These lines are automatically generated once points 2151 to 2156 are added or extracted from the image. This allows the processor to determine the ST (between lines 2132 and 2136), PI (between lines 2134 and 2140), SAA (between lines 2136 and 2138), PFA (between lines 2140 and 2142), and AI (between lines 2138 and 2144) angles.

[0321] like Fig. 23B As shown in FIG. , each point 2151 to 2156 (in Fig.23A 2144) is manipulated and extracted from an X-ray image 2158 of the patient's hip joint in a sitting position. From these points, lines 2132-2144 can be extracted by the processor of the user device / tablet. Once these lines are extracted, various angles can be calculated. If the static PI and SAA angles are inconsistent between sitting and standing, an average value can be used. In some environments, inconsistencies between these angles can lead to errors or user requests for additional input. In some embodiments, inconsistencies between these angles can be used to scale or adjust other angles or refine the placement of points in the sitting and standing X-ray images. The server can use these angles to find appropriate simulation results because these angles define the geometry of the patient's anatomical structure.

[0322] Fig.242160 is a flowchart of an exemplary process, through which a user manipulates landmarks in an X-ray (or other medical image) to determine feature positions, sizes, angles, and spacings that will be used for patient geometry to determine joint performance for various patient activities using pre-filled statistical models of the behavior of each activity on the joint. This can typically be done during the preoperative phase, but can be updated in response to images captured during surgery or based on new data obtained during surgery, such as using a probe tracked by CASS to capture bone features and their relative positions. For example, for knee replacement, performance can include measures of changes in condylar or compartmental gaps, tension or laxity and ligaments, and the degree of patellar tracking in the patellar groove of the femur. At step 2162, an application on a user device loads an anatomical image of the patient into memory. This can be achieved by capturing images from a membrane or screen using a camera, by connecting to stored images in a local data storage device or medical record system, or by creating images (e.g., X-rays taken during surgery) using a local imaging device. In some embodiments, the image may be an X-ray, ultrasound, CT, MRI, or other medical image. The file format of this image may be any suitable file format, such as PDF, JPEG, BMP, TIF, raw image, etc. Once the user device has loaded this image, at step 2164, the image is displayed and optionally analyzed using image recognition software. This optional analysis step may utilize any common image processing software that recognizes salient features in an image that can be used to identify landmarks. For example, the image processing software may be trained to look for certain geographic features, geometric markers, or may be trained through any suitable AI process to recognize the likely locations of landmarks used to determine the geometry of a patient. At step 2166, the results of this analysis step are overlaid on the displayed image, placing marker points at the pixel locations (e.g., the locations of the marker points that the analysis software believes are most likely to be. Figure 23A-23B ). It should be understood that the process described in steps 2162 to 2172 need not be limited to medical images and can be done automatically without a person interacting with the processor. During operation, additional geometric and positional data can be acquired in CASS by various means, including images or by a surgeon using a robotic arm or manipulating a point probe to draw the surface of the relevant features and register their positions with the surgical tracking system. This provides an additional level of improvement in the geometric extraction of relevant features, which can be used to optimize the surgical plan to a degree beyond what can be achieved by preoperative imaging alone.

[0323] In some embodiments, image recognition software is useful and can be a supplement / partial or complete replacement for analyzing experienced surgeons or technicians. In some embodiments, at step 2168, the user of the software has the opportunity to manipulate the precise placement of anatomical features, such as femoral axis, sacral geometry, femoral head and neck features, acetabular cup geometry, condylar center and radius, patellar groove, patellar dome and tendon and ligament contact points. The exact method for the user to manipulate these points will depend on the interface of the computing device. For example, on a tablet computing device with a touch screen, the user can use his or her finger to manipulate and drag the exact placement of these points (already automatically placed at step 2166). In some embodiments, in the case of not yet training suitable image processing software, the user can create these points from scratch by tapping the position of these points on the image after being prompted by the system at step 2168. In response to moving each point, at step 2170, the display creates and updates the model of anatomical geometry (for example, the relationship between condylar features and tibial features and patellar features, and the information suitable for determining the strain on ligaments and tendons, such as quadriceps angle). (In some embodiments, the steps may be temporarily suspended until the user indicates that all points have been moved and the user is satisfied with their placement.) The process continues at step 2172, where the software and user determine whether all points and distances have been manipulated and placed correctly, and the manipulation is repeated for each point and distance.

[0324] Once all points have been moved and the user is satisfied, in step 2174, the user is given the option to select and load additional images. For example, when the planned surgical procedure is a knee replacement, suitable images may include anterior-posterior and medial-lateral images of the patient's knee in a flexed position, as well as images in an extended position or any additional postures required to determine the geometry of the relevant features. For hip replacement, suitable images may include at least a lateral image of the patient in a standing position and an image in a sitting position. In knee replacement, suitable images may include a lateral or anterior view of the knee in full extension and bent at a predetermined angle, such as 90°. Additional images may include the patient in a flexed sitting position or an extended position when standing. If additional images, such as posterior / anterior views or medial / lateral views of the joint or different positions, are available, the images may be loaded again in step 2162.

[0325] Once all images are loaded and analyzed, the placement of the markers can be manipulated by the user (or software image processing AI). The relevant angles and distances can be calculated from the positioning of these points, and the method can proceed to step 2176. At step 2176, the user can select the appropriate activities that the patient wishes to enjoy after surgery. These activities can be adjusted, for example, by the relative activity level of the patient, age, other frail conditions of the patient, or mobility issues. Exemplary activities may include standing, walking, climbing or descending stairs, riding a bicycle, playing golf, low-impact yoga, squatting, sitting cross-legged, gardening / kneeling, etc. It should be understood that some of these activities may be relevant to most or all patients, while some activities may only be relevant to younger or specific patient subsets. For example, only younger or more active patients may choose running activities.

[0326] At step 2178, the selection of activities and the patient-specific geometry calculated from the manipulation point on the patient image can be loaded into the server to (or be set in a processor-accessible memory, which is suitable for) apply the model created from existing simulation data to calculate the appropriate results related to the surgical procedure. For example, for hip replacement, the result can include the figure of the pressure center in the acetabular cup and the range of motion of the joint, and the range of motion is expressed in the polar coordinate diagram of the femoral head relative to the acetabular cup. For knee replacement, the result can include the range of motion between the tibial plateau and the femoral condyle, the contact point diagram, and the stress experienced by the relevant ligaments, the figure of the lateral and medial condylar chamber gaps and ligament tension drawn for the flexion angle range or discrete angles, or the figure of the patellar tendon or ligament that is highlighted in the range of motion, or the mode that the patellar dome sits in the patellar groove during this range. For this first pass of step 2178, the default implementation features of the prosthetic implant (for example, 3 degrees of varus angle and the condylar chamber gap of 9-10mm) can be used. In some embodiments, before step 2178 is performed, the user also selects a starting implant position / orientation for a given implant to use in this initial calculation.

[0327] At step 2180, the processor that applies the statistical model to the anatomical information and activity selection sends this result in the implanted variable for the result to the user device for display. In an embodiment using a client / server model, the step may include sending the result to the user device over a network. In some embodiments, the processor that performs the analysis using the statistical model can be the same processor as the user device. In these embodiments, the statistical model is generally optimized enough to be stored and analyzed locally. The exact display of these results can be in any suitable graphical manner, such as those interfaces described in subsequent figures.

[0328] At step 2182, the user has the opportunity to select to manually manipulate implant features (position, orientation, and in some embodiments, size and type) or request the processor to automatically optimize the implant geometry. In some cases, the user can choose to be automatically optimized by the processor, and then manually refine the exact excision for implantation to adapt to the surgeon's preferences or to give greater emphasis than other activities to certain activities. For example, the surgeon can try to optimize the implant so that the patient can resume playing golf, but still emphasizes the ability of the patient to perform daily activities, such as climbing stairs and sitting comfortably. In certain embodiments, the weighting of the activity can be provided in the patient profile, and the weighting can be used by the optimization algorithm to automatically perform.

[0329] At step 2186, if the user has requested the processor to optimize the implant angle, the optimization algorithm is run by the processor (at the server or on the user device in the non-client / server embodiment) based on the statistical model. This optimization algorithm may include any search algorithm, which searches through a statistical database to find the local or global maximum implementation angle, and the implementation angle provides the best performance or analyzes the extracted transfer function to find the maximum or minimum result. The standard used for this search may vary depending on the implant type implanted. For example, in knee replacement, the algorithm can aim at the reasonable range of the condylar compartment gap for the medial and lateral compartments, identify the most consistent gap through the flexion range according to the guidelines, while maintaining the ligament tension in a suitable range, and identify the femoral implant posture and patella filling and attachment constraints that best allow the patellar dome to track in the patellar groove with minimal soft tissue strain. For hip replacement, the anteversion and abduction angles of the pressure center distribution and range of motion distribution with the lowest risk of edge load or dislocation can be identified by minimizing the pressure amount near the acetabular cup edge and minimizing the range of motion amount of the cup edge of each selected activity that risks impacting.

[0330] In some embodiments, this optimization program may include iteratively changing the resection placement and orientation until suitable or optimal results are achieved. Once the processor processing the statistical model has completed this optimization, the result can be received and displayed by the user device at step 2180. If the user wishes to manually change the implantation angle, then at step 2184, the user can update the selected implantation angle through the user interface. Then, at step 2178, the selection can be uploaded to the processor processing the statistical model. In some embodiments, the user can also change the implant bearing type or pad options. For example, the user can switch from a 0 degree neutral XLPE liner to a 20 degree forward tilt XLPE liner, thereby allowing the user to experiment with range of motion and center of pressure. Similarly, the user can switch from a conventional bearing surface (XLPE) to a dual mobile system, which provides an increased head size and jump distance to allow greater stability in the joint.

[0331] Once the user is satisfied with the results of the manipulation or automatic optimization, at step 2188, the user device updates the surgical plan displayed to the user or within CASS, allowing the robotic surgical system to prepare for implementation of the selected implant position and orientation based on method 2160. In an embodiment using a cutting guide, step 2188 may include sending the surgical plan (including specific maps of the relevant patient bone surfaces and specific locations of the resection cuts relative to these surfaces) to a manufacturing system that manufactures patient-specific cutting guides, and requesting 3-D printing of these cutting guides prior to surgery. In some embodiments, where the robotic arm will position or hold a non-patient-specific cutting guide, the surgical plan may include a request to provide the surgeon with an appropriate cutting guide for use in the surgery, and programming the robotic arm to place the cutting guide at a specific predetermined position and orientation.

[0332] Fig.25 is an exemplary table 2200 that can be displayed to a user to assist in understanding how the preoperative anatomy compares to the expected ranges for a healthy patient. This can help guide the surgeon in understanding the appropriate surgical plan for use with the CAS system. Table 2200 shows the results of calculating various angles in the sitting and standing images using image analysis or manipulation of points as described above for planning hip replacements. Column 2202 lists the various angles calculated for each image, including ST, PI, PFA, and AI. Column 2204 shows the results of analyzing the angles based on the points identified in the standing image. Column 2206 is the analysis performed by the server or user device based on a list of acceptable ranges for these angles, e.g. Fig.21 The guidelines shown in Table 2114 in . In this example, sacral tilt has been identified as normal, while pelvic incidence, pelvic femoral angle, and anti-tilt angle have been identified as abnormal, outside the expected normal range for a healthy patient. Column 2208 shows the angles that have been calculated based on the points identified in the sitting x-ray image. Column 2210, like column 2206, is an identification of whether each angle is within the acceptable range for a healthy patient. Section 2212 is a comparison of various angles between the standing and sitting positions (2204 and 2208), including the differences in sacral tilt and anti-tilt. These increments are then used to determine whether spinal-pelvic mobility falls within the normal range and where spinal-pelvic balance is located. This information can be used to guide the selection of the implant orientation of the prosthesis to improve mobility and balance the patient's hip joint geometry. This information can be considered by the surgical user or can be provided on demand as a teaching tool. As discussed, the standing and sitting angles are used from the patient model database in conjunction with the anteversion and abduction of the acetabular cup (or other implant characteristics) to determine range of motion and center of pressure results from the statistical database of simulation results.

[0333] Fig.26A2220 is an exemplary user interface for selecting a single activity associated with a given patient and displaying results from a statistical database based on previous user input about the patient, including X-ray marker mapping for the patient. The user can select from a variety of individual activities 2222 and can manipulate the abduction angle 2224 and anteversion angle 2226 for the implant (note that these exemplary angles may be different from the real-world values ​​of a given implant). The center of pressure heat map illustration 2228 is an aggregation of all selected single activities 2222 based on patient geometry from X-ray imaging and manipulation of the abduction angle 2224 and anteversion angle 2226. (The option to change the pad and bearing properties is not shown, which may be available in some embodiments.) Based on the statistical model, manipulating the abduction angle 2224 or anteversion angle 2226 will move the heat map relative to a circle that represents the range of the acetabular cup. Placing pressure too close to any edge of the acetabular cup may cause edge loading, which may cause premature wear or failure of the hip implant, or in some cases dislocation. Similarly, the range of motion diagram 2230 is an aggregate of the patient's range of motion for each individual activity and how that motion translates into interaction between the femoral head and the acetabular cup. The breadth of this range of motion should be limited to the circle representing the breadth of the acetabular cup. Any range of motion beyond these breadths will create a high risk of hip dislocation or impingement, which prevents the motion required for that activity.

[0334] In certain embodiments, the optimized button 2232 is presented to the user, allowing the user device or server to automatically change abduction and anteversion angles so that the placement of the center of pressure is optimized and the range of motion in the circle. (In certain embodiments, this automatic recommendation may include implant selection, such as implant size, bearing type, lining type, femoral head features, etc.) This can be iteratively completed using any suitable algorithm to adjust these angles, thereby improving the center area pressure and the range of motion that falls into the acetabular cup circle. The heuristics used may include maximizing the distance between the pressure point and the edge of the acetabular cup for the center pressure, and maximizing the distance between the breadth of the range of motion in the edge of the acetabular cup circle, maximizing the average distance, maximizing the minimum distance, etc.

[0335] In some environments, a graphic 2234 is presented to the user that changes as the abduction and anteversion angles are manipulated to provide the user with visual feedback on how these angles affect acetabular cup placement. Not shown in interface 2220 are buttons that exist in some embodiments to finalize and save the abduction and anteversion angles and load these angles into the surgical plan for the CAS system.

[0336] Fig.26B2236. Instead of a large heat map or a large range of motion, the center of pressure graph illustration 2238 shows a more limited center of pressure attributed to only these activities. Similarly, the range of motion graph illustration 2240 shows the range of motion used by only these activities. Fig.26A ) comparison, illustrating that a wider range of motion results in a wider heat map of the center of pressure and a denser range of motion. For some patients whose mobility is limited by lifestyle or other factors, only certain activities may be important, allowing the user to more easily optimize implant selection and acetabular placement (in hip replacement applications) to ensure that the user can successfully perform these activities. For some patients, it may not be feasible given other constraints to place the acetabular cup in a way that all activities are possible. This may be due to abnormal hip joint geometry determined from (X-ray) images or due to external mobility issues, such as spinal fixation. In some embodiments, the surgeon or user is able to recommend to their patients which activities or positions may pose a risk of impingement, dislocation or excessive wear on their artificial hip joint.

[0337] Fig.26C 22 is a user interface illustrating an embodiment whereby hovering over or otherwise temporarily selecting a single activity 2242 in a group of activities can be used to highlight the contribution of that activity to the center of pressure graph illustration 2228 and the range of motion graph illustration 2230. By selecting going down stairs, a center of pressure graph is associated with the single activity, and the range of motion graph associated with the single activity can be highlighted within the respective graphs. In this example, a heat map illustration 2244 is temporarily highlighted on the center of pressure heat map illustration 2228, showing a concentrated portion of the graph attributable to going down stairs. Similarly, a range of motion curve 2246 can be temporarily highlighted within the range of motion graph illustration 2230 to show the contribution to the range of motion graph that can be attributed to this activity.

[0338] Fig.26D is an illustration of how the user interface changes when manipulating the abduction angle 2224 and anteversion angle 2226. In this example, all activities are selected, and the abduction angle is reduced by 12°, while the anteversion angle is reduced by 2°. This results in an expanded heat map illustration 2248 of central pressure, which creates a risk of increased rim loading of the acetabular cup, while the range of motion map illustration 2249 of these activities is shifted from the center of the acetabular cup, which creates a greater risk of dislocation or injury. Fig.26D The results are not as good as Fig.26A The result is ideal.

[0339] Fig. 27The impact of adding multiple activities together to produce an aggregated center of pressure heat map and range of motion map is shown. In this example, climbing and descending stairs results in a wider center of pressure and a wider range of motion than the individual activity maps.

[0340] Fig.28 Be the flowchart of the exemplary creation of the statistical model database of explanation, the statistical model database can use the patient's anatomical structure and implant feature and the patient's activity selection for estimating the performance of implant to carry out inquiry.This statistical model database is created by using joint anatomical model (such as multibody simulation) to perform (usually) hundreds to tens of thousands of simulations, the behavior of each part of joint when the joint is moved through motion curve is simulated by the model simulation, and the motion curve is associated with at least one common motion that will occur in the patient's joint when the patient participates in a given activity.In certain embodiments, this motion curve can be simulated by using motion capture technology on the sample subject performing a given activity.By capturing the motion of this individual (or multiple individuals) performing the activity, the accurate model of the motion that each joint will experience during the activity can be created.Then, the model of single joint motion can be reviewed to identify the exemplary repetitive motion that a person may experience when performing the activity.Then, this motion curve model can be used for guiding each individual simulation of this activity, wherein anatomical geometry and implant feature change to produce the experimental design including a series of anatomical geometry and implant feature.

[0341] In some embodiments, method 2260 begins at step 2262, where motion capture is used for a single participant while the individual performs exemplary tasks associated with each activity to be simulated. For example, reflective markers may be placed on the model's body as the model climbs and descends stairs in front of one or more cameras. At step 2264, the processor may then extract the motion that these markers experience during the activity. By using an anatomical model, and when these markers relate to individual joints, the motion curve experienced by each joint during the activity may be extracted. Any suitable conventional motion capture technology may be used for this step. For example, in the case of generating a hip or knee motion curve, at least two cameras may capture reflective markers on the model's leg, iliac crest, trunk, femur, tibia, patella, ankle, medial and lateral condyle, etc. as the individual moves. Hip and knee curves may be created simultaneously with sufficient markers. When the individual moves his legs during the activity, lifts and places his feet, the degree of motion and rotation within each degree of freedom may be calculated as the hip and knee move. The processor may then use this to estimate the relative motion of the various components of the joint.

[0342] At step 2266, the processor performing the simulation of the anatomical model will load the anatomical model. An exemplary anatomical model can be a multi-body model of bone and soft tissue components that accurately simulates the behavior of each of these components of the anatomical joint. By using a multi-body model, simulation time can be accelerated by a finite element analysis model. In some embodiments, a finite element analysis model can also be used. Exemplary multi-body simulation tools for use with modeling joints include LIFEMOD available from LIEMODELERINC TM Software. Once loaded, this anatomical model can be customized for a given geometry. For example, component dimensions can be adjusted to achieve any joint geometry to be simulated.

[0343] At step 2268, sample joint parameters (e.g., anatomical geometry and implant size, type, position, and orientation) are selected for the next simulation. The selection of the joint geometry can be based on a pre-planned experimental design or can be randomly assigned in a Monte Carlo style simulation. The components in the model can then be appropriately sized to match the geometry of this given experimental simulation. For example, in the context of TKA, the selected geometry can include any reasonable combination of distal incisions and posterior incisions, as well as other implant features for implanting sample prostheses. In some embodiments, the selected geometry may include additional patient information, such as abnormal motion constraints due to other frail conditions or deformations, or other common medical comorbidities. In some embodiments, age and weight can be added to the model to account for changes in various components that may vary in response to the patient's age or weight (e.g., poorly compliant or thinner soft tissue).

[0344] At step 2270, a simulation of a model whose dimensions have been set according to experimental parameters is performed using a given active joint motion curve. This produces several quantifiable results, such as soft tissue indentation and tension, such as pressure and condylar chamber clearance between the components of the femoral head and acetabular cup (for hip arthroplasty) and patella tracking between the components over a range of motion (in knee arthroplasty). The simulation profile used to run the simulation in step 2270 can define which results should be generated and recorded. At step 2272, once the simulation has been completed, the results of a given combination of anatomical and implant features and motion curves can be stored.

[0345] At step 2274, the processor determines whether additional parameters should be simulated, such as anatomical geometry, implant design, orientation, and position. In many embodiments, the simulation profile defining the experimental design or extensive Monte Carlo simulation will define that hundreds to thousands of different simulations should be performed. Therefore, the loop of steps 2268 to 2274 should be repeated multiple times. Once a sufficient number of simulations have been completed, method 2260 can proceed to step 2276, where the processor mines the results of these multiple simulations to create a transfer function or model of joint behavior based on various geometries. The model or transfer function creates a statistical model for the activity and the joint, which accurately estimates performance based on the patient's anatomical structure and the geometry of the implanted prosthesis. In some embodiments, this can be achieved by performing a statistical fit of a polynomial function, which maps the input implant value to an output value. In some embodiments, a neural network or other AI strategy can be used to create a heuristic model of implant features to performance output. As additional simulation results are added to the database, step 2276 can be repeated. In some embodiments, as clinical execution and monitoring of additional implant surgeries, the measured real-world performance values ​​can be added to the database and mined similarly to the simulation results.

[0346] At step 2278, the processor stores and maintains a statistical model of joint performance that varies with anatomical geometry and implant features (position, orientation and type of selected implant). This stored model can be updated when additional simulation data is available. In some embodiments, the resulting model can be computationally lightweight (e.g., transfer function), allowing the mining and manipulation of this model to be performed by any processor in the system, at the server, or on the user device. At step 2280, the processor mines this statistical model to identify the best implant value for optimizing performance based on a predetermined heuristic. For example, in a model of hip arthroplasty, the diagram of the center of pressure and range of motion can be optimized to minimize the risk of edge load or dislocation, minimizing the range of motion and the degree of pressure that fall near the edge of the acetabular cup for a given activity. For example, in a knee arthroplasty model, the heuristic can include gaps, patella tracking, and ligament tension close to normal anatomical values ​​during a given simulated activity. This optimization can be performed by any statistical or AI method, including finding the best fit of the most approximate ideal anatomical model. Then, at step 2278, these optimized values ​​can be added to the model maintained.

[0347] Once the model has been stored and optimization has been performed to assist in identifying the optimal implant features (position, orientation, implant type, implant size) for a given patient anatomy and activity, the processor may query the model in response to user interaction, such as during a preoperative or intraoperative planning phase, at step 2282. This allows the surgical user to access the statistical database model to develop a surgical plan that may then be used by CASS or displayed to the user.

[0348] Fig.29 2400 is a flow chart of an exemplary method 2400 for creating a preoperative plan using a simulation database and patient-specific geometry extracted from an image and (optional) motion capture. The same method can be used to optimize the preoperative plan for hip (or other) arthroplasty, but will be discussed in the context of knee arthroplasty. At step 2402, the surgical planning system collects preoperative imaging, such as MRI, CT scans, X-rays and ultrasound images. These can be completed at any time before surgery. Image files can be loaded into the patient-specific database for feature extraction. In some embodiments, at step 2404, motion capture technology can be used in preoperative visits to capture the patient's gait by fixing markers to various points on the patient's legs and observing the relative motion of the markers when the patient performs various movements. This can be used to provide additional details of the geometry of the preoperative state of the patient's knee. At step 2406, imaging is analyzed using geometric techniques for image analysis (and any motion capture data is analyzed based on the motion model of the human anatomy). The software performing the image analysis may use any suitable feature extraction technique to identify predetermined features in the patient's images and create a three-dimensional model of the patient's knee based on multiple images including information such as sacrum / pelvis geometry, femoral head / neck and acetabular cup geometry, femoral and tibial axes, condylar centers and dimensions, existing condylar gap, patellar dimensions, and their existing relationship to preoperative soft tissue tension.

[0349] In parallel, and typically prior to these steps, a statistical model is created at step 2408 that takes into account a variety of possible patient geometries for a wide range of patients. Hundreds to thousands of simulations for a variety of possible patient geometries may be performed offline to generate the statistical model. The simulation data may then be mined to create a transfer function or simplified model for a given geometry, allowing performance to be determined or estimated for any given patient geometry. At step 2410, when the anatomical geometry of the individual patient has been determined (step 2406), the statistical model is loaded.

[0350] At step 2412, statistical models are used to explore possible corrections to the patient's given anatomical geometry. This can be done using various suitable methods known in the art, such as applying artificial intelligence search and matching algorithms to identify incremental improvements in implant geometry to correct the patient's frailty. For example, AI can be used to identify test designs to identify possible candidate changes, thereby improving the mechanics of the patient's joint. Similarly, Monte Carlo simulations can be used to study random changes in implant geometry using statistical models to identify the best performing options for implanting TKA prostheses. In some embodiments, at step 2412, many (e.g., dozens or thousands) of changes are used to identify the best implant solution. Since this step is preoperative, if necessary, processing time or overhead can be considerable. Various attributes can be changed, including the posture of the tibial and femoral components, patellar filling and dome geometry, etc., to find a solution that optimizes the performance of the patient's knee, which takes into account the tibia-femoral and patellofemoral joints, rather than treating the patella as an afterthought, as is often the case with existing surgical plans. In some embodiments, the statistical model created at step 2408 can be queried to find the optimal implant posture simply based on the starting patient geometry, for example by using a transfer function with an AI model. In some embodiments, the simulated variations performed at step 2412 can be specific to different patient activities, allowing activity-specific optimization for individual patients.

[0351] In some embodiments, the purpose of step 2412 is also intended to create a patient-specific model that takes into account the imprecise nature of preoperative data. For example, preoperative images and motion capture can produce an estimate of the patient's geometry that is not exact. The data collected during surgery can be used to improve the model of the patient's anatomical geometry later. By considering multiple changes not only in the implantation posture but also in the patient's anatomical geometry (within a certain range), a patient-specific model can be created so that the surgical plan can be modified instantly based on additional data or based on the surgeon's request to change the implantation plan during surgery.

[0352] Once multiple variations of patient geometry and implant posture are considered, the patient-specific model can be stored at step 2414. This model can be stored in non-volatile memory, allowing CASS to access it during surgery. This will refer to Fig.30 Discussion. Any suitable amount of information or format may be used with the goal of streamlining any processing or simulations done during a patient procedure so that changes can be processed in real time without slowing down the procedure.

[0353] At step 2416, the processor may create an optimized surgical plan based on the optimization of the implant posture created at step 2412 and the patient profile. For PKA / TKA, this plan may include implant postures for the femoral and tibial components (including the resections required to achieve these postures), as well as plans for any changes to the dome or patellar button that may be required to fill the patella and achieve the relationship between the patella and femoral components as part of the joint replacement. At step 2420, this optimized preoperative plan is provided to the surgeon and CASS to prepare for surgery.

[0354] In some embodiments, an additional step, step 2418, can be performed whereby variations of the preoperative plan are created to account for possible variations in the patient's anatomical geometry that may be discovered during surgery, as well as any reasonable deviations from the surgical plan that the surgeon may make during surgery. These variations can be associated with the expected performance outcomes of the surgically modified joint. This can make it easier for CASS to provide recommendations during surgery based on additional patient data observed in the operating room or based on the surgeon's request. Effectively, this can result in an extremely low computational load for providing recommendations or calculating the expected performance impact of additional data or decisions during surgery.

[0355] Fig.30 An exemplary method 2430 for updating a surgical plan or providing recommendations to a surgeon during surgery is shown. Once a preoperative plan is created, additional information that can be used to update this plan can be collected in the operating room. Similarly, surgeons often rely on their own experience and expertise and can adjust the surgical plan based on what they find during surgery or inconsistencies with AI-generated recommendations. At step 2432, CASS collects intraoperative imaging and probe data. Intraoperative imaging may include any conventional medical imaging, such as ultrasound data. This intraoperative imaging can be supplemented with preoperative imaging. This can provide additional details or updates to the model of the patient's geometry. Similarly, once the patient's tissue is opened, the probe can be used to "map" the various surfaces of the patient's bones, as explained throughout. This can provide additional details about the exact 3-D properties of the patient's tissue surface, which can be more accurate than models created by two-dimensional or three-dimensional imaging. At step 2434, the processor analyzes the intraoperative imaging and probe data to supplement the model of the patient's geometry by identifying specific features observed in the operating room and comparing them with existing models. Once the geometric model of the patient's anatomy is updated, the surgical user may request an update to the plan at step 2436 (or forgo requesting an update and jump to step 2444 ).

[0356] In response to this new data and the user request, at step 2440, the processor selects the best plan based on the new anatomical geometry model. This is accomplished by first loading a patient-specific model or plan from memory at step 2438. Due to the time-critical nature of intraoperative recommendations, in some embodiments, the model and plan loaded at step 2438 are patient-specific, e.g., Fig. 27 This limits the range of possible geometric changes to the most relevant based on the pre-operative patient anatomy model to speed up the treatment of plan changes. At step 2440, the best plan can be selected based on the observed patient anatomy by any conventional computational methods from the available plans and models. At step 2442, this updated suggestion of the plan can be presented to the user through the user interface of CASS and to CASS to update the plan it will help implement.

[0357] At step 2444, the processor may begin monitoring the user for user actions or requests for plan updates. User actions may be monitored by CASS, for example by monitoring the actual resection performed by the surgeon. For example, a deviation from the surgical plan in a resection may require suggested changes to other resections to limit the impact of the deviation. The user may also manually change the surgical plan based on expertise and experience, such as if the surgeon intentionally deviates from the recommended plan. These deviations from the plan will be recorded by the processor, which will provide feedback to the user. If the user wants suggestions for plan changes at any time, the user interface may be used to request suggestions, which repeats step 2436.

[0358] At step 2446, the processor estimates the performance impact of deviations from the optimal plan provided at step 2440, and provides feedback to the user based on this estimate at step 2448. For example, for PKA / TKA, changes in patella / implant posture that oc...

Claims

1. A method for updating a surgical plan, comprising: receiving a plurality of images of a musculoskeletal anatomy of a patient, the plurality of images including an image of a hip joint of the patient in a standing position and an image of a hip joint of the patient in a sitting position; identifying anatomical angles within the plurality of images to define a preoperative hip joint geometry; selecting one or more postoperative patient activities from a plurality of model activities to consider in optimizing prosthetic hip implant performance from a set of available postoperative patient activities, wherein the prosthetic hip implant performance is indicative of at least one of: implant edge loading, impingement, and implant range of motion during a motion profile corresponding to the patient activity; accessing a statistical patient model computationally created from a simulation database, the statistical patient model predicting prosthetic hip implant performance based on the preoperative hip joint geometry and prosthetic hip implant implant parameters for a motion profile representing each of the plurality of model activities, wherein the simulation database contains results of a plurality of simulations of hip joint performance for the hip models, wherein each simulation models a hip joint having one of a plurality of hypothetical hip joint geometries undergoing a predetermined motion profile corresponding to one of a plurality of patient activities, wherein the hip joint geometry includes at least one anatomical angle, and wherein the plurality of simulations use implant parameters as well as the patient activities; calculating from the statistical patient model a set of proposed hip implant implantation parameters that provide predicted prosthetic hip implant performance consistent with predetermined performance criteria for selected one or more postoperative patient activities and the preoperative hip joint geometry; and The surgical plan was updated using the proposed hip implant placement parameters.

2. The method of claim 1, wherein the anatomical angles include at least one of sacral tilt, pelvic incidence, pelvic femoral angle, and anteversion.

3. A method according to any one of claims 1 and 2, wherein the predetermined performance criteria include an estimate of implant edge loading during each motion curve representing each of one or more selected postoperative patient activities.

4. A method according to claim 1 or 2, wherein the predetermined performance criteria include an estimate of the range of motion of the implant during each motion curve representing each of the selected one or more postoperative patient activities. The method according to claim 1 , wherein the plurality of images are X-ray images.

6. The method according to claim 1 or 2 also includes providing a user interface to allow a user to select the multiple postoperative patient activities, and graphically displaying the expected postoperative performance of the implant for the selected multiple postoperative patient activities in one or more polar coordinate graphs.

7. The method according to any one of claims 1 or 2, wherein the step of identifying anatomical angles is performed manually by identifying landmark anatomical feature locations in the plurality of images.

8. The method of any one of claims 1 or 2, wherein the step of identifying anatomical angles is automatically performed by a processor by identifying locations of landmark anatomical features in the plurality of images.

9. A computer-assisted surgery system for performing a hip replacement procedure, comprising: a simulation database comprising results of a plurality of simulations of hip joint performance for a hip joint model, wherein each simulation models a hip joint having one of a plurality of hypothetical hip joint geometries undergoing a predetermined motion profile corresponding to one of a plurality of patient activities, wherein the hip joint geometry includes at least one anatomical angle, and wherein the plurality of simulations use implant parameters as well as patient activity, and wherein the prosthetic hip implant performance indicates at least one of: implant edge loading, impingement, and implant range of motion during the motion profile corresponding to the patient activity; a statistical model computationally created from the simulation database, the statistical model predicting hip implant performance for a given hip joint geometry performing at least one of the plurality of patient activities; a first set of software instructions that direct one or more processors to receive a plurality of patient images, the plurality of patient images comprising at least an image of the patient's bony anatomy in a sitting position and an image of the patient's bony anatomy in a standing position, and identify anatomical angles within the plurality of patient images to define a preoperative hip joint geometry; and A second set of software instructions instructs the one or more processors to use a statistical model to determine recommended hip implant implantation parameters based on the preoperative hip joint geometry and one or more user-selected postoperative patient activities so that the predicted performance of the hip implant meets predetermined postoperative performance criteria for the preoperative hip joint geometry and a set of user-selected patient activities, and to update the surgical plan to configure the computer-assisted surgery system to assist the surgeon in resecting the patient's bone to achieve the implantation parameters.

10. The computer-assisted surgery system of claim 9, wherein the anatomical angles include at least one of sacral tilt, pelvic incidence, pelvic femoral angle, and anteversion.

11. A computer-assisted surgical system according to any one of claims 9 or 10, wherein the predetermined post-operative performance criteria include an estimate of implant edge loading during each motion curve representing each of a set of patient activities selected by the user.

12. A computer-assisted surgical system according to any one of claims 9 or 10, wherein the predetermined post-operative performance criteria include an estimate of the range of motion of the implant during each motion curve representing each of a set of patient activities selected by the user.

13. The computer-assisted surgery system of claim 9 or 10, wherein the plurality of patient images are X-ray images.

14. The computer-assisted surgery system according to claim 9 or 10 further includes a user interface that allows a user to select a set of patient activities selected by the user and graphically displays the expected postoperative performance of the implant for the set of patient activities selected by the user in one or more polar coordinate graphs.

15. A computer-assisted surgery system according to any one of claims 9 or 10, wherein the first set of software instructions instructs the one or more processors to identify anatomical angles within the multiple patient images by automatically identifying landmark anatomical feature locations in the multiple patient images, and facilitates user manipulation of the placement of the landmark anatomical feature locations.

Images