Method and apparatus for improving a robotic surgical system

Abstract

A method, non-transitory computer-readable medium, and surgical computing device for improving a robotic surgical system are shown. Using this technique, one or more machine learning models are trained based on historical state data obtained for a computer-assisted surgical system (CASS) during each of a plurality of time periods during a plurality of historical knee replacement surgical procedures. One or more of the machine learning models are applied to initial state data of a current knee replacement surgical procedure to generate robotic commands required to achieve one or more future states of the CASS. The initial state data includes a surgical plan. One or more surgical tools of the CASS are then manipulated based on the robotic commands to achieve one or more future states of the CASS and thereby perform at least a portion of the surgical plan.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Applications 62 / 801,245 (filed Feb. 5, 2019), 62 / 801,257 (filed Feb. 5, 2019), 62 / 864,663 (filed Jun. 21, 2019), 62 / 885,673 (filed Aug. 12, 2019), and 62 / 939,946 (filed Nov. 25, 2019), the entireties of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure generally relates to methods, systems, and devices related to computer-assisted surgical systems, which include various hardware and software components that work together to enhance the surgical procedure. The disclosed techniques may be applied to, for example, shoulder, hip, and knee arthroplasty. BACKGROUND OF THE DISCLOSURE

[0004] Common types of joint arthroplasty, such as partial knee arthroplasty (PKA), total knee arthroplasty (TKA), or total hip arthroplasty (THA) utilize surgical plans to define one or more predetermined cutting planes for excising bone to accommodate the implant orientation and position (pose) of a knee or hip implant / replacement joint. By excising bone according to the surgical plan, the patient's bone can be shaped in a standardized, planned manner to receive a joint replacement implant with a given pose. The exact orientation and position of the joint replacement implant are typically planned according to a surgical plan developed prior to beginning the surgery. However, the surgeon will often modify the plan in the operating room based on information collected about the patient's joint. There are various systems to improve surgical planning and workflow, but there is still room for improvement. SUMMARY OF THE DISCLOSURE

[0005] This Summary is provided to comply with 37 C.F.R. § 1.73 and need only briefly indicate the nature and substance of the invention. It is to be understood that it is not intended to interpret or limit the scope or meaning of the disclosure.

[0006] Methods, non-transitory computer-readable media, and surgical computing devices for improving robotic surgical systems are shown. Using this technology, one or more machine learning models are trained based on historical state data obtained for a computer-assisted surgical system (CASS) during each of a plurality of time periods during a plurality of historical knee replacement surgical procedures. One or more of the machine learning models are applied to initial state data of a current knee replacement surgical procedure to generate robotic commands required to achieve one or more future states of the CASS. The initial state data includes a surgical plan. One or more surgical tools of the CASS are then manipulated based on the robotic commands to achieve one or more future states of the CASS and thereby perform at least a portion of the surgical plan.

[0007] According to some embodiments, a robotic arm is controlled to automatically manipulate one or more of the surgical tools of the CASS.

[0008] According to some embodiments, the initial state data further includes one or more of patient anatomy data, implant data, or one or more surgeon preferences.

[0009] According to some embodiments, the machine learning model is further trained based on patient data associated with one or more of the historical knee replacement surgical procedures.

[0010] According to some embodiments, a plurality of visualizations of a future state of the CASS are output to a display device.

[0011] According to some embodiments, the visualizations together depict aspects of an entire current knee replacement surgical procedure.

[0012] According to some embodiments, one or more of the visualizations are modified based on an application of one or more of the machine learning models to obtain planned future state data for the current knee replacement surgical procedure.

[0013] According to some embodiments, one or more of the visualizations are modified to illustrate the impact of the planned future state data on the surgical plan.

[0014] According to some embodiments, the visualizations include a plurality of images corresponding to one or more aspects of the current knee replacement surgical procedure during one or more time periods corresponding to one or more of the future states.

[0015] According to some embodiments, the visualizations together depict one or more changes in a patient's anatomy on which the current knee replacement surgical procedure is being performed during one or more of the time periods.

[0016] In accordance with certain embodiments, one or more of the images are transformed on the display device in response to the received input.

[0017] In accordance with certain embodiments, each of the future states corresponds to one of the time periods.

[0018] In accordance with certain embodiments, approval is obtained before manipulating one or more surgical tools of the CASS based on the robotic command. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various embodiments are illustrated by way of example in the figures of the drawings. Such embodiments are exemplary and are not intended to be exhaustive or exclusive embodiments of the subject matter of the present invention.

[0020] Figure 1 An exemplary computer-assisted surgery system for some embodiments is shown;

[0021] Figure 2A Examples of some control instructions that can be used by a surgical computer in accordance with some embodiments are shown;

[0022] Figure 2B Examples of some data that can be used by a surgical computer in accordance with some embodiments are shown;

[0023] Figure 2C A system diagram is shown that is an example of a cloud-based system used by a surgical computer in accordance with some embodiments;

[0024] Figure 3A A high-level overview of how a surgical computer can generate recommendations is provided in accordance with some embodiments;

[0025] Figure 3B An exemplary implant placement interface in accordance with some embodiments is shown;

[0026] Figure 3C An exemplary gap planning interface in accordance with some embodiments is shown;

[0027] Figure 3D An exemplary optimized parameterization interface in accordance with some embodiments is shown;

[0028] Figure 3E An exemplary response and rationale interface in accordance with some embodiments is shown;

[0029] Figure 4 A system diagram is provided that shows how optimization of surgical parameters can be performed in accordance with some embodiments;

[0030] Figure 5A -F provides an overview of a knee prediction equation in accordance with some embodiments;

[0031] Figure 6 is a flowchart showing a process that can perform optimization of a system of equations according to some embodiments;

[0032] Figure 7A-1 and 7A-2 7A - 3 and 7B provide an overview of an exemplary user interface for some embodiments;

[0033] Figure 8 provides an overview of an exemplary surgical patient care system for some embodiments;

[0034] Figure 9 provides an overview of a machine learning algorithm for some embodiments;

[0035] Figure 10 is a flowchart showing the operation of an exemplary surgical patient care system for some embodiments;

[0036] Figure 11A -B is a flowchart showing the operation of an exemplary surgical patient care system for some embodiments;

[0037] Figure 11C-1 and 11C-2 11C - 3 provide an overview of an exemplary user interface for implant placement for some embodiments;

[0038] Figure 12A-12C provides some outputs that anatomical modeling software can use to visually depict the results of modeling hip joint movement for some embodiments;

[0039] Figure 12D provides an exemplary visualization of a hip implant for some embodiments;

[0040] Figure 13 provides an example of augmented reality visualization in some embodiments;

[0041] Figure 14 is a system diagram showing an augmented reality visualization system for some embodiments;

[0042] Figure 15 is a flowchart showing the operation of an augmented reality system for some embodiments;

[0043] Figure 16 is a system diagram showing an augmented reality visualization system for some embodiments;

[0044] Figure 17A provides an example of augmented reality visualization in some embodiments;

[0045] Figure 17BProvides an example of 3D visualization in some embodiments;

[0046] Figure 18A Provides an example of 3D visualization in some embodiments;

[0047] Figure 18B Provides an example of 3D visualization in some embodiments;

[0048] Figure 19 Provides an example of a 3D model for visualizing knee components in some embodiments;

[0049] Figure 20 Is a system diagram showing an exemplary computing system for some embodiments;

[0050] Figure 21 Is a system diagram showing an exemplary computing system for some embodiments;

[0051] Figure 22 Is an anatomical diagram of hip joint geometry that can be used in some embodiments;

[0052] Figure 23A -B is a diagram of hip joint geometry within an x-ray image that can be used in some embodiments;

[0053] Figure 24 Is a flowchart showing an exemplary process for extracting anatomical landmarks and determining a surgical plan based on modeling performance for some embodiments;

[0054] Figure 25 Is a table depicting exemplary values of hip joint geometry for some embodiments;

[0055] Figure 26A -D is an exemplary user interface for some embodiments;

[0056] Figure 27 Depicts an exemplary combination of model results for different selected activities for a given geometry to display aggregated results;

[0057] Figure 28 Is a flowchart showing an exemplary method for creating a statistical model database based on modeling performance to assist in determining a surgical plan for some embodiments;

[0058] Figure 29 Is a flowchart showing an exemplary method for creating a surgical plan based on modeling performance using a statistical model database for some embodiments;

[0059] Figure 30 Is a flowchart showing an exemplary method for modifying a surgical plan based on modeling performance using a statistical model database for some embodiments;

[0060] Figure 31 are a pair of annotated x-ray images that illustrate exemplary knee geometries that may be used in some embodiments;

[0061] Figure 32 is a system diagram of an exemplary embodiment of a surgical system for some embodiments;

[0062] Figure 33 is a view of a surgical scenario using some of the techniques disclosed herein; and

[0063] Figure 34A -B shows the process of measuring a patient's knee joint at different flexion degrees. DETAILED DESCRIPTION

[0064] The present disclosure is not limited to the specific systems, devices, and methods described, as they may vary. The terms used in the description are for the purpose of describing particular versions or embodiments only and are not intended to limit the scope.

[0065] As used in this document, unless the context clearly dictates otherwise, the singular forms "a," "an," and "the" include plural referents. Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Nothing in the present disclosure should be construed as an admission that the embodiments described in the present disclosure are not entitled to antedate the date of the present disclosure by virtue of a prior invention. As used in this document, the term "comprising" means "including but not limited to."

[0066] The disclosed devices are particularly well-suited for surgical procedures that utilize a surgical navigation system (e.g., a surgical navigation system). Such procedures may include knee replacement and / or revision surgery as well as shoulder and hip surgery. NAVIO is a registered trademark of BLUE BELT TECHNOLOGIES, Inc. of Pittsburgh, Pennsylvania, a subsidiary of SMITH&NEPHEW, Inc. of Memphis, Tennessee.

[0067] DEFINITIONS

[0068] For the purposes of the present disclosure, the term "implant" is used to refer to a prosthetic device or structure manufactured to replace or augment a biological structure. For example, in total hip replacement surgery, a prosthetic acetabular cup (implant) is used to replace or augment a patient's worn or damaged acetabulum. Although the term "implant" is generally considered to denote an artificial structure (as opposed to a transplant), for the purposes of this specification, an implant may include biological tissue or material transplanted to replace or augment a biological structure.

[0069] For purposes of the present disclosure, the term "real-time" is used to refer to computations or operations that are performed instantaneously when an event occurs or when an input is received by an operating system. However, the use of the term "real-time" is not intended to exclude operations that introduce some latency between the input and the response, provided that the latency is an accidental result of the performance characteristics of the machine.

[0070] Although much of the present disclosure relates to surgeons or other medical specialists by specific job title or role, nothing in the present disclosure is intended to be limited to a specific job title or function. A surgeon or medical specialist can include any doctor, nurse, medical specialist, or technician. Any of these terms or positions can be used interchangeably with a user of the system disclosed herein, unless otherwise expressly specified. For example, in some embodiments, a reference to a surgeon can also apply to a technician or a nurse.

[0071] Overview of the CASS Ecosystem

[0072] Figure 1 A diagram of an example computer-assisted surgery system (CASS) 100 according to some embodiments is provided. As described in further detail in the following sections, the CASS uses computers, robotics, and imaging technologies to assist a surgeon in performing orthopedic surgical procedures, such as total knee arthroplasty (TKA) or total hip arthroplasty (THA). For example, a surgical navigation system can assist a surgeon in precisely positioning a patient's anatomy, guiding surgical instruments, and implanting medical devices. Surgical navigation systems such as CASS 100 often employ various forms of computational technology to perform a wide variety of standard and minimally invasive surgical procedures and techniques. Moreover, these systems allow a surgeon to more accurately plan, track, and navigate the position of instruments and implants relative to a patient's body, as well as perform pre-operative and intra-operative body imaging.

[0073] The actuator platform 105 positions surgical tools relative to a patient during a surgery. The exact components of the actuator platform 105 will vary depending on the embodiment employed. For example, for knee surgery, the actuator platform 105 can include an end effector 105B that holds a surgical tool or instrument during its use. The end effector 105B can be a hand-held device or instrument used by a surgeon (e.g., a handpiece or a cutting guide or a clamp), or alternatively, the end effector 105B can include a device or instrument held or positioned by a robotic arm 105A. Although in Figure 1A robotic arm 105A is shown, but in some embodiments, there can be multiple devices. As an example, there can be one robotic arm 105A on each side of the operating table or two devices on one side of the operating table. The robotic arm 105A can be directly mounted to the operating table, located on a floor platform (not shown) next to the operating table, mounted on a floor stand, or mounted on the walls or ceiling of the operating room. The floor platform can be fixed or movable. In one particular embodiment, the robotic arm 105A is mounted on a floor stand located between the patient's legs or feet. In some embodiments, the end effector 105B can include a suture holder or a stapler to assist in closing the wound. Additionally, in the case of two robotic arms 105A, the surgical computer 150 can drive the robotic arms 105A to work together to suture the wound during closure. Alternatively, the surgical computer 150 can drive one or more robotic arms 105A to suture the wound during closure.

[0074] The actuator platform 105 can include a limb positioner 105C for positioning a patient's limb during surgery. An example of the limb positioner 105C is the SMITH AND NEPHEW SPIDER2 TM system. The limb positioner 105C can be manually operated by a surgeon or, alternatively, change the limb position based on instructions received from the surgical computer 150 (described below). Although Figure 1 one limb positioner 105C is shown, in some embodiments there can be multiple devices. As an example, there can be one limb positioner 105C on each side of the operating table or two devices on one side of the operating table. The limb positioner 105C can be directly mounted to the operating table, located on a floor platform (not shown) next to the operating table, mounted on a pole, or mounted on the walls or ceiling of the operating room. In some embodiments, the limb positioner 105C can be used in an unconventional manner, such as a retractor or a specific bone holder. As an example, the limb positioner 105C can include an ankle boot, soft tissue clamps, bone clamps, or a soft tissue retractor spoon, such as a hooked, curved, or angled blade. In some embodiments, the limb positioner 105C can include a suture holder to assist in closing the wound.

[0075] The actuator platform 105 can include tools such as a screwdriver, a light or laser indicating an axis or plane, a level, a pin driver, a pin puller, a plane checker, an indicator, a finger, or some combination thereof.

[0076] The resection device 110 ( Figure 1Bone or tissue resection is performed using, for example, mechanical, ultrasonic, or laser techniques (not shown). Examples of resection device 110 include drilling devices, deburring devices, oscillating sawing devices, vibratory impact devices, reamers, ultrasonic bone cutting devices, radiofrequency ablation devices, reciprocating devices (such as files or broaches), and laser ablation systems. In some embodiments, resection device 110 is held and operated by a surgeon during surgery. In other embodiments, actuator platform 105 can be used to hold resection device 110 during use.

[0077] Actuator platform 105 may also include a cutting guide or fixture 105D for guiding a saw or drill used to resect tissue during surgery. Such a cutting guide 105D can be integrally formed as part of actuator platform 105 or robotic arm 105A, or the cutting guide can be a separate structure that can be matingly and / or removably attached to actuator platform 105 or robotic arm 105A. Actuator platform 105 or robotic arm 105A can be controlled by CASS 100 to position the cutting guide or fixture 105D near the patient's anatomy according to a pre-operative or intra-operative surgical plan such that the cutting guide or fixture will produce precise bone cuts according to the surgical plan.

[0078] Tracking system 115 uses one or more sensors to collect real-time position data for localizing the patient's anatomy and surgical instruments. For example, for a TKA procedure, tracking system 115 can provide the position and orientation of end effector 105B during the procedure. In addition to position data, data from tracking system 115 can also be used to infer the speed / acceleration of the anatomy / instrument, which can be used for tool control. In some embodiments, tracking system 115 can use an array of trackers attached to end effector 105B to determine the position and orientation of end effector 105B. The position of end effector 105B can be inferred based on the position and orientation of tracking system 115 and the known relationship in three-dimensional space between tracking system 115 and end effector 105B. Various types of tracking systems can be used in various embodiments of the present invention, including but not limited to infrared (IR) tracking systems, electromagnetic (EM) tracking systems, video- or image-based tracking systems, and ultrasonic registration and tracking systems. Using data provided by tracking system 115, surgical computer 150 can detect objects and prevent collisions. For example, surgical computer 150 can prevent robotic arm 105A from colliding with soft tissue.

[0079] Any suitable tracking system can be used to track surgical objects and patient anatomy in an operating room. For example, a combination of infrared and visible light cameras can be used in an array. Various lighting sources (such as infrared LED light sources) can illuminate the scene so that three-dimensional imaging can be performed. In some embodiments, this can include stereo, triscopic, tetravue, etc. imaging. In addition to the camera array fixed to the cart in some embodiments, additional cameras can be placed throughout the operating room. For example, a handheld tool or a headpiece worn by an operator / surgeon can include imaging capabilities to transmit images back to a central processor to correlate those images with the images acquired by the camera array. This can provide a more robust image for an environment modeled using multiple perspectives. Additionally, some imaging devices can have a suitable resolution on the scene or have a suitable viewing angle to pick up information stored in a Quick Response (QR) code or barcode. This helps to identify specific objects that are not manually registered with the system. In some embodiments, the cameras can be mounted on the robotic arm 105A.

[0080] In some embodiments, the surgeon can manually register specific objects with the system preoperatively or intraoperatively. For example, by interacting with a user interface, the surgeon can identify the starting position of a tool or a bone structure. By tracking fiducial markers associated with the tool or the bone structure, or by using other conventional image tracking methods, the processor can track the tool or the bone as it moves through the environment in a three-dimensional model.

[0081] In some embodiments, certain markers such as fiducial markers for identifying individuals, important tools, or bones in an operating room can include passive or active identifiers that can be picked up by a camera or a camera array associated with the tracking system. For example, an infrared LED can flash a pattern that conveys a unique identifier to the source of the pattern, thus providing a dynamic identification marker. Similarly, one-dimensional or two-dimensional optical codes (barcodes, QR codes, etc.) can be fixed to objects in the operating room to provide passive identification that can occur based on image analysis. If these codes are placed asymmetrically on the object, they can also be used to determine the orientation of the object by comparing the position of the identifier with the extent of the object in the image. For example, a QR code can be placed in the corner of a tool tray, thus allowing the orientation and identification of the tray to be tracked. Other tracking methods will be described throughout the text. For example, in some embodiments, surgeons and other personnel can wear augmented reality headpieces to provide additional camera angles and tracking capabilities.

[0082] In addition to optical tracking, certain features of an object can be tracked by registering the physical properties of the object and associating them with an object that can be tracked, such as fiducial markers fixed to a tool or bone. For example, a surgeon can perform a manual registration process, whereby the tracked tool and the tracked bone can be manipulated relative to each other. By striking the tip of the tool against the surface of the bone, a three-dimensional surface can be mapped for the bone, which is associated with the position and orientation of a reference frame relative to the fiducial marker. By optically tracking the position and orientation (pose) of the fiducial marker associated with the bone, the model of the surface can be tracked in the environment by extrapolation.

[0083] The registration process of registering the CASS 100 to the relevant anatomy of a patient may also involve using anatomical landmarks, such as landmarks on bone or cartilage. For example, the CASS 100 may include a 3D model of the relevant bone or joint, and the surgeon can use a probe attached to the CASS to collect data on the position of bone landmarks on the patient's actual bone during the operation. Bone landmarks can include, for example, the medial and lateral malleoli, the ends of the proximal femur and distal tibia, and the center of the hip joint. The CASS 100 can compare and register the position data of the bone landmarks collected by the surgeon with the probe with the position data of the same landmarks in the 3D model. Alternatively, the CASS 100 can construct a 3D model of a bone or joint without preoperative image data by using the position data of bone landmarks and bone surfaces collected by the surgeon using the CASS probe or other means. The registration process can also include determining the respective axes of the joint. For example, for TKA, the surgeon can use the CASS 100 to determine the anatomical and mechanical axes of the femur and tibia. The surgeon and the CASS 100 can identify the center of the hip joint by moving the patient's leg in a circular motion (i.e., circumduction) in a spiral direction so that the CASS can determine the position of the hip joint center.

[0084] The tissue navigation system 120 ( Figure 1 not shown) provides intraoperative real-time visualization of the patient's bone, cartilage, muscle, nerve, and / or vascular tissue around the surgical area. Examples of systems that can be used for tissue navigation include fluorescence imaging systems and ultrasound systems.

[0085] The display 125 provides a graphical user interface (GUI) that displays images collected by the tissue navigation system 120 and other information related to the surgery. For example, in one embodiment, the display 125 overlays image information collected preoperatively or intraoperatively from various modalities (e.g., CT, MRI, X-ray, fluoroscopy, ultrasound, etc.) to provide the surgeon with various views of the patient's anatomy and real-time status. The display 125 can include, for example, one or more computer monitors. As an alternative or supplement to the display 125, one or more of the surgical staff can wear an augmented reality (AR) head-mounted device (HMD). For example, in Figure 1 the surgeon 111 wears the AR HMD 155, which can, for example, overlay preoperative image data on the patient or provide surgical planning suggestions. Various exemplary uses of the AR HMD 155 in surgical procedures are described in detail in the following sections.

[0086] The surgical computer 150 provides control instructions to the various components of the CASS 100, collects data from those components, and provides general processing for the various data required during the surgery. In some embodiments, the surgical computer 150 is a general-purpose computer. In other embodiments, the surgical computer 150 can be a parallel computing platform that uses multiple central processing units (CPUs) or graphics processing units (GPUs) to perform processing. In some embodiments, the surgical computer 150 is connected to a remote server via one or more computer networks (e.g., the Internet). The remote server can be used, for example, for data storage or execution of computationally intensive processing tasks.

[0087] Various techniques known in the art can be used to connect the surgical computer 150 to the other components of the CASS 100. Moreover, the computers can use a variety of techniques to connect to the surgical computer 150. For example, the end effector 105B can be connected to the surgical computer 150 via a wired (i.e., serial) connection. The tracking system 115, the tissue navigation system 120, and the display 125 can similarly be connected to the surgical computer 150 using a wired connection. Alternatively, the tracking system 115, the tissue navigation system 120, and the display 125 can be connected to the surgical computer 150 using wireless techniques such as, but not limited to, Wi-Fi, Bluetooth, near field communication (NFC), or ZigBee.

[0088] Power Impactor and Acetabular Reamer Device

[0089] As described above with respect to Figure 1Part of the flexibility of the described CASS design lies in the ability to add additional or alternative devices to the CASS 100 as needed to support specific surgical procedures. For example, in the case of hip surgery, the CASS 100 can include a powered impact device. The impact device is designed to repeatedly apply an impact force that a surgeon can use to perform activities such as implant alignment. For example, in total hip arthroplasty (THA), a surgeon typically uses an impact device to insert a prosthetic acetabular cup into the acetabulum of the implant host. Although the impact device can be manual in nature (e.g., operated by a surgeon striking an impactor with a mallet), a powered impact device is generally easier and faster to use in a surgical setting. The powered impact device can be powered, for example, using a battery attached to the device. Various attachments can be connected to the powered impact device to allow the impact force to be directed in various ways as needed during surgery. Also in the case of hip surgery, the CASS 100 can include a powered, robotically controlled end effector to ream the acetabulum to accommodate the acetabular cup implant.

[0090] In robot-assisted THA, CT or other image data, identification of anatomical landmarks, a tracker array attached to the patient's bone, and one or more cameras can be used to register the patient's anatomy to the CASS 100. The tracker array can be mounted on the iliac crest using clamps and / or bone screws, and such a tracker array can be mounted externally through the skin or internally (posterolateral or anterolateral) through an incision made to perform THA. For THA, the CASS 100 can utilize one or more femoral cortical screws inserted into the proximal femur as checkpoints to assist in the registration process. The CASS 100 can also utilize one or more checkpoint screws inserted into the pelvis as additional checkpoints to assist in the registration process. The femoral tracker array can be fixed or mounted in the femoral cortical screws. The CASS 100 can employ a step in which verification is performed using a probe precisely placed by the surgeon on key areas of the proximal femur and pelvis identified for the surgeon on the display 125. The tracker can be located on the robotic arm 105A or the end effector 105B to register the arm and / or the end effector to the CASS 100. The verification step can also utilize proximal and distal femoral checkpoints. The CASS 100 can utilize color cues or other cues to inform the surgeon that the registration process of the bone and the robotic arm 105A or the end effector 105B has been verified to a certain degree of accuracy (e.g., within 1 mm).

[0091] For THA, CASS 100 may include broach tracking selection using a femoral array to allow the surgeon to obtain the position and orientation of the broach intraoperatively and calculate the patient's hip length and offset values. Based on the information provided regarding the patient's hip joint and the planned implant position and orientation after broach tracking, the surgeon can modify or adjust the surgical plan.

[0092] For robot-assisted THA, CASS 100 may include one or more powered reamers connected or attached to the robotic arm 105A or the end effector 105B that prepare the pelvic bone to receive the acetabular implant according to the surgical plan. The robotic arm 105A and / or the end effector 105B may notify the surgeon and / or control the power of the reamer to ensure that the acetabulum is resected (reamed) according to the surgical plan. For example, if the surgeon attempts to resect bone outside the boundaries of the bone to be resected according to the surgical plan, CASS 100 may cut off the power of the reamer or instruct the surgeon to cut off the power of the reamer. CASS 100 may provide the surgeon with the option to turn off or disengage the robotic control of the reamer. Compared to using different colored surgical plans, the display 125 may show the progress of the bone being resected (reamed). The surgeon may view the display of the bone being resected (reamed) to guide the reamer to complete reaming according to the surgical plan. CASS 100 may provide the surgeon with visual or auditory cues to warn the surgeon of a resection that is not in accordance with the surgical plan.

[0093] After reaming, CASS 100 may employ a manual or powered impactor attached or connected to the robotic arm 105A or the end effector 105B to impact the trial implant and the final implant into the acetabulum. The robotic arm 105A and / or the end effector 105B may be used to guide the impactor to impact the trial implant and the final implant into the acetabulum according to the surgical plan. CASS 100 may display the position and orientation of the trial implant and the final implant relative to the bone to inform the surgeon how to compare the orientation and position of the trial implant and the final implant with the surgical plan, and the display 125 may show the position and orientation of the implant as the surgeon manipulates the leg and hip. If the surgeon is not satisfied with the initial implant position and orientation, CASS 100 may provide the surgeon with the option to re-plan and redo the reaming and implant impaction by preparing a new surgical plan.

[0094] Before surgery, CASS 100 can formulate a proposed surgical plan based on a three-dimensional model of the hip joint and other patient-specific information (such as the mechanical and anatomical axes of the leg bone, the epicondylar axis, the femoral neck axis, the dimensions of the femur and hip (e.g., length), the midline axis of the hip joint, the ASIS axis of the hip joint, and the positions of anatomical landmarks such as the lesser trochanter landmark, the distal landmark, and the center of rotation of the hip joint). The surgical plan formulated by CASS can provide the recommended optimal implant size and the position and orientation of the implant according to the three-dimensional model of the hip joint and other patient-specific information. The surgical plan formulated by CASS can include recommended details regarding offset values, inclination and anteversion values, center of rotation, cup size, medialization value, superior-inferior fit, femoral stem size and length.

[0095] For THA, the surgical plan formulated by CASS can be viewed preoperatively and intraoperatively, and the surgeon can modify the surgical plan formulated by CASS preoperatively or intraoperatively. The surgical plan formulated by CASS can show the planned hip resection, and the planned implant can be superimposed on the hip joint according to the planned resection. CASS 100 can provide the surgeon with options for different surgical procedures, and the options will be presented to the surgeon according to the surgeon's preferences. For example, the surgeon can select from different workflows based on the number and type of anatomical landmarks examined and acquired and / or the position and number of tracker arrays used during the registration process.

[0096] According to some embodiments, the powered impact device used with CASS 100 can be operated in a variety of different settings. In some embodiments, the surgeon adjusts the settings via a manual switch or other physical mechanism on the powered impact device. In other embodiments, a digital interface can be used, which allows, for example, setting input via a touch screen on the powered impact device. Such a digital interface can allow the available settings to vary based on, for example, the type of attachment connected to the power attachment device. In some embodiments, the settings can be changed by communicating with a robot or other computer system within CASS 100 rather than adjusting the settings on the powered impact device itself. A Bluetooth or Wi-Fi networking module on the powered impact device, for example, can be used to establish such a connection. In another embodiment, the impact device and the end component can contain features that allow the impact device to know what end component (cup impacter, broach handle, etc.) is attached without the surgeon having to take any action, and the settings are adjusted accordingly. This can be achieved, for example, by QR codes, barcodes, RFID tags, or other methods.

[0097] Examples of settings that can be used include cup impact settings (e.g., unidirectional, specified frequency range, specified force and / or energy range); broach impact settings (e.g., bidirectional / oscillating within a specified frequency range, specified force and / or energy range); femoral head impact settings (e.g., unidirectional / single strike at a specified force or energy); and dry impact settings (e.g., unidirectional at a specified frequency with a specified force or energy). Additionally, in some embodiments, the powered impact device includes settings related to impacting the acetabular liner (e.g., unidirectional / single strike at a specified force or energy). There may be multiple settings for each type of liner (e.g., polymeric, ceramic, oxinium, or other materials). Further, the powered impact device can provide settings for different bone qualities based on preoperative testing / imaging / knowledge and / or the surgeon's intraoperative assessment. In some embodiments, the powered impact device can have a dual function. For example, the powered impact device can not only provide reciprocating motion to provide impact force, but also provide reciprocating motion for a broach or rasp.

[0098] In some embodiments, the powered impact device includes a feedback sensor that collects data during use of the instrument and sends the data to a computing device, such as a controller within the device or a surgical computer 150. The computing device can then record the data for later analysis and use. Examples of data that can be collected include but are not limited to acoustic waves, the predetermined resonance frequency of each instrument, the reaction force or rebound energy from the patient's bone, the position of the device relative to the registered bone anatomy in imaging (e.g., fluoroscopy, CT, ultrasound, MRI, etc.), and / or external strain gauges on the bone.

[0099] Once the data is collected, the computing device can execute one or more algorithms in real-time or near real-time to assist the surgeon in performing the surgical procedure. For example, in some embodiments, the computing device uses the collected data to derive information such as the correct final broach size (femur); when the stem is fully seated (femoral side); or when the cup is seated for THA (depth and / or orientation). Once this information is known, it can be displayed for the surgeon to view, or it can be used to activate a tactile or other feedback mechanism to guide the surgical procedure.

[0100] Additionally, the data derived from the aforementioned algorithms can be used to drive the operation of the device. For example, during insertion of the prosthetic acetabular cup with the powered impact device, the device can automatically extend the impact head (e.g., end effector), move the implant to the appropriate position, or turn off the power of the device once the implant is fully seated. In one embodiment, the derived information can be used to automatically adjust the settings for bone quality, where the powered impact device should use less power to reduce the risk of femoral / acetabular / pelvic fractures or damage to surrounding tissues.

[0101] Robot arm

[0102] In some embodiments, CASS 100 includes a robotic arm 105A that serves as an interface for stabilizing and holding various instruments used during a surgical procedure. For example, in the case of hip surgery, these instruments can include, but are not limited to, retractors, sagittal or reciprocating saws, reamer handles, cup impactors, broach handles, and stem inserters. The robotic arm 105A can have multiple degrees of freedom (similar to the Spider device) and has the ability to lock in place (e.g., by pressing a button, voice activation, the surgeon removing their hand from the robotic arm, or other means).

[0103] In some embodiments, the movement of the robotic arm 105A can be achieved by using a control panel built into the robotic arm system. For example, the display screen can include one or more input sources, such as physical buttons that direct the movement of the robotic arm 105A or a user interface with one or more icons. The surgeon or other healthcare professional can engage with one or more input sources during the surgical procedure to position the robotic arm 105A.

[0104] A tool or end effector 105B can be attached or integrated into the robotic arm 105A, which can include, but are not limited to, a deburring device, a scalpel, a cutting device, a retractor, a joint tensioning device, etc. In embodiments where the end effector 105B is used, the end effector 105B can be positioned at the end of the robotic arm 105A such that any motor control operations are performed within the robotic arm system. In embodiments where a tool is used, the tool can be fixed at the distal end of the robotic arm 105A, but the motor control operations can be located within the tool itself.

[0105] The robotic arm 105A can be motorized internally to stabilize the robotic arm, thereby preventing it from falling and hitting the patient, the operating table, the surgical staff, etc., and allowing the surgeon to move the robotic arm without having to fully support its weight. While the surgeon is moving the robotic arm 105A, the robotic arm can provide some resistance to prevent the robotic arm from moving too quickly or activating too many degrees of freedom at once. The position and locked state of the robotic arm 105A can be tracked, for example, by a controller or a surgical computer 150.

[0106] In some embodiments, the robotic arm 105A can be moved to its desired position and orientation for the task being performed by hand (e.g., by a surgeon) or with an internal motor. In some embodiments, the robotic arm 105A can be capable of operating in a “free” mode, allowing the surgeon to position the arm in a desired location without restriction. In the free mode, as described above, the position and orientation of the robotic arm 105A can still be tracked. In one embodiment, during a specified portion of the surgical plan being tracked by the surgical computer 150, certain degrees of freedom can be selectively released when there is input from a user (e.g., a surgeon). A design in which the robotic arm 105A is powered internally by hydraulics or a motor or provides resistance to external manual movement by similar means can be described as a powered robotic arm, while an arm that is manually maneuvered without power feedback but can be manually or automatically locked in place can be described as a passive robotic arm.

[0107] The robotic arm 105A or the end effector 105B can include a trigger or other device to control the power of a saw or drill. Engagement of the trigger or other device by the surgeon can cause the robotic arm 105A or the end effector 105B to transition from a motorized alignment mode to a mode in which the saw or drill is engaged and powered on. Additionally, the CASS 100 can include a foot pedal (not shown) that, when activated, causes the system to perform certain functions. For example, the surgeon can activate the foot pedal to instruct the CASS 100 to place the robotic arm 105A or the end effector 105B in an automatic mode that positions the robotic arm or end effector relative to the patient's anatomy for performing the necessary resection. The CASS 100 can also place the robotic arm 105A or the end effector 105B in a collaborative mode that allows the surgeon to manually maneuver the robotic arm or end effector and position it in a particular location. The collaborative mode can be configured to allow the surgeon to move the robotic arm 105A or the end effector 105B medially or laterally while restricting movement in other directions. As discussed, the robotic arm 105A or the end effector 105B can include a cutting device (saw, drill, and sharpener) or a cutting guide or jig 105D that will guide the cutting device. In other embodiments, the movement of the robotic arm 105A or the robot-controlled end effector 105B can be completely controlled by the CASS 100 without any assistance or input from a surgeon or other medical professional, or with only minimal assistance or input. In still other embodiments, a surgeon or other medical professional can use a control mechanism separate from the robotic arm or robot-controlled end effector device, such as a joystick or an interactive monitor or display control device, to remotely control the movement of the robotic arm 105A or the robot-controlled end effector 105B.

[0108] The following examples describe the use of a robotic device in the case of hip surgery; however, it should be understood that the robotic arm may have other applications in surgical procedures involving the knee, shoulder, etc. An example of the use of a robotic arm in forming an anterior cruciate ligament (ACL) graft tunnel is described in PCT / US2019 / 048502, titled "Robotic Assisted Ligament Graft Placement and Tensioning," filed on August 28, 2019, the entire content of which is incorporated herein by reference.

[0109] The robotic arm 105A can be used to hold a retractor. For example, in one embodiment, the surgeon can move the robotic arm 105A to a desired position. At this time, the robotic arm 105A can be locked in place. In some embodiments, the robotic arm 105A is provided with data regarding the patient's position such that if the patient moves, the robotic arm can adjust the retractor position accordingly. In some embodiments, multiple robotic arms can be used, thereby allowing multiple retractors to be held or more than one action to be performed simultaneously (e.g., retractor holding and reaming).

[0110] The robotic arm 105A can also be used to assist in stabilizing the surgeon's hand during the creation of a femoral neck incision. In this application, certain restrictions can be imposed on the control of the robotic arm 105A to prevent soft tissue damage. For example, in one embodiment, the surgical computer 150 tracks the position of the robotic arm 105A while it is operating. If the tracked position approaches an area where tissue damage is predicted, a command can be sent to the robotic arm 105A to cause it to stop. Alternatively, in the case where the robotic arm 105A is automatically controlled by the surgical computer 150, the surgical computer can ensure that no instructions are provided to the robotic arm that would cause it to enter an area where soft tissue damage may occur. The surgical computer 150 can impose certain restrictions on the surgeon to prevent the surgeon from reaming too deeply into the medial wall of the acetabulum or reaming at an incorrect angle or orientation.

[0111] In some embodiments, the robotic arm 105A can be used to hold a cup impacter at a desired angle or orientation during cup impaction. When the final position has been reached, the robotic arm 105A can prevent any further seating to prevent damage to the pelvis.

[0112] The surgeon can use the robotic arm 105A to position a broach handle in a desired position and allow the surgeon to impact the broach into the femoral canal in a desired orientation. In some embodiments, once the surgical computer 150 receives feedback that the broach is fully seated, the robotic arm 105A can restrict the handle to prevent further advancement of the broach.

[0113] The robotic arm 105A can also be used for surface reconstruction applications. For example, the robotic arm 105A can stabilize the surgeon's hand while using traditional instruments and provide certain constraints or limitations to allow for proper placement of implant components (e.g., guide wire placement, chamfer cutter, sleeve cutter, flat cutter, etc.). In the case of using only a bone drill, the robotic arm 105A can stabilize the surgeon's handpiece and can apply limitations to the handpiece to prevent the surgeon from removing unwanted bone in violation of the surgical plan.

[0114] The robotic arm 105A can be a passive arm. As an example, the robotic arm 105A can be the CIRQ robotic arm available from Brainlab AG. CIRQ is a registered trademark of Brainlab AG, Olof-Palme-Str. 98 1829 Munich, Germany. In a particular embodiment, the robotic arm 105A is an intelligent grasping arm, as disclosed in U.S. Patent Application No. 15 / 525,585 to Krinninger et al., U.S. Patent Application No. 15 / 561,042 to Nowatschin et al., U.S. Patent No. 15 / 561,048 to Nowatschin et al., and U.S. Patent No. 10,342,636 to Nowatschin et al., the entire contents of each of the above patents being incorporated herein by reference.

[0115] Generation and collection of surgical procedure data

[0116] The various services provided by medical professionals for treating a clinical condition are collectively referred to as the "care episode". For a particular surgical procedure, the care episode can include three phases: pre-operative, intra-operative, and post-operative. During each phase, data that can be used to analyze the care episode is collected or generated in order to understand various aspects of the procedure and identify patterns that can be used, for example, in training models to make decisions with minimal human intervention. The data collected during the care episode can be stored as a complete data set at the surgical computer 150 or the surgical data server 180 (shown in Figure 2). Thus, for each care episode, there is a data set that includes all the data collectively collected pre-operatively about the patient, all the data collected or stored intra-operatively by the CASS 100, and any post-operative data provided by the patient or by the medical professionals monitoring the patient.

[0117] As further detailed, data collected during the care period can be used to enhance the performance of a surgical procedure or provide an overall understanding of the surgical procedure and patient outcomes. For example, in some embodiments, data collected during the care period can be used to generate a surgical plan. In one embodiment, when collecting data during surgery, an advanced preoperative plan is refined intraoperatively. In this way, the surgical plan can be considered to change in real-time or near real-time as new data is collected by components of the CASS 100. In other embodiments, preoperative images or other input data can be used to develop a robust plan that is simply executed during surgery. In this case, data collected by the CASS 100 during surgery can be used to make recommendations to ensure that the surgeon stays within the preoperative surgical plan. For example, if a surgeon is unsure how to achieve certain prescribed cuts or implant alignments, the surgical computer 150 can be queried for recommendations. In still other embodiments, preoperative and intraoperative planning scenarios can be combined such that a refined preoperative plan can be dynamically modified as needed or desired during the surgical procedure. In some embodiments, biomechanics-based models of the patient's anatomy contribute simulation data to be considered by the CASS 100 in formulating preoperative, intraoperative, and postoperative / rehabilitation procedures to optimize the patient's implant performance outcomes.

[0118] In addition to changing the surgical procedure itself, data collected during the care period can also be used as input for other surgical assistance procedures. For example, in some embodiments, care period data can be used to design implants. Example data-driven techniques for designing, sizing, and fitting implants are described in U.S. Patent Application No. 13 / 814,531, filed August 15, 2011, entitled "Systems and Methods for Optimizing Parameters for Orthopaedic Procedures"; U.S. Patent Application No. 14 / 232,958, filed July 20, 2012, entitled "Systems and Methods for Optimizing Fit of an Implant to Anatomy"; and U.S. Patent Application No. 12 / 234,444, filed September 19, 2008, entitled "Operatively Tuning Implants for Increased Performance", the entire contents of each of the above patents are hereby incorporated by reference into this patent application.

[0119] In addition, the data can be used for educational, training, or research purposes. For example, using the following inFigure 2C In the web-based solution described, other doctors or students can remotely view the surgery in the interface, enabling them to selectively view data collected from various components of the CASS 100. After the surgical procedure, a similar interface can be used to "replay" the surgery for training or other educational purposes, or to identify the root cause of any problems or complications during the surgery.

[0120] Data obtained during the preoperative phase typically includes all information collected or generated before the surgery. Thus, for example, information about the patient can be obtained from a patient intake form or an electronic medical record (EMR). Examples of patient information that can be collected include, but are not limited to, patient demographics, diagnosis, medical history, medical records, vital signs, historical information, allergies, and laboratory test results. Preoperative data can also include images related to the anatomical region of interest. These images can be obtained, for example, using magnetic resonance imaging (MRI), computed tomography (CT), X-rays, ultrasound, or any other means known in the art. Preoperative data can also include quality-of-life data obtained from the patient. For example, in one embodiment, preoperative patients use a mobile application ("app") to answer a questionnaire about their current quality of life. In some embodiments, the preoperative data used by the CASS 100 includes information about the patient's demographics, anthropometrics, culture, or other specific characteristics, which can be matched with activity levels and specific patient activities to customize the surgical plan for the patient. For example, people of certain cultures or demographics may be more likely to use a squat toilet on a daily basis.

[0121] Figure 2A and 2B Examples of data that can be obtained during the intraoperative phase of the care period are provided. These examples are based on the various components of the CASS 100 described above with reference to Figure 1 However, it should be understood that other types of data can be used based on the type of equipment used during the surgery and its use.

[0122] Figure 2A FIG. shows an example of some control instructions provided by the surgical computer 150 to other components of the CASS 100 according to some embodiments. It should be noted that Figure 2A the example assumes that the components of the actuator platform 105 are all directly controlled by the surgical computer 150. In embodiments where the components are manually controlled by the surgeon 111, instructions can be provided on the display 125 or the AR HMD 155 (as shown in Figure 1 ) to indicate to the surgeon 111 how to move the components.

[0123] The various components included in the actuator platform 105 are controlled by a surgical computer 150, which provides position instructions that indicate the positions at which the components move within a coordinate system. In some embodiments, the surgical computer 150 provides instructions to the actuator platform 105 that define how to react when a component of the actuator platform 105 deviates from the surgical plan. These commands are referred to as "haptic" commands in Figure 2A . For example, the end effector 105B can provide a force to resist movement outside the planned resection area. Other commands that the actuator platform 105 can use include vibration and audio cues.

[0124] In some embodiments, the end effector 105B of the robotic arm 105A is operatively coupled to a cutting guide 105D, as shown in Figure 1 . In response to an anatomical model of the surgical scenario, the robotic arm 105A can move the end effector 105B and the cutting guide 105D to appropriate positions to match the positions of the femoral or tibial cuts to be made according to the surgical plan. This can reduce the likelihood of errors, thus allowing a vision system and a processor utilizing the vision system to implement the surgical plan to place the cutting guide 105D in an exact position and orientation relative to the tibia or femur to align the slot of the cutting guide with the cut to be performed according to the surgical plan. The surgeon can then use any suitable tool, such as a vibrating or rotary saw or drill, to perform the cut (or drill) with perfect placement and orientation, since the tool is mechanically constrained by the features of the cutting guide 105D. In some embodiments, the cutting guide 105D can include one or more pin holes that the surgeon uses to drill and screw or pin the cutting guide into place before using the cutting guide to excise patient tissue. This can free the robotic arm 105A or ensure that the cutting guide 105D is fully secured and does not move relative to the bone to be resected. For example, this procedure can be used to create the first distal cut of the femur during a total knee replacement. In some embodiments, in the case where the joint replacement is a hip replacement, the cutting guide 105D can be fixed to the femoral head or acetabulum for the corresponding hip replacement resection. It should be understood that any joint replacement utilizing precise cuts can be performed in this manner using the robotic arm 105A and / or the cutting guide 105D.

[0125] The resection device 110 is provided with a variety of commands to perform bone or tissue operations. Similar to the actuator platform 105, position information can be provided to the resection device 110 to specify where it should be positioned when performing a resection. Other commands provided to the resection device 110 can depend on the type of resection device. For example, for mechanical or ultrasonic resection tools, the commands can specify the speed and frequency of the tool. For radiofrequency ablation (RFA) and other laser ablation tools, these commands can specify the intensity and pulse duration.

[0126] Some components of the CASS 100 do not need to be directly controlled by the surgical computer 150; rather, the surgical computer 150 only needs to activate the components, which then execute software locally to specify the manner in which data is collected and provided to the surgical computer 150. In Figure 2A the example of, there are two components that operate in this manner: the tracking system 115 and the tissue navigation system 120.

[0127] The surgical computer 150 provides any visualization required by the surgeon 111 during the surgery to the display 125. For the monitor, the surgical computer 150 can use techniques known in the art to provide instructions for displaying images, GUIs, etc. The display 125 can include various aspects of the workflow of the surgical plan. For example, during the registration process, the display 125 can display the preoperatively constructed 3D bone model and show the position of the probe when the surgeon uses the probe to collect the positions of the anatomical landmarks on the patient. The display 125 can include information about the surgical target area. For example, in conjunction with TKA, the display 125 can show the mechanical and anatomical axes of the femur and tibia. The display 125 can show the varus and valgus angles of the knee joint based on the surgical plan, and the CASS 100 can show how the expected corrections to the surgical plan will affect such angles. Therefore, the display 125 is an interactive interface that can dynamically update and display how changes to the surgical plan will affect the procedure and the final position and orientation of the implant installed on the bone.

[0128] As the workflow progresses to the preparation for bone cutting or resection, the display 125 can show the planned or recommended bone cuts before any cutting is performed. The surgeon 111 can manipulate the image display to provide different anatomical views of the target area and can have the option to change or correct the planned bone cuts based on the intraoperative assessment of the patient. The display 125 can show how the selected implant will be installed on the bone if the planned bone cuts are performed. If the surgeon 111 chooses to change a previously planned bone cut, the display 125 can show how the modified bone cut will change the position and orientation of the implant when installed on the bone.

[0129] The display 125 can provide the surgeon 111 with various data and information regarding the patient, the planned surgical procedure, and the implant. Various patient-specific information can be displayed, including real-time data regarding the patient's health, such as heart rate, blood pressure, etc. The display 125 can also include information regarding the anatomy of the surgical target area, including the location of landmarks, the current state of the anatomy (e.g., whether any resections have been performed, the depth and angle of planned and executed bone cuts), and the future state of the anatomy as the surgical plan progresses. The display 125 can also provide or show additional information regarding the surgical target area. For a TKA, the display 125 can provide information regarding the gap between the femur and the tibia (e.g., gap balance) and how such a gap will change if the planned surgical procedure is executed. For a TKA, the display 125 can provide additional relevant information regarding the knee joint, such as data regarding the tension of the joint (e.g., ligament laxity) and information regarding the rotation and alignment of the joint. The display 125 can show how the positioning and location of the planned implant will affect the patient when the knee joint is flexed. The display 125 can show how the use of different implants or the use of the same implant in different sizes will affect the surgical plan and can preview how such implants will be positioned on the bone. The CASS 100 can provide such information for each planned osteotomy in a TKA or THA. In a TKA, the CASS 100 can provide robotic control for one or more planned osteotomies. For example, the CASS 100 can only provide robotic control for the initial distal femoral cut, and the surgeon 111 can perform the other resections (anterior, posterior, and chamfer cuts) manually using conventional means (e.g., a 4-in-1 cutting guide or jig 105D).

[0130] The display 125 can use different colors to inform the surgeon of the status of the surgical plan. For example, unresected bone can be displayed in a first color, resected bone can be displayed in a second color, and planned resections can be displayed in a third color. Implants can be overlaid on the bone in the display 125, and the implant color can change or correspond to different types or sizes of implants.

[0131] The information and options presented on the display 125 can vary according to the type of surgical procedure being performed. Additionally, the surgeon 111 can request or select a specific surgical workflow display that matches or aligns with his or her surgical planning preferences. For example, for a surgeon 111 who typically performs tibial cutting before femoral cutting in a TKA, the display 125 and associated workflow can be adapted to account for this preference. The surgeon 111 can also pre-select to include or remove certain steps from the standard surgical workflow display. For example, if the surgeon 111 uses resection measurements to finalize the implant plan but does not analyze ligament gap balance when finalizing the implant plan, the surgical workflow display can be organized into modules, and the surgeon can select the modules to be displayed and the order in which the modules are presented based on the surgeon's preferences or the circumstances of a particular surgery. For example, modules related to ligament and gap balance can include pre-resection and post-resection ligament / gap balance, and the surgeon 111 can select which modules to include in their default surgical planning workflow based on whether such ligament and gap balance is performed before or after (or before and after) the osteotomy is carried out.

[0132] For more specialized display devices, such as an AR HMD, the surgical computer 150 can use a device-supported data format to provide images, text, etc. For example, if the display 125 is a holographic device such as a Microsoft HoloLens TM or Magic LeapOne TM the surgical computer 150 can use the HoloLens application programming interface (API) to send commands that specify the position and content of the holograms displayed in the surgeon 111's field of view.

[0133] In some embodiments, one or more surgical planning models can be incorporated into the CASS 100 and used in the formulation of the surgical plan provided to the surgeon 111. The term "surgical planning model" refers to software that simulates the biomechanical properties of anatomical structures in various scenarios to determine the optimal way to perform cuts and other surgical activities. For example, for a knee replacement surgery, the surgical planning model can measure parameters of functional activities, such as deep knee bend, gait, etc., and select the cutting positions on the knee to optimize implant placement. An example of a surgical planning model is the LIFEMOD TM simulation software from SMITH AND NEPHEW. In some embodiments, the surgical computer 150 includes a computational architecture (e.g., a GPU-based parallel processing environment) that allows for the full execution of the surgical planning model during surgery. In other embodiments, the surgical computer 150 can be connected via a network to a remote computer that allows such execution, such as a surgical data server 180 (see Figure 2C)。As an alternative to full execution of the surgical planning model, in some embodiments, a set of transfer functions is derived that simplifies the mathematical operations obtained by the model into one or more prediction equations. Then, instead of performing a full simulation during the surgery, the prediction equations are used. More details regarding the use of the transfer functions are described in PCT / US2019 / 046995, titled "Patient Specific Surgical Method and System," filed on August 19, 2019, the entire content of which is incorporated herein by reference.

[0134] Figure 2B FIG. shows an example of some types of data that can be provided to the surgical computer 150 from the various components of the CASS 100. In some embodiments, the components can transmit data streams to the surgical computer 150 in real time or near real time during the surgery. In other embodiments, the components can queue the data and send it to the surgical computer 150 at set intervals (e.g., per second). Any format known in the art can be used to transmit the data. Thus, in some embodiments, all components transmit data to the surgical computer 150 in a common format. In other embodiments, each component can use a different data format, and the surgical computer 150 is configured with one or more software applications capable of converting the data.

[0135] Generally, the surgical computer 150 can be used as a central point for collecting CASS data. The exact content of the data will depend on the source. For example, each component of the actuator platform 105 provides a measured position to the surgical computer 150. Thus, by comparing the measured position with the position initially specified by the surgical computer 150, the surgical computer can identify deviations that occur during the surgery.

[0136] The ablation device 110 can send various types of data to the surgical computer 150 depending on the type of device used. Exemplary data types that can be sent include measured torque, audio signature, and measured displacement values. Similarly, the tracking technology 115 can provide different types of data depending on the tracking method employed. Exemplary tracking data types include the item being tracked (e.g., anatomical structure, tool, etc.), ultrasound images, and position values of surface or fiducial collection points or axes. When the system is operating, the tissue navigation system 120 provides anatomical positions, shapes, etc. to the surgical computer 150.

[0137] Although the display 125 is typically used to output data for presentation to the user, it can also provide data to the surgical computer 150. For example, for embodiments in which a monitor is used as part of the display 125, the surgeon 111 can interact with the GUI to provide input that is sent to the surgical computer 150 for further processing. For AR applications, the measured position and displacement of the HMD can be sent to the surgical computer 150 such that it can update the presented view as needed.

[0138] During the postoperative phase of the care period, various types of data can be collected to quantify the overall improvement or deterioration of the patient's condition due to the surgery. The data can take the form of, for example, self-reported information reported by the patient through a questionnaire. For example, in the case of a knee replacement surgery, the Oxford knee score questionnaire can be used to measure the functional status, and the postoperative quality of life can be measured through the EQ5D-5L questionnaire. Other examples in the case of hip replacement surgery can include the Oxford hip score, the Harris hip score, and the WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index). Such questionnaires can be administered, for example, directly by healthcare professionals in a clinical setting or using a mobile application that allows patients to answer questions directly. In some embodiments, a patient can be equipped with one or more wearable devices that collect data related to the surgery. For example, after a knee surgery, the patient can be equipped with a knee brace that includes sensors for monitoring knee position, flexibility, etc. This information can be collected and transmitted to the patient's mobile device for the surgeon to view to evaluate the outcome of the surgery and address any issues. In some embodiments, one or more cameras can acquire and record the movement of the patient's body part during a specified activity after the surgery. This movement acquisition can be compared with a biomechanical model to better understand the function of the patient's joint, and better predict the rehabilitation progress and determine any corrections that may be needed.

[0139] The postoperative phase of the care period can continue throughout the patient's life cycle. For example, in some embodiments, the surgical computer 150 or other components including the CASS 100 can continue to receive and collect data related to the surgical procedure after the surgery is performed. This data can include, for example, images, question answers, "normal" patient data (e.g., blood type, blood pressure, condition, medications, etc.), biometric data (e.g., gait, etc.), and objective and subjective data regarding specific issues (e.g., knee or hip pain). This data can be explicitly provided to the surgical computer 150 or other CASS components by the patient or the patient's physician. Alternatively or additionally, the surgical computer 150 or other CASS components can monitor the patient's EMR and retrieve relevant information when it is available. This longitudinal view of the patient's recovery allows the surgical computer 150 or other CASS components to provide a more objective analysis of the patient's outcome to measure and track the success or failure of a given procedure. For example, by performing a regression analysis on the various data items collected during the care period, the conditions experienced by the patient a long time after the surgical procedure can be linked to the surgery. This analysis can be further enhanced by analyzing groups of patients with similar procedures and / or similar anatomies.

[0140] In some embodiments, data is collected at a central location to provide easier analysis and use. In some cases, data can be manually collected from the various CASS components. For example, a portable storage device (e.g., a USB stick) can be attached to the surgical computer 150 to retrieve the data collected during the surgery. The data can then be transferred, for example, via a desktop computer to a centralized storage device. Alternatively, in some embodiments, the surgical computer 150 is directly connected to the centralized storage device via the network 175, as Figure 2C shown.

[0141] Figure 2C illustrates a "cloud-based" implementation where the surgical computer 150 is connected to the surgical data server 180 via the network 175. The network 175 can be, for example, a private intranet or the Internet. In addition to the data from the surgical computer 150, other sources can also transfer relevant data to the surgical data server 180. Figure 2CThree additional data sources are shown: patient 160, healthcare professional 165, and EMR database 170. Thus, patient 160 can send pre-operative and post-operative data to surgical data server 180, for example, using a mobile application. Healthcare professional 165 includes the surgeon and his or her staff, as well as any other professionals working with patient 160 (e.g., private doctor, rehabilitation specialist, etc.). It should also be noted that EMR database 170 can be used for pre-operative and post-operative data. For example, assuming patient 160 has given sufficient permission, surgical data server 180 can collect the patient's pre-operative EMR. Then, surgical data server 180 can continue to monitor the EMR for any updates after the surgery.

[0142] At surgical data server 180, a care period database 185 is used to store various data collected during a patient's care period. Care period database 185 can be implemented using any techniques known in the art. For example, in some embodiments, an SQL-based database can be used, where all the various data items are structured in two SQL collections in a way that allows them to be easily incorporated into rows and columns. However, in other embodiments, a No-SQL database can be employed to allow unstructured data while providing the ability to process and respond to queries quickly. As understood in the art, the term "No-SQL" is used to define a class of databases that are not related in their design. Various types of No-SQL databases can generally be grouped according to their underlying data models. These groupings can include databases using a column-based data model (e.g., Cassandra), a document-based data model (e.g., MongoDB), a key-value-based data model (e.g., Redis), and / or a graph-based data model (e.g., Allego). Any type of No-SQL database can be used to implement the various embodiments described herein, and in some embodiments, different types of databases can support care period database 185.

[0143] Data can be transferred between the various data sources and surgical data server 180 using any data formats and transfer techniques known in the art. It should be noted that Figure 2C the architecture shown allows for transfer from the data sources to surgical data server 180, as well as retrieval of data from surgical data server 180 through the data sources. For example, as explained in detail below, in some embodiments, surgical computer 150 can use data from past surgeries, machine learning models, etc. to help guide the surgical procedure.

[0144] In some embodiments, the surgical computer 150 or the surgical data server 180 may perform a de-identification process to ensure that the data stored in the episode-of-care database 185 meets the Health Insurance Portability and Accountability Act (HIPAA) standards or other requirements specified by law. HIPAA provides a list of certain identifiers that must be removed from the data during de-identification. The foregoing de-identification process may scan for these identifiers in the data being transmitted to the episode-of-care database 185 for storage. For example, in one embodiment, the surgical computer 150 performs a de-identification process before initially transmitting a particular data item or set of data items to the surgical data server 180. In some embodiments, a unique identifier is assigned to data from a particular episode of care to enable re-identification of the data if necessary.

[0145] Although Figure 2A-2C data collection in the context of a single episode of care has been discussed, it should be understood that the general concepts can be extended to data collection across multiple episodes of care. For example, surgical data can be collected across an entire episode of care each time the CASS 100 is used for a surgery and stored at the surgical computer 150 or the surgical data server 180. As further explained in detail below, a robust database of episode-of-care data allows for the generation of optimized values, measurements, distances, or other parameters, as well as other recommendations related to the surgical procedure. In some embodiments, various data sets are indexed in a database or other storage medium in a manner that allows for quick retrieval of relevant information during a surgical procedure. For example, in one embodiment, a patient-centric set of indexes can be used so that data related to a particular patient or a group of patients similar to a particular patient can be easily extracted. The concept can be similarly applied to surgeons, implant characteristics, CASS component models, etc.

[0146] More details regarding the management of episode-of-care data are described in PCT / US2019 / 067845, titled "Methods and Systems for Providing an Episode of Care," filed on December 20, 2019, the entire content of which is incorporated herein by reference.

[0147] Open and Closed Digital Ecosystems

[0148] In some embodiments, CASS is designed to be used as a stand-alone or “closed” digital ecosystem. Each component of CASS is specifically designed to be used within the closed ecosystem, and devices external to the digital ecosystem generally cannot access the data. For example, in some embodiments, each component includes software or firmware that implements proprietary protocols for activities such as communication, storage, security, etc. The concept of a closed digital ecosystem may be ideal for a company that wants to control all components of CASS to ensure that certain compatibility, security, and reliability standards are met. For example, CASS can be designed such that new components cannot be used with CASS unless they are certified by the company.

[0149] In other embodiments, CASS is designed to be used as an “open” digital ecosystem. In these embodiments, components can be produced by a variety of different companies, and the components implement standards for activities such as communication, storage, and security. Thus, by using these standards, any company can freely build independent, compliant components of the CASS platform. Data can be transferred between components using publicly available application programming interfaces (APIs) and open, shareable data formats.

[0150] CASS Queries and CASS Recommendations

[0151] Simple joints, such as ball and socket joints (e.g., hip and shoulder joints) or pivot joints (e.g., elbow joints), or more complex joints, such as condylar joints (e.g., knee joints), are incredibly complex systems whose performance can be significantly affected by a variety of factors. Procedures for replacing, resurfacing, or otherwise repairing these joints are common, for example, in response to joint damage or other degeneration. For example, total knee arthroplasty (TKA), which replaces the articular surfaces of the femur, tibia, and / or patella with artificial implants, is a common procedure for patients suffering from knee degeneration or trauma.

[0152] Selecting the best parameters for performing joint surgery is challenging. Continuing with the example of knee replacement surgery, a surgeon can place a first prosthesis on the distal end of the femur and a second prosthesis at the proximal end of the tibia, or the surgeon can install the prostheses in the reverse order. The surgeon seeks to place the prostheses optimally with respect to various parameters, such as the gap between the prostheses throughout the range of motion. Misplacement of the implant can have a negative impact on the quality of life of the patient after surgery. For example, if the gap between the tibia and the femur is too small at any time during the range of motion, the patient may experience pain and stiffness. On the other hand, if the gap is too large, the knee joint is too loose and may become unstable.

[0153] In some embodiments, the CASS (or preoperative planning application) 100 is configured to generate recommendations based on queries received from a surgeon or surgical staff. Examples of recommendations that may be provided by the CASS 100 include, but are not limited to, optimizing one or more surgical parameters, optimizing implant placement and orientation relative to one or more reference points such as anatomical or mechanical axes, modifying the surgical plan, or a description of how to achieve a particular outcome. As described above, the various components of the CASS 100 generate various types of data that collectively define the system state. Additionally, the CASS 100 can access various types of preoperative data (e.g., patient demographics, preoperative images, etc.), historical data (e.g., from other surgeries performed by the same or different surgeons), and simulation results. Based on all of this data, the CASS 100 can operate in a dynamic manner and allow the surgeon to instantaneously and intelligently modify the surgical plan as needed. In some embodiments, these modifications are made prior to surgery (e.g., before printing the cutting guide). In some embodiments, the modifications can be made by the CASS during surgery without the use of a custom cutting guide (e.g., a non-patient-specific cutting guide that can be selected and placed by the CASS). Thus, for example, in some embodiments, the CASS 100 notifies the surgeon of the modified surgical plan or optimization based on conditions not detected prior to surgery via the display 125.

[0154] In some embodiments, a surgical plan can be created prior to surgery using preoperative images and data. These images can include X-rays, CT, MRI, and ultrasound images. The data can include characteristics of the patient, including joint geometry, age, weight, activity level, etc., as well as data about the prosthesis to be implanted. The plan can then be modified based on additional information collected during the surgery. For example, additional medical images can be taken during the surgical procedure, and the additional medical images can be used to modify the surgical plan based on additional physiological details gathered from such images. In some embodiments, the surgical plan is based on patient information without the need to capture three-dimensional images, such as via a CT or MRI scan of the patient. Additional optical or X-ray images can be taken during the surgery to provide additional details and alter the surgical plan, thereby allowing the surgical plan to be developed and modified without the need for expensive preoperative medical imaging.

[0155] The processor of the CASS 100 can recommend any aspect of a surgical plan and modify this recommendation based on new data collected during the surgery. For example, the processor of the CASS 100 can optimize the anteversion and abduction angles of cup placement (in hip arthroplasty) or the depth and orientation of the distal and posterior femoral cut planes and patellar configuration (in PKA / TKA) in response to images captured before or during the surgery. Once an initial default plan is generated, the surgeon can request a recommendation regarding a specific aspect of the surgery and can deviate from the initial surgical plan. A request for a recommendation may result in a new plan, a partial deviation from the initial or default plan, or the confirmation and approval of the initial plan. Thus, by using the processor of the CASS 100 and a data-driven approach, the surgical plan can be updated and optimized as the surgery occurs. As has been explained throughout, these optimizations and recommendations can be generated by the processor before or during the surgery based on a statistical model or transfer function from multiple simulated patient anatomies, the statistical model or transfer function being informed by the specific details of the patient anatomy receiving the surgery. Thus, any additional data collected regarding the patient anatomy can be used to update the statistical model for that patient to optimize implant characteristics, thereby maximizing the performance criteria of the expected outcome of the surgery in the surgical plan.

[0156] Figure 3A A high-level overview of how recommendations can be generated is provided. This workflow begins at 305 where the surgical staff performs a surgical plan. The surgical plan can be an original plan generated based on pre-operative or intra-operative imaging and data, or the plan can be a modification of the original surgical plan. At 310, a surgeon or a member of the surgical staff requests a recommendation on how to address one or more issues in the surgical procedure. For example, the surgeon can request a recommendation on how to best align an implant based on intra-operative data (e.g., acquired using a point probe or new images). In some embodiments, the request can be made by manually entering a specific request into the GUI or voice interface of the CASS 100. In other embodiments, the CASS 100 includes one or more microphones that collect oral requests or queries from the surgeon, the oral requests or queries being translated into a formal request (e.g., using natural language processing techniques).

[0157] Continue to refer to Figure 3A, at 315, recommendations are provided to the surgeon. Various techniques can be used to provide recommendations. For example, when the recommendation provides a recommended cut to be made or a recommended implant orientation alignment, a graphical representation of the recommendation can be depicted on the display 125 of the CASS 100. In one embodiment, the recommendation can be overlaid on the patient's anatomy in the AR HMD 155. The resulting performance characteristics can be presented (e.g., for PKA / TKA, for various flexions and bends showing ligament tension for range of motion, medial and lateral condylar bone gaps and patellar groove tracking, or for THA, range of motion and center of pressure between the femoral head and acetabulum and plots of edge loading stress). In some embodiments, the information can be conveyed to the user via the display 125 (which can include the HMD 155) in the form of a drawing, numerically, or by changing colors or indicators. For example, in PKA / TKA, when a planned change or patient data indicates via a statistical model that the patella will encounter tracking problems with respect to the patellar groove or excessive strain of the patellar ligament without additional changes, an image of the patella (e.g., overlaid on the patient image) will glow red or flash. For THA, a portion of the acetabular cup can glow to indicate locations of increased edge loading or dislocation risk. In some embodiments, the interface can then invite the user to click to obtain a recommended solution, such as a patellar fill, ligament release, or a change in the orientation of the patellar implant or femoral implant (PKA / TKA) or the acetabular cup anteversion and abduction angles (THA) to optimize performance.

[0158] In addition to the recommendation, in some embodiments, the CASS / Planning System 100 may also provide the reason for the recommendation. For example, for the recommended alignment or orientation of an implant, the CASS 100 may provide a list of patient-specific characteristics or activities that influenced the recommendation. The CASS-recommended alignment or orientation of an implant may further refer to a reference system or reference point, such as an anatomical or mechanical axis or a distance from a bone or bony landmark. Additionally, as shown at 320, the CASS may simulate how selecting a particular recommendation will affect the remainder of the surgical procedure. For example, prior to surgery, a default cutting guide may be generated based on a preoperative three-dimensional CT or MRI scan. During surgery, once an incision is made, the surgeon may obtain high-resolution data of the patient's anatomy from an MRI or using a point probe or an optical camera. The CASS 100 may use such data to create a new or updated plan or recommendation for resection of bone tissue using a resection tool or cutting guide. At step 320, the impact of this revised plan or recommendation may be presented in the form of a revised alignment instruction or the like. Multiple recommendations may also be presented to the surgeon and the impact of each recommendation on the surgical plan may be observed. For example, two viable recommended bone resection plans or recommendations may be generated and the surgeon may decide which one to execute based on the impact of each recommendation on the subsequent steps of the surgery. Animations of the range of motion or dynamic activities (such as walking or climbing stairs, etc.) that are the functional outcome of each recommendation may also be presented to the surgeon to allow the surgeon to better understand how the recommendation will affect the patient's movement characteristics. In some embodiments, the surgeon is able to select a single data item or parameter (such as alignment, tension, and flexion gap, etc.) for optimizing the recommendation. Finally, once the surgeon selects a particular recommendation, it is executed at step 325. For example, in an embodiment where a custom cutting guide is manufactured prior to surgery, step 325 may be executed by printing the cutting guide and delivering it to the surgeon for use during surgery. In an embodiment where a robotic arm holds the cutting guide at a specific predetermined position, a command to place the cutting guide may be sent to the robotic arm as part of the CASS workflow. In an embodiment where no cutting guide is used, the CASS may receive instructions to assist the surgeon in resecing the femoral component and tibia according to the recommendation.

[0159] The CASS 100 may present the recommendation to the surgeon or surgical staff at any time before or during surgery. In some cases, the surgeon may explicitly request as described above regarding Figure 3ARecommended Discussions. In other embodiments, CASS 100 can be configured to execute a recommendation algorithm as a background process when performing a surgery based on available data. When a new recommendation is generated, CASS 100 can notify the surgeon using one or more notification mechanisms. For example, a visual indicator can be presented on the display 125 of CASS 100. Ideally, the notification mechanism should be relatively non-intrusive so that it does not interfere with the surgery. For example, in one embodiment, different text colors can be used to indicate that a recommendation is available when the surgeon is navigating through the surgical plan. In some embodiments, when using an AR or VR headset 155, the recommendation can be provided into the user's field of view and highlighted to draw the user's attention to the recommendation or the availability of the recommendation. The user (e.g., surgeon or technician) can then interact with the recommendation or the solicitation of the recommendation in the AR or VR user interface using any of the means described below. In these embodiments, CASS 100 can treat the user headset 155 as an additional display and use any conventional means for communicating with a display to convey the information to be displayed thereon.

[0160] To illustrate one type of recommendation that can be performed with CASS 100, a technique for optimizing surgical parameters is disclosed below. As used herein, the term "optimize" refers to selecting the best parameters based on certain specified criteria. In an extreme case, optimization can refer to selecting the best parameters based on data from the entire care period, including any preoperative data, the CASS data state at a given point in time, and postoperative goals. Moreover, historical data can be used to perform the optimization, such as data generated during past surgeries involving, for example, the same surgeon, past patients with physical characteristics similar to the current patient, etc.

[0161] The optimized parameters can depend on the part of the patient's anatomy on which the surgery is to be performed. For example, for a knee surgery, the surgical parameters can include positioning information of the femoral and tibial components, including but not limited to rotational alignment (e.g., varus / valgus rotation, external rotation, flexion rotation of the femoral component, posterior tilt of the tibial component), resection depth (e.g., varus knee, valgus knee), and the type, size, and position of the implant. The positioning information can also include surgical parameters for combining implants, such as overall limb alignment, combined tibiofemoral hyperextension, and combined tibiofemoral resection. Other examples of parameters that CASS 100 can optimize for a given TKA femoral implant include the following:

[0162]

[0163] Additional examples of parameters that CASS can optimize for a given TKA tibial implant include the following:

[0164]

[0165] For hip surgery, surgical parameters can include femoral neck resection location and angle, cup tilt angle, cup anteversion angle, cup depth, femoral stem design, femoral stem size, femoral stem fit within the canal, femoral offset, leg length, and the femoral version of the implant.

[0166] Shoulder parameters can include, but are not limited to, humeral resection depth / angle, humeral shaft version, humeral offset, glenoid version and inclination, and reverse shoulder parameters such as humeral resection depth / angle, humeral shaft version, glenoid inclination / version, glenoid ball orientation, glenoid ball offset, and offset direction.

[0167] There are various conventional techniques for optimizing surgical parameters. However, these techniques typically require a large amount of computation and, thus, parameters are typically determined preoperatively. As a result, the surgeon's ability to modify the optimized parameters based on problems that may arise during the surgery is limited. Also, conventional optimization techniques typically operate in a "black box" manner with little or no explanation of the recommended parameter values. Thus, if a surgeon decides to deviate from the recommended parameter values, the surgeon will typically do so without fully understanding the impact of the deviation on the remainder of the surgical procedure or the impact of the deviation on the patient's quality of life after surgery.

[0168] To address these and other drawbacks of conventional optimization techniques, in some embodiments, optimization can be performed using buttons or other widgets presented to the surgeon in the GUI during the surgical workflow (e.g., on display 125 or AR HMD 155). For purposes of the following discussion, the surgeon or other healthcare professional can use any means, e.g., an oral request / command or manual input (e.g., using a touchscreen or button), to invoke a request for a recommendation or input from CASS 100. For purposes of this application, these types of queries or requests for a recommended course of action, recommended parameter optimizations, or other feedback provided to the surgeon or healthcare professional in response to such queries or requests are referred to as CASS recommendation requests or "CASSRRs". A CASSRR can be invoked or activated by the surgeon or healthcare professional at any time during the surgery. For example, for TKA, a CASSRR can be invoked during the femoral implant planning phase, the tibial implant planning phase, and / or the gap planning phase. In a THA surgery, a CASSRR can be used during femoral neck resection, acetabular implant placement, femoral implant placement, and implant selection (e.g., size, offset, bearing type, etc.). A CASSRR can be invoked, for example, by pressing a button or by speaking a specific command (e.g., "optimize gap"). As described above, the recommendation system can be configured to provide or prompt the surgeon to seek a recommendation or optimization at any time during the surgery.

[0169] Figure 3B-3Eillustrates an example of a GUI of a CASS / planning app that can be used during a surgical workflow, such as Figure 1 the app depicted in. These GUIs can be displayed, for example, on the display 125 of the CASS 100 or on a workstation during the planning phase of an upcoming surgery. In each example, an AI-driven recommendation request can be invoked using a button that is presented as a visual widget of the interface. Specifically, Figure 3B illustrates an exemplary implant placement interface 330, where a recommendation button 335 (labeled CASSRR in this example) is shown in the lower left corner. Similarly, Figure 3C illustrates an exemplary gap planning interface 340 with a button 335.

[0170] Invoking CASSRR can cause the optimized parametric interface 370 to be displayed as shown in Figure 3D when the user requests a recommendation. In the example of Figure 3D , the CASSRR / recommendation request is invoked using an activated button on the implant placement interface 330 (as shown in Figure 3B ). The optimized parametric interface 370 includes degrees of freedom (DoF) buttons 345 related to the movement of the implant (e.g., translation, rotation, etc.). During the optimization analysis, activation of any DoF button 345 "locks" the corresponding degree of freedom. For example, if the surgeon is satisfied with the anterior or posterior positioning, the surgeon can activate the "A" button or the "P" button, respectively, to lock the anterior position or the posterior position. In some embodiments, the lock button changes color or provides a different type of visual indication when switched to the locked position. In Figure 3DIn an example, the "S" button (corresponding to the upper positioning) has been locked, as depicted by the graphical illustration of the lock to the right of the button. It should be noted that using a button for locking the position is merely one example of how a surgeon can dock with the CASS 100; for example, in other embodiments, the surgeon can verbally request to lock a specific position (e.g., "lock the upper positioning"), or in a VR environment, gestures can be employed. The values that can be fixed can depend on the available optimization factors and the comfort level of the surgeon in optimizing different aspects of the surgery. For example, so far, it has been assumed that the optimization is performed from a fully functional kinematic perspective. Thus, the optimization system does not truly understand how the implant interacts with the bone. This can be addressed, for example, by adding image-based analysis to the optimization. However, for embodiments that do not perform such analysis, the surgeon may wish to constrain the problem based on how he / she views the bone fit. For example, regarding the femoral component of the implant in a TKA surgery, the anterior overhang may not be known when the optimization is performed. This means that the surgeon may wish to adjust the A-P position of the femoral component because he / she is observing the implant based on the bone fit rather than the kinematic performance. Thus, the surgeon can determine the A-P position, rotation, and possibly the joint line that may be used to determine the final implant position, orientation, and location. In this way, the surgeon can supplement the knowledge provided by the computer-generated optimization run, thereby allowing the surgical plan implemented by the surgeon to deviate from the computed recommendations.

[0171] The button can also be used to provide boundary control for a given parameter used in the optimization. In Figure 3D an example, there are two boundary control buttons 355 for the posterior tilt angle. These parameters can be used to set the minimum or maximum values of the relevant parameters that will bound the parameter in the optimization. If the right boundary control button 355 is locked, the optimization can be configured to produce values higher than the current value specified for the posterior tilt angle. Conversely, if only the left boundary control button 355 is locked, the optimization can be configured to produce values lower than the current value specified for the posterior tilt angle. If both boundary control buttons 355 are locked, then the optimization is configured such that it does not change the specified posterior tilt angle value. On the other hand, if neither of the boundary control buttons 355 is locked, the optimization can freely change the value (within the device specifications).

[0172] The optimization parameterization interface 370 includes an optimization button 350 that, when activated, optimizes the implant placement parameters using any of the data-driven / AI methods described herein. It should be noted that this general concept is not limited to implant placement; in fact, generally, any surgical parameter or group of parameters can be optimized using similar interfaces and techniques. This optimization process is further detailed below. After the optimization, the surgeon can return to the implant placement interface 330 (as Figure 3B shown) to continue the surgery with the optimized parameters.

[0173] The switching button 365 allows the surgeon to switch between any two views or aspects of a surgical procedure or surgical procedure plan. For example, the switching button 365 can provide the surgeon with the current bone condition and future bone condition based on the surgical plan or partial or full execution of the current planned implant location and alternative (e.g., recommended) implant locations. The switching button 365 can also provide the surgeon with alternative future conditions of the bone and / or implant based on whether the surgeon chooses to take one course of action rather than an alternative course of action. Activation of this button causes the various images and data presented on the optimization parameterization interface 370 to be updated with the current or previous alignment information. Thus, the surgeon can quickly view the impact of any changes. For example, the switching feature can allow the surgeon to visualize the prosthetic positioning changes recommended by the optimizer relative to the concept of their previous proper implant placement. In one embodiment, during the initial use of the system, the user can choose to plan a case without optimization and wishes to visualize the impact of automation. Similarly, the user may wish to visualize the impact of "locking" various aspects of the plan.

[0174] If the surgeon wishes to understand the rationale behind the optimization, the response and rationale button 360 on the optimization parameterization interface 370 can be activated to display Figure 3E the response and rationale interface 375 shown in. This interface 375 includes an animation screen 380 that provides an animation of the anatomical structure of interest during a performance measurement activity (e.g., deep knee bend). This animation can be provided as an output of the anatomical modeling software that performs the optimization (e.g., LIFEMOD TM ), or alternatively, separate software can be used to generate the animation based on the output of the optimization software. For example, an animated GIF file or a small video file can be used to depict the animation. In an embodiment using the AR HMD 155, an animated hologram can be provided on the relevant anatomical structure to provide further contextualization of the simulated behavior.

[0175] The response and rationale interface 375 also includes a response screen 385 that displays plots of various performance or condition metrics (e.g., measured v flexion angle). A set of performance measurement selection buttons 390 on the right hand side of the response and rationale interface 375 allows the surgeon to select various relevant performance measurements and update the plots shown in the response screen 385. In Figure 3E the example of, these performance measurements include internal-external (IE) rotation, medial and lateral rollback, MCL and LCL strain, iliotibial band (ITB) strain, bone stress, varus-valgus (V-V) rotation, medial-lateral (ML) patellar shear force, quadriceps force, and bone interface force. The example shown in screen 385 depicts the lateral and medial gaps throughout the flexion range, which is a traditional estimate of TKA performance. Traditionally, the lateral and medial gaps have only been considered at two flexion degrees.

[0176] To support the various interfaces described above, the algorithms that support the CASSRR / recommendation buttons should preferably be executed as quickly as possible to ensure that the surgical workflow is not disrupted. However, the calculations involved in performing the optimization can be computationally intensive. Thus, in some embodiments, to simplify the processing required, a set of predictive equations can be generated based on a training data set and simulated performance measurements. These predictive equations provide a simplified form of the parameter space that can be optimized near real-time. These processes can be executed locally or on a remote server, such as in the cloud.

[0177] Figure 4 A system diagram is provided that illustrates how the optimization of surgical parameters is performed according to some embodiments. Such optimization can be performed during the pre-operative planning phase, such as in embodiments where a custom cutting guide is created prior to surgery, or during surgery, such as in embodiments where CASS can adjust the exact pose of the resection plane on a robot or by other practical means such as haptic feedback. Briefly, the surgeon 111 provides certain patient-specific parameters and parameters related to the implant to the surgical computer 150 (via the GUI presented on the display 125). The surgeon 111 requests that the optimization be performed. The surgical computer 150 uses the parameters to retrieve a set of predictive equations from the equation database 410 stored on the surgical data server 180. In embodiments that do not use a cloud-based architecture, this equation database 410 can be stored directly on the surgical computer 150. The optimization of the set of equations provides the required optimization (e.g., best implant alignment and positioning). This information can then be presented to the surgeon 111 via the display 125 of the CASS 100 (see, for example Figure 3D and 3E ).

[0178] As explained in more detail below, each equation dataset provides kinematic and dynamic responses for a set of parameters. In some embodiments, the equation database 410 is populated with equation datasets derived by the simulation computer 405 based on a set of training data. The training dataset includes surgical datasets previously collected by the CASS 100 or another surgical system. Each surgical dataset may include information such as, for example, the geometry of the patient, how the implant was positioned and aligned during the surgery, ligament tension, etc. Any technique known in the art may be used to collect the data. For example, for ligament tension, robotic-assisted techniques may be employed as described in PCT / US2019 / 067848, titled "Actuated Retractor with Tension Feedback," filed on December 20, 2019, the entire content of which is incorporated herein by reference. Another example is provided in PCT / US2019 / 045551 and PCT / US2019 / 045564, titled "Force-Indicating Retractor Device and Methods of Use," filed on August 7, 2019, the entire disclosures of which are incorporated herein by reference.

[0179] For each surgical dataset, the simulation computer 405 performs an anatomical simulation on the surgical dataset to determine a set of kinematic and dynamic responses. Non-limiting examples of suitable anatomical modeling tools that may be used include LIFEMOD TM or KNEESIM TM(Available from LIFEMODELER, INC., San Clemente, California, a subsidiary of SMITH AND NEPHEW, INC.). Additional examples of the use of biomechanical modeling during surgery are described in the following documents: U.S. Patent No. 8,794,977, titled "Implant Training System"; U.S. Patent No. 8,712,933, titled "Systems and methods for determining muscle force through dynamic gain optimization of a muscle PID controller for designing a replacement prosthetic joint"; U.S. Patent No. 8,412,669, titled "Systems and methods for determining muscle force through dynamic gain optimization of a muscle PID controller", the entire contents of which are incorporated herein by reference.

[0180] In addition to determining the response of the surgical data set, the simulation computer 405 can also be used to supplement the real-world surgical data set with an artificially generated surgical data set that fills any gaps in the training data set. For example, in one embodiment, a Monte Carlo analysis is performed using small permutations of the various factors in the training set to see how they affect the response. Thus, a relatively small real-world surgical data set (e.g., 1,000 data sets) can be substantially extrapolated to produce an exponentially larger data set that covers a variety of patient anatomies, implant geometries, etc. Once the data set has been populated, one or more equation fitting techniques commonly known in the art can be used to derive an equation data set stored in the equation database 410.

[0181] To determine the kinematic and dynamic responses for each equation dataset, the simulations performed by the analog computer 405 can model and simulate various activities that stress the anatomical structures of interest. For example, in the context of a TKA or other knee surgery, weighted deep knee bends can be used. During a deep knee bend, the knee flexes downward and returns to an upright position at various angles (e.g., 120°, 130°, etc.) under a certain load. During a deep knee bend, the load appears on the leg extensors (i.e., quadriceps), leg flexors (i.e., hamstrings), passive ligaments in the knee, etc. Thus, a deep knee bend stresses the anterior cruciate ligament (ACL), posterior cruciate ligament (PCL), lateral collateral ligament (LCL), and medial collateral ligament (MCL). Additionally, a deep knee bend allows for the measurement of various kinematics (e.g., how the patella moves relative to the femoral component, how the femur moves relative to the tibia, etc.). It should be noted that this is one example of a performance measurement that can be applied, and various other measurements can be used as a supplement or alternative to a deep knee bend. Knee kinematics can also be simulated using a knee model to perform approximate real-world movements associated with dynamic activities, such as going up and down stairs or swinging a golf club. Other joints in the body can be simulated as simple ideal components, while the individual ligaments and implant components of interest that perform the movement can be simulated in detail under the exemplary loads associated with each activity being considered.

[0182] Figure 5A-5F Exemplary joint prediction equations that can be used in an equation dataset in some embodiments are described. While these figures will be described with respect to knee arthroplasty, these concepts are equally applicable to other joint arthroplasties, such as hip arthroplasty. Regardless of the joint arthroplasty being performed, the basic categories can be the same, and the specific data within each category is related to the surgery being performed. Figure 5A An overview of the knee prediction equation is provided. As will be explained below, the terms of this equation are simplified to allow for an understanding of the individual terms. Thus, it should be understood that the exact mathematical constructs can be different from those shown in the figures.

[0183] In these prediction equations, the terms on the left - hand side are referred to as "factors", while the terms on the right - hand side are referred to as "responses". The response and factors can be associated with specific numerical values, but in at least some embodiments, at least some can be represented as probability distributions (e.g., bell curves) or another way of reflecting the uncertainty about the actual values of the factors or responses. Thus, the equations can account for the uncertainty of certain aspects of this process. For example, in at least some embodiments, it may be difficult to identify the soft - tissue attachment location deterministically, and thus, uncertainty information can be used to reflect the probability distribution that the soft - tissue attachment location is actually located based on the estimated location identified during image processing. Similarly, in at least some embodiments, rather than determining the exact optimal position and orientation of an orthopedic implant, it may be desirable to determine the optimal position and orientation in the context where the location and orientation where the implant will actually be positioned may vary (e.g., to account for tolerances in manufacturing custom cutting - guide instruments, variability in the surgical techniques of the surgeon, etc.).

[0184] Figure 5B Patient - specific parameters for a knee - joint prediction equation are shown. These parameters can be measured by the surgical staff based on pre - operative or intra - operative data. As Figure 5B shown in the example of, an X - ray measurement tool can be used to measure various anatomical features in an image. Examples of patient - specific parameters that can be utilized include load - bearing access (LBA), pelvic width, femoral ML width, tibial ML width, femoral length, etc.

[0185] Figure 5C Soft - tissue balance parameters included in the knee - joint prediction equation are shown. The soft - tissue balance parameters can be derived from multiple sources. For example, by default, the parameters can be derived from an atlas of landmarks based on the patient's bone geometry. This can be supplemented with the results of, e.g., anterior drawer test, varus - valgus stability measurements, tissue attachment estimations, and tissue condition measurements (e.g., stiffness). The data can also be supplemented with intra - operative data such as joint distraction tests, instrumented tibial inserts, force - sensing gloves, etc.

[0186] Figure 5D Implant - geometry parameters for the knee - prediction equation are shown. These parameters can include, for example, femoral and tibial clearances, distal or posterior radii, patellar geometry and alignment or filling, and the anterior - posterior / lateral - medial placement of the implant on the femur. It should be noted that for a given patient, there may be many possible implants (e.g., models, sizes, etc.). Thus, different knee - joint prediction equations can be designed to have the same patient - specific and tissue - balance parameters, but different implant - geometry parameters. For example, a range of sizes can be represented by a set of key prediction equations. In some embodiments, something like LIFEMOD can be used TMThe anatomical modeling software programmatically determines the implant geometry. Exemplary techniques for optimizing parameters related to the anatomical and biomechanical fit of an implant or implant system implanted into a patient's joint are described in U.S. Patent Application No. 13 / 814,531, which is hereby incorporated by reference.

[0187] Figure 5E Shows implant alignment and positioning parameters that can be used in knee prediction equations. As described above, these can be used as variables during optimization. As Figure 5E shown, example parameters include femoral superior-inferior (S-I) position, femoral anterior-posterior (A-P) position, femoral varus-valgus (V-V) position, femoral internal-external (I-E) position, tibial posterior slope, tibial V-V position, tibial I-E position, and tibial depth, as determined by the extension gap.

[0188] Figure 5F Shows the response portion of the knee prediction equation. The response can include a data set that includes kinematic and kinetic data related to the knee. Kinematic data provides a measure of how the kinematics of a particular patient and component are set compared to a specified target. For example, kinematics can measure the internal-external rotation of the femoral component relative to the tibia and what that marker looks like over the flexion profile of a deep knee bend event. This can then be compared to the target measurement of internal-external rotation. This concept can extend to the patella and other anatomical structures related to knee movement. Kinetic data provides a measure of the loads on the various components of the knee (e.g., LCL, MCL, etc.). As Figure 5F shown, the simulation derives this data for several different knee flexion degrees (e.g., 30, 60, 90, 120, etc.). Thus, for a given set of parameters on the left-hand side of the equation, a set of equations with different response values can be specified.

[0189] Figure 6 Shows a process that can perform the optimization of a system of equations. Starting at step 605, patient-specific parameters and soft tissue balance parameters are input by the surgical staff. As described above, the patient-specific parameters can be derived based on any combination of pre-operative and intra-operative data. The tissue balance parameters are measured by the surgeon (or default values can be used). At step 610, the implant geometry or implant manufacturer, model, and manufacturer and product ID are input by the surgical staff. In some embodiments, the surgical staff can manually input each implant geometry parameter. In other embodiments, the surgical staff can specify a particular implant manufacturer, model, and / or size (e.g., SMITH&NEPHEW, II left femoral implant, (No. 6), and appropriate implant parameters (e.g., geometry, dimensions, or other implant characteristics) can be retrieved from a local or remote database.

[0190] Continuing to refer Figure 6 , at step 615, a set of equations is selected based on patient-specific parameters, soft tissue parameters, and implant geometry. The set of equations includes one or more prediction equations, where each equation provides a different response value. Next, at step 620, the set of equations is optimized to obtain implant alignment and position recommendations. In at least some embodiments, it may not be possible to fully solve all the equations because these factors can affect the various responses in different ways. Thus, in some embodiments, the responses can be associated with weighting values such that the optimization process gives some responses greater weight than others. These weighting values can serve as desirable factors or functions that quantify the relative importance of the various responses. For example, in some embodiments, the optimization is performed using a goal programming (GP) algorithm, where the weights of the response variables are obtained through a group decision-making (GDM) process. Finally, at step 625, the implant alignment and position recommendations are depicted visually, e.g., on the display 125 of the CASS 100.

[0191] In some embodiments, the relationship between the factors and the responses can be defined by a set of trained neural networks rather than a series of equations. Statistical and modeling tools similar to those described above can be used to define and train the neural networks and the factors used therein. In some embodiments, tools available from NEURODIMENSIONS, INC. of Gainesville, Florida, such as NEUROSOLUTIONS 6.0, can further facilitate the development and training of neural networks. In some embodiments, an information database collected from previous orthopedic procedures or studies can be used to train the neural networks, and as additional data is collected over time, the neural networks can be further refined to enhance the optimization process described herein. In some embodiments, kernel methods can be used to explore the relationship between the factors and the responses. Kernel-based learning algorithms can be used to solve complex computational problems, such as clustering, classification, etc., to detect and exploit complex patterns in the data.

[0192] In some embodiments, the relationship between the factors and the responses can be defined by one or more trained support vector machines. Similar to some neural networks, support vector machines can be trained to identify patterns in existing data, such as data collected from previous orthopedic procedures or studies, and once trained, used to predict the response of an orthopedic procedure for a particular patient based on the settings of specific factors.

[0193] Although the foregoing discussion relates to recommendations in the context of knee surgery, the factors and responses used in the predictive equations can be modified as needed based on the anatomy that is the subject of the surgical procedure. For example, surgery to repair a torn or damaged anterior cruciate ligament ("ACL") can benefit from the use of the CASS and CASSRR concepts described above. The application of a robotic surgical system to ACL surgery is described in PCT / US2019 / 048502, titled "Robotic Assisted Ligament Graft Placement and Stressing," filed on August 28, 2019, which application is hereby incorporated by reference in its entirety.

[0194] Another surgical intervention that can benefit from the use of the CASS and CASSRR concepts described above is a high tibial osteotomy ("HTO") procedure. In an HTO procedure, an incision is made in the tibia, and a bone wedge can be removed or added from the incision in the tibia to better align the tibia and femur in the knee joint. For example, CASSRR can be used to optimize the amount of bone added or removed to achieve a desired kinetic response (i.e., removing the affected compartment to delay further cartilage damage). Additionally, changes in the tibial slope that are difficult to plan can be simulated and implemented with a robot.

[0195] In the context of a THA procedure, the "implant alignment / position" factors discussed above Figure 5E can be replaced with parameters such as cup tilt, cup anteversion, cup size, cup depth, bearing type (conventional, ceramic-on-ceramic, dual mobility, surface replacement, etc.), femoral stem design, femoral stem form, combined anteversion, femoral stem size, femoral stem offset (STD, HIGH), and head offset. Another component of implant alignment / position can be screw placement. Software can make recommendations as to how many screws should be used, the length and trajectory of each screw, and problem areas to avoid (soft tissue, blood vessels, etc.). For hip revision surgery, the "implant geometry" factor can be modified to make recommendations as to which type of acetabular or femoral component will best fill the missing anatomy.

[0196] In addition, although the above discussion regarding the generation of intraoperative recommendations addressed a recommendation system, it should be noted that recommendations can also be applied during the preoperative and postoperative phases of the care period. For example, based on preoperative data and historical data, a recommended surgical plan can be developed. Similarly, if a surgical plan has been generated, recommendations can be generated based on the circumstances of changes that occur after the preoperative data has been generated. After surgery, data collected during the earlier stages of the care period can be used to generate recommended postoperative recovery regimens (e.g., goals, exercises, etc.). Examples of data that can affect the recovery regimen include, but are not limited to, implant type and size, surgical time, hemostasis time, tissue release, and intraoperative flexion. In addition to the activities of the recovery regimen, devices for rehabilitation and recovery can also be customized based on care period data. For example, in one embodiment, care period data is used to generate the design of a custom insole that can be 3D printed for the patient. In addition to generating postoperative recommendations for the patient, postoperative care period data can also be used as a feedback mechanism in the CASSRR to further improve the machine learning model used to provide recommendations for performing surgical procedures on other patients.

[0197] Slider interface for providing interactive anatomical modeling data

[0198] In some embodiments, as an alternative or supplement to the interfaces described above, dynamic sliders can be used to depict various measurements, as shown in FIG. 7A. Although FIG. 7A continues the example of knee implant alignment, it should be understood that the general concepts shown in FIG. 7A can be applied to various types of measurements performed during surgery or the preoperative phase. As depicted in FIG. 7A, the surgeon manually aligns the implant (as depicted in image 705), or manipulates the implant on the display screen using the optimization process described above (as shown in image 710) to produce a response 715. In this case, the response 715 is shown as having multiple sliders with settings corresponding to multiple flexion angles (30, 60, 90, and 120 degrees). These sliders are "dynamic" in the sense that they are updated in real time as the surgeon changes the alignment. Thus, for example, if the surgeon makes a manual movement 705 of the implant, the response 715 will be recalculated and each slider will be updated accordingly.

[0199] Figure 7B A further illustration of the slider content is provided, showing the sagittal alignment response of the tibial implant. As Figure 7B depicted, the desired or preferred alignment configuration is 5 degrees of flexion, but the current flexion measurement is only 4 degrees. Indications, tips, or annotations can be provided to indicate that the flexion should be changed (e.g., decreased or increased) to avoid cutting the fibula. The current implant alignment is depicted with a notation mark on the lower portion of the slider, while the surgeon's 5-degree goal is shown as a notation mark on the upper portion of the slider. The FDA 510(k) limits (which define the allowable parameters in compliance with FDA regulations) are shown with an external bar overlaid on the slider.

[0200] In Figure 7B the example, the slider further includes an internal strip covering the slider, and the internal strip shows the surgeon's limitations on the response values. These limitations can be derived or determined by reviewing and analyzing certain historical data available from CASS (i.e., limitations from past surgeries), or the surgical staff or other technicians can input this information before the surgery. In some embodiments, not every limitation is explicitly provided; rather, the limitations are sourced from a set of general instructions provided by the surgeon. For example, in one embodiment, the surgeon or the surgeon's staff provides a text description of the surgeon's preferences. Then, a natural language processing algorithm is applied to extract relevant information from the text. Based on the extracted text, rules for generating the text are generated. The following table provides an example set of rules generated based on the shown input text.

[0201]

[0202] Surgical patient care system

[0203] The general concept of optimization can be extended to the entire care period using the surgical patient care system 820, which uses surgical data as well as other data from the patient 805 and healthcare professionals 830 to optimize outcomes and patient satisfaction, as Figure 8 shown.

[0204] Conventionally, preoperative diagnosis, preoperative surgical planning, intraoperative execution of the established plan, and postoperative management of total joint replacement are all based on personal experience, published literature, and the surgeon's training knowledge base (ultimately, the individual surgeon's tribal knowledge and their peer "networks" and journal publications), as well as their instinct to use guidance and visual cues for accurate intraoperative tactile discrimination of "balance" and accurate manual execution of plane resection. This existing knowledge base and execution method are limited in terms of optimizing the outcomes provided to patients in need of care. For example, there are limitations in the following aspects: accurately diagnosing the patient for appropriate, minimally invasive established care; aligning the dynamic patient, healthcare economy, and surgeon's preferences with the desired patient outcomes; executing the surgical plan to correctly align the bones and maintain balance, etc.; and receiving data from disconnected sources with different biases that are difficult to reconcile into the overall patient framework. Therefore, data-driven tools that more precisely simulate anatomical responses and guide surgical planning can improve the existing methods.

[0205] The surgical patient care system 820 is designed to utilize patient-specific data, surgeon data, healthcare facility data, and historical outcome data to develop algorithms that recommend or suggest an optimal overall treatment plan for the entire care period of a patient (pre-operative, intra-operative, and post-operative) based on desired clinical outcomes. For example, in one embodiment, the surgical patient care system 820 tracks compliance with the recommended or suggested plan and adjusts the plan based on patient / care provider performance. Once the surgical treatment plan is complete, the surgical patient care system 820 records the collected data in a historical database. This database is accessible to future patients and used to develop future treatment plans. In addition to using statistical and mathematical models, simulation tools (e.g., ) can be used to simulate outcomes, alignment, kinematics, etc. based on a preliminary or suggested surgical plan and reconfigure the preliminary or suggested plan based on the patient's profile or surgeon's preferences to achieve desired or optimal outcomes. The surgical patient care system 820 ensures that each patient is receiving personalized surgical and rehabilitation care, thereby increasing the chances of successful clinical outcomes and reducing the financial burden on the facility associated with recent revisions.

[0206] In some embodiments, the surgical patient care system 820 employs data collection and management methods to provide a detailed surgical case plan that has different steps monitored and / or executed using CASS 100. The user's execution is calculated at the completion of each step and used to suggest changes to subsequent steps of the case plan. The generation of the case plan depends on a series of input data stored in a local or cloud storage database. The input data can be either related to the patient currently receiving treatment or historical data from patients who have received similar treatment.

[0207] Patient 805 provides inputs such as current patient data 810 and historical patient data 815 to the surgical patient care system 820. A variety of methods commonly known in the art can be used to collect such inputs from patient 805. For example, in some embodiments, patient 805 fills out a paper or digital survey that the surgical patient care system 820 parses to extract patient data. In other embodiments, the surgical patient care system 820 can extract patient data from existing information sources such as electronic medical records (EMRs), health history files, and payer / provider history files. In still other embodiments, the surgical patient care system 820 can provide an application programming interface (API) that allows external data sources to push data to the surgical patient care system. For example, patient 805 may have a mobile phone, wearable device, or other mobile device that collects data (e.g., heart rate, pain or discomfort level, movement or activity level, or patient-submitted responses regarding compliance with any number of pre-operative planning criteria or conditions) and provides that data to the surgical patient care system 820. Similarly, patient 805 may have a digital application on their mobile or wearable device that can collect data and transmit it to the surgical patient care system 820.

[0208] Current patient data 810 can include, but is not limited to: activity level, pre-existing conditions, comorbidities, pre-rehabilitation performance, health and fitness level, pre-operative expectation levels (relating to the hospital, surgery, and rehabilitation), Metropolitan Statistical Area (MSA)-driven scores, genetic background, previous injuries (sports, trauma, etc.), previous joint replacements, previous trauma surgeries, previous sports medicine surgeries, treatment of the contralateral joint or limb, gait or biomechanical information (back and ankle tissues), pain or discomfort level, care infrastructure information (payer coverage type, home medical infrastructure level, etc.), and an indication of the expected ideal outcome of the surgery.

[0209] Historical patient data 815 can include, but is not limited to: activity level, pre-existing conditions, comorbidities, pre-rehabilitation performance, health and fitness level, pre-operative expectation levels (relating to the hospital, surgery, and rehabilitation), MSA-driven scores, genetic background, previous injuries (sports, trauma, etc.), previous joint replacements, previous trauma surgeries, previous sports medicine surgeries, treatment of the contralateral joint or limb, gait or biomechanical information (back and ankle tissues), pain or discomfort level, care infrastructure information (payer coverage type, home medical infrastructure level, etc.), the expected ideal outcome of the surgery, the actual outcome of the surgery (patient-reported outcomes [PRO], implant survival, pain level, activity level, etc.), the size of the implant used, the location / orientation / alignment of the implant used, the soft tissue balance achieved, etc.

[0210] A healthcare professional 830 performing a surgery or treatment can provide various types of data 825 to a surgical patient care system 820. For example, the healthcare professional data 825 can include a description of known or preferred surgical techniques (e.g., cruciate retaining (CR) vs. posterior stabilized (PS), size augmentation vs. size reduction, with tourniquet vs. without tourniquet, femoral stem style, preferred options for THA, etc.), the training level of the healthcare professional 830 (e.g., years of practice, position trained for, place of training, techniques they emulate), the previous success level including historical data (outcomes, patient satisfaction), and the expected ideal outcomes regarding range of motion, number of recovery days, and lifespan of the device. The healthcare professional data 825 can be obtained, for example, via a paper or digital survey provided to the healthcare professional 830, via the healthcare professional's input to a mobile application, or by extracting relevant data from an EMR. Additionally, the CASS 100 can provide data such as profile data (e.g., patient-specific knee implant profile) or a history describing the use of the CASS during the surgery.

[0211] Information related to the facility where the surgery or treatment is to be performed can be included in the input data. This data can include, but is not limited to, the following: ambulatory surgery center (ASC) vs. hospital, facility trauma level, comprehensive joint replacement care episode (CJR) or bundled candidate, MSA-driven score, community vs. urban, academic vs. non-academic, postoperative network access (skilled nursing facility [SNF] only, home health, etc.), availability of healthcare professionals, availability of implants, and availability of surgical equipment.

[0212] These facility inputs can be obtained, for example but not limited to, via a survey (paper / digital), surgical planning tools (e.g., applications, websites, electronic medical records [EMR], etc.), hospital information databases (on the Internet), etc. Input data related to associated healthcare economics can also be obtained, including but not limited to the patient's socioeconomic profile, the expected reimbursement level the patient will receive, and whether the treatment is patient-specific.

[0213] These healthcare economics inputs can be obtained (e.g., but not limited to) via a survey (paper / digital), direct payer information, socioeconomic status databases (providing zip codes on the Internet), etc. Finally, data derived from a simulation of the procedure is obtained. The simulation inputs include implant size, position, and orientation. A custom or commercially available anatomical modeling software program (e.g., AnyBody or OpenSIM) can be used for the simulation. It should be noted that the above data inputs may not be available for every patient, and the available data will be used to generate a treatment plan.

[0214] Before surgery, patient data 810, 815, and healthcare professional data 825 can be obtained and stored in a cloud-based database or an online database (e.g., Figure 2C the surgical data server 180 shown in

[0215] ). Information related to the procedure is provided to the computing system manually using wireless data transfer or portable media storage. The computing system is configured to generate a case plan for the CASS 100. Generation of the case plan will be described below. It should be noted that the system can access historical data of previously treated patients, including implant sizes, positions, and orientations automatically generated by a computer-aided patient-specific knee instrument (PSKI) selection system or the CASS 100 itself. To this end, a surgical sales representative or a case engineer uploads case log data to the historical database using an online portal. In some embodiments, the data transfer to the online database is wireless and automated.

[0216] Figure 9 An example of determining a prediction equation from machine learning using seed data according to some embodiments is shown. An input signal including information from the online database (previously described) is introduced into the system as an input node. Each input node is connected to a series of downstream nodes for calculations in the hidden layer. Each node is typically represented by a real number (usually between 0 and 1), but the connections between the nodes also have weighted values that change as the system "learns". To train the system, a set of seed or training data is provided with associated known output values. The seed data is iteratively passed through the system, and the weighted values between the nodes are changed until the system provides a result that matches the known output. In this embodiment, the weighted values in the hidden layer of the network are captured in a weighted matrix that can be used to characterize surgical performance. The weighted matrix values are used as coefficients in the prediction equation that relates the database inputs to the results and satisfaction. Initially, the RNN will be trained with seed data developed from clinical studies and registry data. Once a sufficient number of cases have been established in the database, the system will use the historical data for system improvement and maintenance. Note that using the RNN will act as a filter to determine which input data has a greater impact on the output. The system operator can select a sensitivity threshold such that input data that has no significant impact on the output can be ignored and no longer captured for analysis.

[0217] Figure 10 Embodiment 1000 shows a way in which a surgical patient care system 820 can be used during a surgical procedure. Starting at step 1005, the surgical staff begins the surgery with CASS. CASS can be an image-based or image-free system, as is commonly understood in the art. Regardless of the type of system employed, at step 1010, the surgical staff can access or obtain a 3D representation of the relevant body anatomy of the patient (conventional probe mapping, 3D imaging with reference mapping, visual edge detection, etc.). In many cases, the 3D representation of the anatomy can be achieved mathematically by capturing a series of Cartesian coordinates representing the tissue surface. Example file formats include, but are not limited to,.stl,.stp,.sur,.igs,.wrl,.xyz, etc. The 3D representation of the relevant body anatomy of the patient can be generated preoperatively based on, for example, image data, or the 3D representation can be generated during the surgery using CASS.

[0218] Certain input data of the current patient can be loaded onto the computing system, for example, by wireless data transmission or using a portable storage medium. The input file is read into the neural network, and at step 1015, the resulting prediction equation is generated. Next, at step 1020, global optimization of the prediction equation (e.g., using direct Monte Carlo sampling, stochastic tunneling, parallel tempering, etc.) is performed to determine the optimal size, position, and orientation of the implant to achieve the best result or satisfaction level, and the corresponding resection to be performed is determined based on the implant size, position, and orientation. During the optimization phase, the system operator can choose to ignore aspects of the equation. For example, if the clinician deems the inputs related to the patient's economic status to be irrelevant, the coefficients related to these inputs can be removed from the equation (e.g., based on inputs provided through the GUI of CASS 100).

[0219] In some embodiments, instead of using an RNN to calculate the prediction equation, an experimental design (DOE) method is used to calculate it. DOE will provide sensitivity values that relate each input value to the output value. The valid input combinations are incorporated into the mathematical formula previously described as the prediction equation.

[0220] Regardless of how the prediction equation is configured or determined, optimization of this equation can provide a recommended, preferred, or optimized implant positioning, e.g., in the form of a homogeneous transformation matrix. The transformation mathematically determines the implant component size and the orientation of the implant component relative to the patient's anatomy. The Boolean intersection of the implant geometry and the patient's anatomy results in a volume representation of the bone to be removed. This volume is defined as the "cutting envelope". In many commercially available orthopedic robotic surgical systems, the bone removal tool (using optical tracking and other methods) is tracked relative to the patient's anatomy. Using position feedback control, the speed or depth of the cutting tool is adjusted based on the position of the tool within the cutting envelope (i.e., when the position of the tool tip is within the cutting envelope, the cutting tool will rotate, and when its position is outside the cutting envelope, the cutting tool stops or retracts).

[0221] Once the procedure is complete, at step 1025, all patient data and available result data are collected, including the implant size, position, and orientation determined by the CASS, and stored in the historical database. Any subsequent calculations of the objective equation by the RNN will in that way include data from previous patients, enabling continuous improvement of the system.

[0222] In addition to or as an alternative to determining implant positioning, in some embodiments, the prediction equation and associated optimization can be used to generate a resection plane for use with a PSKI system. When used with a PSKI system, the calculation and optimization of the prediction equation are completed prior to surgery. The patient's anatomy is estimated using medical image data (X-ray, CT, MRI). Global optimization of the prediction equation can provide the ideal size and position of the implant components. The Boolean intersection of the implant components and the patient's anatomy is defined as the resection volume. A PSKI can be generated to remove the optimized cutting envelope. In this embodiment, the surgeon cannot change the surgical plan intraoperatively.

[0223] The surgeon can choose to change the surgical case plan at any time before or during the surgery. If the surgeon chooses to deviate from the surgical case plan, the size, position, and / or orientation of the changed component are locked, and the global optimization is refreshed (using the techniques described previously) based on the new size, position, and / or orientation of the component to find the new ideal positions of the other components, as well as the corresponding resections that need to be performed to achieve the new optimized size, position, and / or orientation of the components. For example, if the surgeon determines that the size, position, and / or orientation of the femoral implant in a TKA need to be updated or modified intraoperatively, the position of the femoral implant will be locked relative to the anatomy, and the new optimal position of the tibia will be calculated (through global optimization) by considering the surgeon's changes to the size, position, and / or orientation of the femoral implant. Additionally, if the surgical system used to implement the case plan is robot-assisted (e.g., using or MAKO Rio), bone removal and bone morphology during the surgery can be monitored in real time. If the resection performed during the procedure deviates from the surgical plan, the processor can optimize the subsequent placement of the additional components by considering the actual resection that has been performed.

[0224] Figure 11A illustrates how a surgical patient care system 820( Figure 8 ) can be adapted to perform a case plan matching service. In this example, data 810 related to the current patient 805 is obtained and compared to all or a portion of a historical database of patient data and associated outcomes 815. For example, a surgeon can choose to compare the plan for the current patient to a subset of the historical database. The data in the historical database can be filtered to include, for example, only data sets with favorable outcomes, data sets corresponding to historical surgeries of patients with profiles the same as or similar to the current patient profile, data sets corresponding to a particular surgeon, data sets corresponding to a particular aspect of the surgical plan (e.g., surgeries that only preserve a particular ligament), or any other criteria selected by the surgeon or medical professional. For example, if the current patient data matches or correlates with the data of a previous patient who had a favorable outcome, the case plan of the previous patient can be accessed and adapted or adopted for the current patient. Prediction equations can be used in conjunction with intraoperative algorithms that identify or determine actions related to the case plan. Based on relevant information from the historical database and / or preselected information, the intraoperative algorithm determines a series of recommended actions for the surgeon to perform. Each execution of the algorithm results in the next action in the case plan. If the surgeon performs the action, the outcome is evaluated. The outcome of the surgeon performing the action is used to refine and update the input to the intraoperative algorithm for generating the next step in the case plan. Once the case plan has been fully executed, all data related to the case plan (including any deviations by the surgeon from performing the recommended actions) is stored in a database of historical data. In some embodiments, the system uses preoperative, intraoperative, or postoperative modules in a segmented manner, rather than for the entire continuum of care. In other words, the caregiver can prescribe any arrangement or combination of treatment modules, including using a single module. These concepts are illustrated in Figure 11B and can be applied to any type of surgery using the CASS 100.

[0225] Surgical procedure display

[0226] As described above with respect to Figure 1-2CAs described, the various components of the CASS 100 generate detailed data records during the operation. The CASS 100 can track and record the various actions and activities of the surgeon during each step of the operation and compare the actual activities with the preoperative or intraoperative surgical plan. In some embodiments, software tools can be employed to process the data into a format that can effectively "replay" the operation. For example, in one embodiment, one or more GUIs can be used that show all the information presented on the display 125 during the operation. This can be supplemented with graphs and images showing the data collected by different tools. For example, a GUI that provides a visual illustration of the knee during tissue resection can provide the measured torque and displacement of the resection device adjacent to the visual illustration to better provide an understanding of any deviation from the planned resection area that occurred. The ability to view a replay of the surgical plan or switch between different aspects of the actual operation and the surgical plan can be beneficial to the surgeon and / or surgical staff, enabling such personnel to identify any deficiencies or challenging aspects of the operation so that they can be modified in future operations. Similarly, in an academic setting, the aforementioned GUI can be used as a teaching tool to train future surgeons and / or surgical staff. Additionally, since the dataset effectively records many aspects of the surgeon's activities, it can also be used as evidence of the correct or incorrect performance of a particular surgical procedure for other reasons (e.g., legal or compliance reasons).

[0227] Over time, as more and more surgical data is collected, a rich database may be obtained that describes surgical procedures performed by different surgeons on different patients for various types of anatomies (knee, shoulder, hip, etc.). Moreover, aspects such as implant type and size, patient demographics, etc. can be further used to enhance the overall dataset. Once the dataset has been established, it can be used to train a machine learning model (e.g., an RNN) to predict how the operation will proceed based on the current state of the CASS 100.

[0228] Training of the machine learning model can be performed as follows. During surgery, the overall state of CASS100 can be sampled over multiple time periods. The machine learning model can then be trained to transform the current state of the first time period into a future state for a different time period. By analyzing the entire state of CASS 100 rather than individual data items, any causal effects of interactions between different components of CASS 100 can be obtained. In some embodiments, multiple machine learning models can be used instead of a single model. In some embodiments, the machine learning model can be trained not only with the state of CASS 100, but also with patient data (e.g., obtained from EMR) and the identity of surgical staff. This allows the model to make predictions with greater specificity. Moreover, if necessary, it allows surgeons to selectively make predictions based solely on their own surgical experience.

[0229] In some embodiments, the predictions or recommendations made by the aforementioned machine learning model can be directly integrated into the surgical process. For example, in some embodiments, the surgical computer 150 can execute the machine learning model in the background to make predictions or recommendations for upcoming actions or surgical conditions. Therefore, multiple states can be predicted or recommended for each period. For example, the surgical computer 150 can predict or recommend the state of the next 5 minutes in increments of 30 seconds. Using this information, the surgeon can use the "process display" view of the operation to allow visualization of future states. For example, Figure 11C shows a series of images that can be displayed to the surgeon, showing the implant placement interface. The surgeon can, for example, enter a specific time in the display 125 of CASS 100 or instruct the system to use tactile, verbal or other instructions to advance or rewind the display in a specific time increment to traverse these images. In one embodiment, the process display can be presented in the upper part of the surgeon's field of view in the AR HMD. In some embodiments, the process display can be updated in real time. For example, when the surgeon moves the resection tool around the planned resection area, the process display can be updated so that the surgeon can see how his or her actions affect other aspects of the operation.

[0230] In some embodiments, rather than simply using the current state of the CASS 100 as the input to the machine learning model, the input to the model can include a planned future state. For example, a surgeon can indicate that he or she is planning a particular osteotomy of the knee joint. This indication can be manually entered into the surgical computer 150, or the surgeon can provide the indication verbally. The surgical computer 150 can then generate a film showing the expected effect of the incision on the surgery. Such a film can show, at specific time increments, how the surgery would be affected if the expected course of action were to be performed, including, for example, changes in the patient's anatomy, changes in the position and orientation of implants, and changes related to the surgical procedure and instruments. The surgeon or medical professional can call or request this type of film at any time during the surgery to preview how the expected course of action would affect the surgical plan if it were to be performed.

[0231] It should be further noted that using a well-trained machine learning model and a robotic CASS can automate various aspects of the surgery, such that the surgeon only needs to be minimally involved, for example, by simply providing approval for each step of the surgery. For example, over time, robotic control using an arm or other means can be gradually integrated into the surgical process, with less and less manual interaction between the surgeon and the robotic operation. In such a case, the machine learning model can learn which robotic commands are needed to achieve certain states of the CASS implementation plan. Eventually, the machine learning model can be used to generate a film or similar view or display that can predict and preview the entire surgery from an initial state. For example, an initial state can be defined that includes patient information, surgical plan, implant characteristics, and surgeon preferences. Based on this information, the surgeon can preview the entire surgery to confirm that the plan recommended by the CASS meets the surgeon's expectations and / or requirements. Moreover, since the output of the machine learning model is the state of the CASS 100 itself, commands can be derived to control the components of the CASS to achieve each predicted state. Thus, in the extreme case, the entire surgery can be automated based solely on the initial state information.

[0232] Preoperative planning using anatomical modeling software

[0233] In some embodiments, anatomical modeling software, such as LIFEMOD TM, which can be used to develop preoperative or intraoperative plans to guide surgery. For example, in the context of hip surgery, if the anatomical modeling software has knowledge of the relationship between the spine and pelvis during multiple functional activities, the software can better predict the optimal implant position. Studies have shown that individuals with limited or abnormal spine-pelvis mobility have a higher risk of dislocation. For these patients, surgeons recommend taking lateral X-rays in several positions (e.g., standing, sitting, flexed standing) to understand how the spine and pelvis interact during various activities. These images can be fed into 3D biomechanical simulations to better predict the optimal implant position and orientation. Additionally, as an alternative to the manual process of taking radiographs, anatomical modeling software can also be used to simulate the positions of the lumbar spine and pelvis during a range of activities. In the context of knee surgery, if the anatomical modeling software knows the mechanical axis of the joint, the condylar axis, and the central axes of the femur and tibia, as well as the relationship between the existing flexion and extension gaps, the software can better determine how changes in the size and posture (position and orientation) of the implant components can affect the mechanics of the replaced knee. More specifically, if the software integrates the relationship between these variables across the range of motion and the exemplary forces of a given patient's activities, the performance of the implant can be modeled.

[0234] Figure 12A-12C Some outputs are provided that the anatomical modeling software can use to visually depict the results of modeling hip joint activities using the anatomical modeling software. Figure 12A and 12B shows a hip range of motion (ROM) graph. In this case, the anatomical modeling software can be used to perform a ROM "test" by placing the patient in various positions and modeling the movement of the hip joint as the patient's leg moves in different positions. For example, the software can virtually simulate the position and orientation of the implant relative to the bone anatomy during various activities that the patient may experience postoperatively. These can include standard stability checks performed during a total hip procedure or even activities that have a high risk of impingement and dislocation (crossing legs while seated, deep flexion while sitting, hyperextension while standing, etc.). After performing the test, the collected ROM data can be presented on a ROM table as shown in Figure 12A . Additionally, the anatomical modeling software can identify any ROM of impingement where there is abnormal contact and wear contact between the patient's anatomy and the implant components. As shown in Figure 12B , after determining the ROM that is free from impingement, it can be graphically overlaid on a 3D model of the patient's anatomy.

[0235] Figure 12CA 2D graphic is shown which presents recommendations for the desired or "safe" range of positions for positioning a hip implant within the acetabulum. In such cases, anatomical modeling software can be used to identify the safe range of placement positions by modeling functional activities up to the point of failure. For example, initially, the anatomical modeling software can assume that the implant can be placed anywhere within a large bounding box around the anatomical structure of interest. Then, for each possible implant position, the anatomical modeling software can be used to test whether that position causes failure of the anatomy or the implant under normal functional activities. If failure occurs, the position is discarded. Once all possible points have been evaluated, the remaining points are considered "safe" for implant placement. As Figure 12B shown, for hip surgery, the safe positions can be indicated graphically by abduction relative to anteversion. In this case, the Lewinnek safe zone is superimposed on the graphic. As is commonly understood in the art, the Lewinnek safe zone is based on clinical observations that if the acetabular cup is placed within 30 degrees - 50 degrees of abduction and 5 - 25 degrees of anteversion, then dislocation is less likely to occur. However, in this case, some of the "safe" positions for the patient (depicted in "red") are outside the Lewinnek safe zone. Thus, based on the characteristics of the patient's anatomy, more flexibility is given to the surgeon to deviate from the standard recommendations.

[0236] An additional output of the anatomical modeling software can be a 3D rendering of the final implant components relative to the bone. In some embodiments, the 3D rendering can be displayed in an interface that allows the surgeon to rotate around the entire image and view the rendering from different perspectives. The interface can allow the surgeon to articulate the joint to visualize how the implant will perform and identify locations where impingement, misalignment, excessive strain, or other problems may occur. The interface can include functionality that allows the surgeon to hide certain implant components or anatomical features in order to best visualize certain regions of the patient's anatomy. For example, for a knee implant, the interface can allow the surgeon to hide the femoral component and only display the bearing surface and the tibial component in the visualization. Figure 12D Exemplary visualizations are provided in the context of hip implants.

[0237] In some embodiments, the anatomical modeling software can provide an animation that shows the positions of the implant and the bone when the patient performs different body activities. For example, in the context of knee surgery, deep knee flexion can be presented. For hip surgery, the spine and pelvis can be shown during walking, sitting, standing, and other activities that may represent challenges to the implant.

[0238] Revision hip or knee arthroplasty surgery involves removing one or more existing hip / knee implants and replacing the removed implants with new implants in a single surgical procedure. In some embodiments, the preoperative planning phase may anticipate dealing with bone spurs or other tasks in addition to the planar resection required to receive the new implant. For example, in some embodiments, for revision surgery, the output of the anatomical modeling software may be a 3D image or bone map showing the placement of each component and problem areas that may require special preparation by the surgeon. The software in the surgical plan can highlight areas that will prevent the implant from seating fully, to show the surgeon which areas of bone need to be resected. The software can be interactive, allowing the surgeon to virtually "expand" or "grind" the bone open in order to optimally prepare the bone to receive the implant. This can be performed, for example, by allowing the surgeon / engineer to selectively remove individual pixels or groups of pixels from an X-ray image or 3D representation by touching areas or using the "virtual drill" component of the interface. In some embodiments, images in a hybrid mode can be registered and overlaid to provide a view of the structure of the anatomical area of interest, anatomical structures, etc. In some embodiments, the anatomical modeling software can provide recommended areas for removing bone burrs. For example, a machine learning model can be trained based on how surgeons (or other surgeons) have performed deburring in the past to identify areas for deburring. Then, using the X-ray image or other patient measurements as input, the machine learning model can input the recommended deburring areas for the surgeon to view during the virtual deburring procedure. In some embodiments, the virtual deburring process can be performed interactively with other aspects of the anatomical modeling discussed above. The deburring process can also be used with primary joint arthroplasty.

[0239] Obtaining high resolution of critical areas during hip surgery using a point probe

[0240] The use of a point probe is described in U.S. Patent Application No. 14 / 955,742, titled "Systems and Methods for Planning and Performing Image Free Implant Revision Surgery", the entire content of which is incorporated herein by reference. Briefly, an optically tracked point probe can be used to map the actual surface of the target bone that requires a new implant. The mapping is performed after removal of a defective or worn implant and after removal of any diseased or other unwanted bone. By brushing or scraping the entire remaining bone with the tip of the point probe, multiple points can be collected on the bone surface. This is referred to as tracking or "drawing" the bone. The points collected are used to create a three-dimensional model or surface map of the bone surface in a computer planning system. The created 3D model of the remaining bone is then used as a basis for planning the surgery and the necessary implant dimensions. An alternative technique for determining the 3D model using X-rays is described in U.S. Patent Application No. 16 / 387,151, titled "Three Dimensional Guide with Selective Bone Matching", filed on April 17, 2019, the entire content of which is incorporated herein by reference.

[0241] For hip applications, point probe mapping can be used to obtain high-resolution data of key regions such as the acetabular rim and acetabular fossa. This can enable the surgeon to obtain a detailed view before starting reaming. For example, in one embodiment, the point probe can be used to identify the bottom (fossa) of the acetabulum. As is well known in the art, in hip surgery, it is important to ensure that the bottom of the acetabulum is not damaged during reaming to avoid breaching the medial wall. If the medial wall is inadvertently breached, the surgery will require an additional bone grafting step. With this in mind, during the surgical procedure, information from the point probe can be used to provide operating guidance for the acetabular reamer. For example, the acetabular reamer can be configured to provide haptic feedback to the surgeon when the surgeon reaches the bottom or otherwise deviates from the surgical plan. Alternatively, the CASS 100 can automatically stop the reamer when the bottom is reached or when the reamer is within a threshold distance.

[0242] As an additional safeguard, the thickness of the region between the acetabulum and the medial wall can be estimated. For example, once the acetabular rim and fossa are mapped and registered to the preoperative 3D model, the thickness can be easily estimated by comparing the position of the acetabular surface with the position of the medial wall. Using this knowledge, the CASS 100 can provide an alert or other response in the event that any surgical activity is predicted to protrude through the acetabular wall during reaming.

[0243] The point probe can also be used to collect high-resolution data on common reference points used to orient 3D models to a patient. For example, for pelvic plane landmarks such as the ASIS and symphysis pubis, a surgeon can use the point probe to map the bone to represent the true pelvic plane. With a more complete view of these landmarks, the registration software will have more information to orient the 3D model.

[0244] The point probe can also be used to collect high-resolution data describing proximal femur reference points that can be used to improve the accuracy of implant placement. For example, the relationship between the tip of the greater trochanter (GT) and the center of the femoral head is commonly used as a reference point for aligning femoral components during hip arthroplasty. The alignment height depends on the correct position of the GT; thus, in some embodiments, the point probe is used to map the GT to provide a high-resolution view of the area. Similarly, in some embodiments, a high-resolution view of the lesser trochanter (LT) may be useful. For example, during hip arthroplasty, the Dorr classification helps select a stem that will maximize the ability to achieve a press-fit during surgery, thereby preventing micromotion of the femoral component postoperatively and ensuring optimal bone ingrowth. As is commonly understood in the art, the Dorr classification measures the ratio between the canal width at the LT and the canal width 10 cm below the LT. The accuracy of the classification highly depends on the correct position of the relevant anatomy. Thus, mapping the LT to provide a high-resolution view of the area may be advantageous.

[0245] In some embodiments, the point probe is used to map the femoral neck to provide high-resolution data that allows a surgeon to better understand where to make the neck cut. The navigation system can then guide the surgeon as they perform the neck cut. For example, as is understood in the art, the femoral neck angle is measured by placing one line below the center of the femoral stem and a second line below the center of the femoral neck. Thus, a high-resolution view of the femoral neck (and possibly also the femoral stem) will provide a more accurate calculation of the femoral neck angle.

[0246] High-resolution femoral head and neck data can also be used for navigated surface reconstruction procedures, where software / hardware helps the surgeon prepare the proximal femur and place the femoral component. As is commonly understood in the art, during hip surface reconstruction, the femoral head and neck are not removed; rather, the head is trimmed and covered with a smooth metal covering. In this case, it would be advantageous for the surgeon to map the femur and the cap such that a precise assessment of their respective geometries can be understood and used to guide the trimming and placement of the femoral component.

[0247] Registering preoperative data to the patient anatomy using the point probe

[0248] As described above, in some embodiments, a 3D model is developed based on 2D or 3D images of an anatomical region of interest during the pre-operative phase. In such embodiments, registration between the 3D model and the surgical site is performed prior to the surgical procedure. The registered 3D model can be used to track and measure the patient's anatomical structures and surgical tools intraoperatively.

[0249] During the surgical procedure, fiducials are acquired to facilitate registration of the pre-operative 3D model to the patient's anatomy. For knee surgery, these points may include the center of the femoral head, the distal femoral axis point, the medial and lateral epicondyles, the medial and lateral malleoli, the proximal tibial mechanical axis point, and the tibial A / P direction. For hip surgery, these points may include the anterior superior iliac spine (ASIS), the pubic symphysis, points along the acetabular rim and within the hemisphere, the greater trochanter (GT), and the lesser trochanter (LT).

[0250] In revision surgery, the surgeon may draw certain regions containing anatomical defects to better visualize and navigate implant insertion. These defects can be identified based on the analysis of pre-operative images. For example, in one embodiment, each pre-operative image is compared to a library of images showing "healthy" anatomy (i.e., without defects). Any significant deviation between the patient image and the healthy image can be marked as a potential defect. Then, during the surgery, the surgeon can be warned of a possible defect by a visual alert on the display 125 of the CASS 100. The surgeon can then draw regions to provide the surgical computer 150 with more detailed information about the potential defect.

[0251] In some embodiments, the surgeon can use non-contact methods to perform registration of the intraosseous incision of the bone anatomy. For example, in one embodiment, laser scanning is employed for registration. The laser stripe is projected onto the anatomical region of interest, and the height variations of the region are detected as variations of the line. Other non-contact optical methods, such as white light interferometry or ultrasound, can also alternatively be used for surface height measurement or for registering the anatomical structure. For example, where there is soft tissue between the registration point and the bone being registered (e.g., the ASIS, the pubic symphysis in hip surgery), ultrasound technology may be beneficial, thus providing a more precise definition of the anatomical plane.

[0252] Surgical Navigation with Mixed Reality Visualization

[0253] In some embodiments, a surgical navigation system utilizes an augmented reality (AR) or mixed reality (MR) visualization system to further assist a surgeon during robot-assisted surgery. By using graphical and informational overlays (e.g., holographic or heads-up display / HUD) to guide the surgical procedure, conventional surgical navigation can be enhanced with AR. Exemplary systems allow for the implementation of multiple head-mounted devices to share the same hybrid or different reality experiences in real time. In a multi-user usage scenario, multiple user profiles can be implemented for selective AR display. This can allow the head-mounted devices to work together or independently, displaying different subsets of information to each user.

[0254] Embodiments utilizing AR / MR include surgical systems for enhancing surgery by visually tracking an operation in a surgical environment, the visual tracking including a head-mounted display (HMD) worn by one or more individuals performing surgical functions. For example, a surgeon may have an HMD, and some or all of the nurses or laboratory technicians assisting the surgeon (or residents, other surgeons, etc.) may have their own HMDs. By using the HMD, the surgeon can view information related to the surgery, including information traditionally associated with enhanced robotic surgery, without the surgeon having to shift their field of view away from the patient. This can make the surgery faster as the surgeon does not need to perform a context switch between the display and the patient. In some embodiments, a virtual holographic monitor can be selectively presented to the surgeon, which reflects the display of an LCD screen conventionally mounted on a cart during surgery. The HMD interface can allow the surgeon to move the holographic monitor to appear fixed in space at any location in the space of her choosing, such as adjacent to the exposed patient tissue in front of the surgical drape.

[0255] In some embodiments, various types of HMDs can be used. Generally, an HMD includes a headgear worn on a user's head and a communication interface. The communication interface can be wired, such as USB, serial port, SATA, or a proprietary communication interface, or preferably wireless, such as Wi-Fi or Bluetooth (however, timing and bandwidth limitations can restrict the practical selection of some faster conventional communication interfaces, such as Wi-Fi or USB 3.0). An exemplary HMD also has a power source (e.g., a battery or a hardwired power connector), an on-board computer (including a processor, GPU, RAM, and non-volatile data and instruction memory), and one or more displays for overlaying information within the user's field of view. An exemplary HMD may also include an array of cameras that capture 3-D images of the environment (which may include optical sensors, IR sensors, and illumination sources). In some embodiments, the HMD or an external processor can use image processing algorithms to create a model of the user's environment, which identify important features of the environment and create a 3D model of the environment by processing stereo or IR data. By calibrating the HMD's display to the user's field of view, information displayed on a holographic display (e.g., using direct retinal or semi-reflective projection) can be reliably overlaid onto the user's field of view to enhance the user's observation of the environment.

[0256] The HMD worn by surgical staff can include commercially available off-the-shelf HMDs, such as the Oculus Rift TM 、Microsoft HoloLens TM 、Google Glass TM 、Magic Leap One TM or custom-designed hardware for the surgical environment. In some embodiments, supplementary hardware is added to a commercially available HMD to enhance its use in the surgical environment using off-the-shelf HMD components and custom hardware. In some embodiments, the HMD hardware can be integrated into traditional surgical caps and masks, thereby allowing the HMD to act as a personal protective device and allowing information to be displayed by reflecting the light of the mask that the surgeon is already familiar with wearing. There are various ways in the HMD technologies on the market to provide a mixed reality environment for users. For example, virtual reality headsets, such as the Oculus Rift TM 、HTC Vive TM 、Sony PlayStation VR TM or Samsung Gear VR TM, blocking the user's natural vision and replacing the user's entire field of view with a stereoscopic screen to create a 3D environment. These systems can use one or more cameras to recreate an enhanced form of the three-dimensional environment for display to the user. This allows the natural environment to be captured and redisplayed to the user with mixed reality components. Other AR headsets, such as Google Glass TM and Microsoft HoloLens TM , enhance reality by providing the user with supplementary information that is displayed as holograms within the user's field of view. Since the user observes the environment directly or through clear lenses, the additional display information projected from a reflective surface in front of the user's eyes or directly onto the user's retina is translucent to the user.

[0257] Commercially available HMDs typically include one or more outward-facing cameras to collect information from the environment. These cameras can include visible light cameras and IR cameras. The HMD can include a light source for the auxiliary camera to collect data from the environment. Most HMDs include some form of user interface that the user can use to interact with the processor via the HMD's display. In some systems, the user interface can be a handheld remote controller that is used to select and engage the displayed menu. This may not be ideal for a surgical environment due to sterilization issues and fluid-covered hands. Other systems, such as Microsoft HoloLens TM and Google Glass TM , use one or more cameras to track gestures or MEMS accelerometers to detect the movement of the user's head. The user can then use gestures to interact with the virtual interface. For example, a virtual keyboard can be holographically displayed, and the camera can track the movement of the user's fingers to allow the user to type on the virtual floating keyboard. Some HMDs can also have a voice interface. In addition to the display, the headset can provide tactile feedback through actuators or provide an audio signal to the user through the headset or speakers. Other on-board sensors of the HMD can include gyroscopes, magnetometers, laser or optical proximity sensors. In some embodiments, the HMD can also have a laser or other projection device that allows information to be projected into the environment rather than projected onto the user holographically.

[0258] Although AR headsets can provide a more natural feeling to the user than VR because most of the images the user sees are natural, it may be difficult to correctly superimpose and align the displayed information with the user's viewing point. The industry has had many software initiatives to address this issue, which have been pioneered by AR headset manufacturers. Therefore, AR headsets and VR headsets typically come with the software tools necessary to superimpose information onto the user's field of view to align the information with what the user sees in the environment.

[0259] In some embodiments, information similar to that displayed on a traditional cart-mounted monitor is provided to a surgeon as part of a robotic-assisted surgical system such as the NAVIO surgical system. In some embodiments, different HMDS worn by different people in the operating room can display different information at any time. For example, a surgeon can see information related to what is currently in his field of view, while the HMD worn by a surgical resident can display what the attending surgeon sees and any enhanced camera views that the attending surgeon sees. In some embodiments, the resident can see additional patient information that helps the resident learn or communicate to the surgeon, such as preoperative imaging, patient documentation, information from tool or medical device manufacturers, etc.

[0260] The HMD includes one or more cameras that capture the wearer's field of view (or a wider or narrower form thereof) and provide the images to a processor, thereby allowing the processor to analyze the two-dimensional images captured by the HMD and their relationship to the 3D model of the operating room. For example, one or more tracking cameras of the HMD can capture the presence of fiducial markers on the patient's bone and / or tools to determine how the wearer's angle relates to the 3D model of the patient and / or tools. This can allow the image processor to extract features from the captured images, such as bone / tissue and tools, and use this information to display enhanced information on the image that the wearer is viewing. For example, when a surgeon is operating on a knee, the camera on the surgeon's HMD can capture what the surgeon sees, allowing the image processing software to determine the location in three-dimensional space that the surgeon is looking for and identify the specific patient features that the surgeon is looking for. These specific patient features can be located in the two-dimensional image by the image processing software using pattern matching or machine learning techniques that are informed of the location that the surgeon sees in the surgical scene.

[0261] For example, a surgeon can observe the tibial plateau and femoral condyles. The camera on the HMD will capture this image in real time (including actual processing and communication delays) and send the image to the image processing software. The image processing software can identify the objects being observed as the tibial plateau and femoral condyles (or receive cues based on the perspective) and look for patterns in the image to identify the extent of these features within the surgeon's field of view. Then, information about the tibial plateau and femoral condyles can be overlaid on the image that the surgeon sees. If fiducial markers are available in the image (or the most recent image), or if the surgeon's HMD includes fiducial markers captured by a robotic vision system or by the cameras of other HMDS in the room to allow calculation of the pose of the surgeon's HMD camera, the software can accurately position the two-dimensional image relative to the three-dimensional model of the surgical scene.

[0262] In some embodiments, information is holographically overlaid in the user's field of view. In some embodiments, information may be digitally projected from the HMD into the environment. For example, a laser array MEMS mirror coupled to the headgear may project an image directly onto a surface in the environment. Because this projection originates from approximately the same location as the user's eyes, this projection can be easily juxtaposed with the user's field of view, thereby overlaying information into the environment in a more robust manner than presenting a floating hologram to the user. For example, as Figure 13 shown in, the HMD 1100 projects a computer-generated image of a cutting envelope onto a portion of the patient's knee 1102 to accurately indicate to the wearer where she should cut on the bone in the knee without distracting the wearer or interfering with her peripheral vision.

[0263] Surgical systems using an optical tracking mode, such as the NAVIO system, may be well-suited for use with an HMD. One or more cameras included in the HMD make it particularly convenient to adapt the HMD for use in an operating room. As Figure 14 shown in, one or more cameras 1110 mounted to a cart or fixed in the surgical environment use optical and IR tracking to capture the positions of fiducial markers 1112, 1114 mounted to tools and the patient's bone. Adding one or more HMDs 1100 to this environment can supplement this tracking system by providing an additional perspective of optical or IR tracking. In some embodiments, multiple HMDs 1100 may be used simultaneously in the operating room, thereby providing various perspectives to assist in tracking fiducial markers of tools and patient anatomy. In some embodiments, using multiple HMDs provides a more robust tracking mode because additional perspectives can be compared, weighted, correlated, etc. in software to verify and refine the 3D model of the environment. In some embodiments, cameras permanently mounted in the operating room (mounted to walls, carts, or surgical lights) may utilize higher quality IR components and optical components than those in the HMD. When there is a disparity in the quality of the components, 3D model refinement using multiple perspectives can assign heuristic weights to different camera sources, thereby allowing refinement based on higher quality components or more reliable perspectives.

[0264] Given the rapid development of optical sensor technology driven by the mobile device market, HMD optical sensor technology is evolving rapidly. In some embodiments, a camera array mounted on a trolley is not necessary for optical tracking modalities. Optical sensors and IR sensors on the HMDs worn by surgeons, residents, and nurses can provide sufficient viewpoints to track fiducial markers on the patient and tools without the need for a stand-alone trolley. This can reduce the cost of adding an HMD to a conventional tracking modality or reduce the overall system cost. The price of HMDs with existing components has been rapidly decreasing as they are accepted by the consumer market. In some embodiments, a camera mounted on a trolley or a wall-mounted camera can be added to a system using optical sensors and IR sensors of lower quality than traditional trolley-mounted tracking systems to supplement embodiments that otherwise rely entirely on the IR sensors and optical sensors of the HMD. (Optical sensors include IR sensors / cameras and optical sensors / cameras and can generally be described as cameras, but for clarity, these components can be listed separately; embodiments can include any subset of available optical sensors.)

[0265] As Figure 14 shown, each camera array (of the trolley-mounted tracking system 1110 and each HMD 1100) has a camera pose that defines the reference frame of the camera system. In the case of a trolley-mounted or wall-mounted camera, the camera pose can be fixed throughout the operation. In the case of an HMD camera, the camera pose will generally change during the surgery as the user moves their head. Thus, in some embodiments where the HMD works with a trolley-mounted or wall-mounted tracking system to supplement tracking information, fiducial markers are used to at least identify the position of the other camera systems in the environment. In some embodiments, fiducial markers can be rigidly applied to these camera systems so that the other camera systems can identify the position and pose of each other camera system. This can be used to calculate the camera viewpoints to determine a robust 3D model of the surgical environment. It should be understood that when the user moves in the HMD, the field of view may ignore the other camera systems, but the fiducial markers on the system allow the camera systems to quickly identify the other cameras and their poses after moving back into the field of view.

[0266] Once the camera has identified the position and / or pose of the other cameras in the environment, the camera can identify the position and orientation of the fiducial markers fixed to the patient's bone or tool. When two cameras have the same fiducial marker in their field of view, a central processor or peer-to-peer processing can correlate the position and orientation of those markers relative to each camera to produce a model of the 3D environment that includes the position and pose of each camera and the position and pose of each viable surgical bone of the patient. In Figure 14In the example shown, fiducial markers 1112 and 1114 mounted to each patient bone include four reflective points with known geometry. Each of these fiducial markers 1112 and 1114 has an orientation defined in three-dimensional space. Since this geometry is known, this orientation can be related through a registration process to define a reference frame for each bone as a transfer function of the orientation of the corresponding fiducial marker. Thus, when the orientation can be determined from camera information by a processor, the pose of each bone 1116 and 1118 relative to the field of view of each camera can be calculated. In this example, the headset 1100 and the tracking system 1110 mounted on the cart communicate wirelessly with each other or with a central processor.

[0267] The robustness of the 3D model is improved by the number of fiducial markers observed by multiple cameras. Since cameras capture the analog world using digital signals and are limited by the quality of the optical components, the accuracy with which each camera can locate fiducial markers in space includes a degree of error. Using a multi-camera system or HMD can reduce this error, resulting in a more robust 3D model, the level of accuracy of which cannot be achieved with a single tracking mode mounted on a cart. Any method of calculating a 3D model of the environment from pose information captured by multiple cameras known in the art can be applied to a multi-HMD operating room environment.

[0268] Figure 15An exemplary method 1200 for using an AR head-mounted device in a surgical environment using fiducial markers on a patient and tools is shown. Each camera system in the operating room may include a camera on the HMD and fixed cameras, such as cameras mounted to a cart or wall, and the camera systems may perform steps 1202 through 1212. Each camera system may be initialized at step 1202. This may include preparing any startup procedures and calibrations required for the camera to operate during the surgery. At step 1204, the camera system captures images and applies any techniques to prepare these images for processing, including removing distortion or any image preprocessing. At step 1206, each camera system may optionally estimate its own pose. This may include gyroscopic sensors, accelerometer sensors, magnetic sensors or compasses and models based on previous poses updated with image information. At step 1208, each camera system attempts to identify any other camera systems in its field of view. This may be used to determine the geometric relationships between the various camera systems. In some embodiments, this identification step looks for certain markers in the visible field that may include fiducial markers placed on cameras or IR beacons, etc. At step 1210, each camera system identifies fiducial markers in the captured images. The fiducial markers may be physical markers placed on a patient's anatomy, tools, other cameras or signs in the environment or physical markers fixed to a patient's anatomy, tools, other cameras or signs. A processor associated with each camera system may determine pose information based on these fiducial markers. This pose information may include the pose of the camera system and / or the pose of an object in the environment. For example, a tool may have multiple IR / UV reflective or unique color or patterned markers placed on it such that the position and orientation of the tool can be identified from one or more images. Similarly, multiple fiducial markers may be placed on a camera to allow the position and orientation of the camera to be calculated by other camera systems. At step 1212, this information about the position and orientation of other objects in the environment may be reported wirelessly or over a hard-wired network to a central processor of a management system. The above steps (1204 - 1212) may be repeated as each camera system continuously captures images and prepares them for analysis.

[0269] At step 1214, the central processor receives image data and pose or fiducial marker information from each camera system. At step 1216, the central processor creates or updates models of all the camera systems in the environment and their poses relative to the environmental reference frame. At step 1218, the central processor identifies fiducial markers in the received images to associate the markers captured by the multiple camera systems. By capturing such markers in multiple fields of view, different perspectives of the object can be used to optimize the determination of the position and orientation of the object in the environment. Once correlated, at step 1220, the central processor can calculate the position and orientation of each object with a fiducial marker, such as tools, environmental signs, other HMDS or camera systems, and patient anatomy. At step 1222, this information is used to update the 3D model of the environment, calculate the position and orientation of the objects relative to a fixed reference frame, and identify the poses of the reference frames defined by each object. For example, a patient's femur has a given pose based on the way the patient is lying in the operating room. The femur also has its own reference frame, which can be used to identify portions of the femur that may need to be resected during the surgery when correlating preoperative imaging with the structure of the patient's femur. At step 1224, the central processor can use the updated 3D model to send information about the environment to the surgical robotic system and any HMDs in the operating room. The central processor continues to receive images and update the environmental model. In some embodiments, this process can be enhanced for any object in the operating room via other sensors, such as accelerometers, compasses, etc. Once this 3D model is generated, the central processor is free to interact with the various HMDS according to many of the software applications described herein.

[0270] Figure 16It is a system diagram of the augmented reality system 1300 used during surgery. In this example, fiducial markers 1302A-G are placed on each HMD and camera system, as well as on the femur and tibia of the patient. In this example, a partial or total knee replacement is being performed. A camera system 1110 mounted on a cart (e.g., a camera system that can be used for the NAVIO surgical system) is placed to have a view of the surgical scene. Fiducial markers 1302E-F with multiple fiducial markers (e.g., three or more spherical IR reflective markers) are temporarily mechanically fixed to the femur and tibia of the patient. This allows the camera system 1110 mounted on the cart to track the pose and movement of the tibia and femur. This can be used to determine the pivot center between these bones, as well as other information about the patient's anatomy during movement. The surgical robot 1306 can also use this information to determine the ideal placement for cutting and replacing the knee portion. The camera system 1110 mounted on the cart communicates with a central processor 1304 via a local area network (e.g., a secure Wi-Fi network), and the central processor includes software and memory sufficient to process images from the camera mounted on the cart to determine an environmental model and calculate any information that may be useful for the surgical robot 1306 to assist the surgeon during the surgery. In some embodiments, the central processor 1304 is part of the cart that supports the camera system. In these embodiments, the camera system 1110 mounted on the cart can communicate directly with the central processor 1304. Then, the central processor 1304 communicates with any robotic system 1306 used during the procedure.

[0271] In addition to this traditional robotic surgical system, doctors and internal staff wear multiple HMDs 1100, 1100A, and 1100B during the procedure. Each HMD has an identifier that allows it to be recognized in the visual plane of the camera system 1110 mounted on the cart. In some embodiments, the identifier includes an IR emitter that sends a binary modulated code that identifies a particular HMD or wearer.

[0272] In this embodiment, each HMD 1100-1100B has a camera array (IR and / or visible spectrum) that captures the user's field of view (or a sub-part or wider angle thereof) from multiple positions, thereby generating a stereoscopic (or higher-order) view of the scene for image processing. This allows each headset to capture image data sufficient to provide three-dimensional information about the scene. Each HMD captures a different foreground of the surgical scene. As Figure 15As explained, the camera array of each HMD captures an image of the scene, estimates the current pose of the HMD (which can also be estimated by non-optical sensors), and identifies other cameras and any fiducial markers in the field of view. In some embodiments, the HMD has an IR emitter to illuminate the reflective fiducial markers. These captured images can then be processed on the HMD or sent to a central processor for more powerful image processing. In some embodiments, a preprocessing step is performed on each HMD to identify fiducial markers and other cameras and estimate the poses of the fiducial markers and other cameras. Each HMD can communicate with other HMDs or with the central processor 1304 via a local network 1310, such as a secure Wi-Fi network. The central processor 1304 receives image information from each HMD and further optimizes the 3D model of the surgical space.

[0273] Based on the images captured from each HMD, the central processor 1304 can determine which objects in the three-dimensional model of the operating room correspond to the reference features in the two-dimensional image plane of each HMD. This information can then be transmitted back to the HMD via the network 1310. In some embodiments, each HMD periodically receives information about the three-dimensional model, but uses on-board image processing to track objects based on the three-dimensional model information received from the central processor. This allows the headset to accurately track objects within the field of view in real time without latency caused by network communication.

[0274] In some embodiments, the HMD 1100 receives information about an object from the central processor 1304 and processes current and recent images to identify salient features of the object corresponding to the three-dimensional model information received from the central processor. The HMD1100 uses local image processing to track those features (and associated objects) in the visual plane. This allows the HMD 1100 to overlay information related to the object at the reference pixel positions.

[0275] For example, the surgeon's HMD 1100 can capture fiducial markers 1302E and 1302F related to the tibia and femur. Based on communicating with the central processor 1304 to identify the three-dimensional models of the tibia and femur, the surgeon's HMD 1100 can identify which features in the two-dimensional image correspond to the features of the tibial plateau and femoral condyles. By tracking the fiducial markers on these bones or by tracking the image features identified as the tibial plateau or femoral condyles, the surgeon's HMD 1100 can track these features of the bones in real time (taking into account any processing and memory latency) without having to worry about network latency. If the software running on the HMD 1100 wishes to overlay a visual indication of the cutting location on the tibial plateau, the HMD can track the tibial plateau as the surgeon's head moves or the patient's leg moves. The surgeon's HMD 1100 can accurately estimate where the reference points of this bone feature are and where to display the enhanced holographic features.

[0276] In some embodiments, the HMDS 1100-1100B can communicate directly to assist each other in updating any location models. Using peer-to-peer communication, the HMDS 1100-1100B can show their environmental models to other HMDS, allowing them to use arbitration rules to correct their own models.

[0277] In some embodiments, objects within the surgical space can include QR codes or other visual indicators that can be captured by the cameras on the HMD1100 to convey information about the object directly to the HMD. For example, a surgical tray 1308 with tools for a given surgery can have a QR code indicating the identity of the tray. By consulting a database, the HMD 1100 that captures the QR code can precisely identify the objects and surgical tools on the tray before the surgery begins. This can allow the HMD 1100 to identify important objects related to the surgery, such as various knives, cutting tools, sponges, or implantable devices. The HMD 1100 can then track those objects within the surgical space, taking into account the last known location of those objects. In some embodiments, the HMDS 1100-1100B can share information about the last known location of objects within the operating room so that each headset can more quickly or easily identify the objects entering the field of view of each HMD. This can help the surgeon quickly determine the tool or object to grasp for the next step of the surgery. For example, the tray can include a series of cutting guides for various surgeries. However, each patient may only need a single cutting guide for use on their femur. The HMD 1100 can identify the cutting guide within the tray based on the tray's QR code and initial layout and track the single cutting guide. The holographic display of the HMD 1100 can overlay an indication of the cutting guide for a given step in the procedure onto the surgeon.

[0278] In different embodiments, various information can be displayed to the user of the HMD. For example, in Figure 17A , a holographic overlay 1315 of the resection area on the supracondylar femur is presented to the user to indicate where the surgeon should remove tissue for placement of a replacement knee portion that will adhere and be fixed to the femoral head. Figure 17A is the view seen by the surgeon, including the natural scene and the holographic shapes placed on the bone surface. In some embodiments, rather than using holograms as part of a conventional AR display, lasers or projectors mounted to the HMD can project the information directly onto the surface of the bone. The cameras on the HMD can provide a feedback loop to ensure proper and consistent placement of the protrusions on the bone.

[0279] In Figure 17BAn exemplary display of a three-dimensional model is shown. Conventionally, this display may appear on the computer screen of a robotic surgery system. It accurately indicates on the three-dimensional model where the surgeon's tool should go and where the tool should cut the bone. This information can be adapted for an AR display, so that a portion of the three-dimensional model of the bone is holographically overlaid on the bone, thereby allowing the resection area to be shown to a user observing the real-world bone. In some embodiments, only the three-dimensional model of the resection area is displayed, while the three-dimensional model of the bone is compared with the field of view of the HMD to ensure the correct placement of the three-dimensional model of the holographic display of the resection area. Additionally, Figure 17B Any menu or additional information shown therein can be presented to the user of the HMD as a holographic menu, allowing the user to select certain user interface menus via a standard AR user interface to learn more information or views.

[0280] Figure 18A Additional three-dimensional model views that can be shown to the user are presented. Figure 18A Points identified by a point probe during femoral head mapping are depicted. Such information can be shown to the surgeon via a holographic image of the portion of the bone that has been probed to identify the shape of the femoral head. In some embodiments, this information can be shown on the HMD of a resident. Figure 18A The images shown therein can be displayed on a conventional computer monitor in the operating room. Individual portions of the monitor, such as the three-dimensional model of the femoral head, can be holographically displayed onto the bone using an AR display. Again, any of these menus shown can be presented to the user's headgear, or can be shown on a two-dimensional LCD type monitor on a cart in the room. These menus can be selected by the user of the HMD to change the view or obtain additional information. The user of the AR HMD can use any conventional AR selection means, such as an air click or head / hand gestures that can be detected by sensors or cameras in the HMD, to select the menu. In some embodiments, in the case of using a robotic cutting tool, this display can indicate where the robot is to cut, allowing the surgeon to make a visual confirmation before the cutting begins.

[0281] Figure 18B Another three-dimensional model of the femur is shown, which shows the proper placement of a femoral cutting guide. This can be shown to the surgeon on a hologram to accurately indicate where the femoral cutting guide should be placed. During the alignment of the cutting guide, the surgeon can consult this holographic overlay to ensure that the placement is generally correct. The robotic vision system in the room can provide a final confirmation before the cutting guide is installed. In hip surgery, this same technique can be used to assist the surgeon in placing a custom femoral neck cutting guide or to assist in placing an acetabular jig for the cup. In some embodiments, Figure 18B The display in Figure 18AThe menu selection in [the system] allows the surgeon to switch between the step history and a model of the next steps that can be completed. This can allow the user to effectively rewind and fast forward through the procedure, seeing the steps to be performed in the future and the steps that have already been performed.

[0282] In some embodiments, the user can also select and modify the proposed cutting angle, allowing the processor to use the anatomical model, such as LIFEMOD manufactured by SMITH AND NEPHEW, INC., to calculate how changes in the cutting angle can affect the geometry of the replacement knee. TM The information displayed can include the static and dynamic forces that will be exerted on the patient's ligaments and tissues in the case of a change in the cutting geometry. This can allow the surgeon to instantaneously modify the replacement knee procedure to ensure proper placement of the replacement knee components during the procedure to optimize the patient's outcome.

[0283] Figure 19 A three-dimensional model of a total replacement knee system that includes models of ligament and tendon forces is shown. If the surgeon wishes to see how a cutting decision during surgery affects the hinge geometry and the stresses exerted on the ligaments and tendons, a portion of the model can be holographically displayed to the surgeon during the surgery. The placement of the implant and the exemplary modeling of other parameters for the joint replacement procedure are described in U.S. Patent Application No. 13 / 814,531, which is hereby incorporated by reference in its entirety. The surgeon can change the parameters of the joint replacement via the AR interface to produce hypothetical results that can be displayed through the head-mounted device, such as whether a change in parameters during a total knee arthroplasty (TKA) makes the patient's gait more varus or valgus, or how to track the patella. The optimized parameters can also be modeled and displayed to the surgeon holographically for the procedure.

[0284] In some embodiments, the wearer of the HMD can selectively request the display of patient history information, including preoperative scans of the patient's tissues. In some embodiments, the display of the scan can be holographically overlaid onto the existing patient tissue observed in the scene by aligning the features of the imaging with the features found in the patient. This can be used to guide the surgeon in determining the appropriate resection area during the procedure.

[0285] In some embodiments, a video of the three-dimensional data model can be recorded, logged, and timestamped. The playback of the video can be performed after the procedure, or the video can be called up on the HMD display during the procedure to view one or more steps. This can be a valuable teaching tool for residents or surgeons who wish to see when a particular cut or step was performed. The playback can be a useful tool for creating changes to the surgical plan based on events during the procedure. The head or hand pose of the HMD operator can rewind or fast forward the virtual viewing of the video or 3-D model information.

[0286] Exemplary Use Cases

[0287] By using an HMD to add AR to the operating room, many improvements to the surgical process in various surgeries can be achieved. The following are some examples of ways in which AR can be used in various surgical procedures.

[0288] The software can utilize the HMD camera and display to determine (with the assistance of preoperative planning and the processor) the ideal starting position of the incision. Through holographic overlay (or by direct projection onto the patient's skin), lines defining the range of incision positions can be shown to the surgeon observing the patient. The exact position can take into account the specific patient anatomy and intraoperative point registration, where the user more accurately registers the patient geometry to the system.

[0289] Soft tissue dissection can utilize the built-in camera of the HMD and a model of the patient to highlight certain muscle groups, ligaments of the joint, such as the hip joint capsule or knee joint, nerves, vascular structures, or other soft tissues to assist the surgeon in reaching the hip or knee joint during dissection. The enhanced display can show the surgeon an indication of the position of these soft tissues, for example, through a holographic display of a 3D model or preoperative imaging.

[0290] During hip replacement or repair, the software can overlay lines indicating the ideal neck cut on the proximal femur based on preoperative planning. This can define the exact position to cut the bone to place the replacement femoral head prosthesis. Similar to the above example, during acetabular reaming, the head-up display can show the amount of bone that needs to be removed by overlaying different colors on the patient's acetabulum. This allows the surgeon to know when he / she is approaching the bottom surface of the acetabulum. In some embodiments, the extent of the resection area for reaming the acetabulum can be overlaid on the patient's bone. For example, color indications such as green, yellow, and red can indicate to the surgeon (based on the position of his / her tool) the depth of the reaming relative to the predetermined resection area. This provides a simple indication to avoid removing too much bone during the surgery. In some embodiments, the head-up display can overlay an image of the reamer handle (or other tool) to indicate the appropriate tilt and anteversion from the preoperative planning (or from the surgeon's input). It can also display the actual tilt / anteversion of the reamer / tool handle to allow the surgeon to correct his / her approach angle.

[0291] During the cup impact step for hip replacement, similar to reaming, the inclination and anteversion values can be displayed on the heads-up unit together with the values determined from the preoperative plan. A superimposed image of the cup impactor (or the long axis line) can be shown to assist the surgeon in positioning the implant. The HMD can also display an indication of the distance of the cup from being fully seated. For example, the measured value or color change of the superimposed cup or impactor can be used to indicate whether the cup is fully seated, or to overlay a model highlighting the final ideal position of the cup that differs from what the surgeon is currently seeing.

[0292] The HMD can highlight the screw holes that should be used to attach any hardware to the bone. When using a drill and drill guide, the head-up display can superimpose the "ideal" axis for screw insertion to assist in positioning the screw within the screw hole.

[0293] During femoral canal preparation, when the surgeon inserts the broach into the femoral canal, the HMD can superimpose an image of the broach in the appropriate orientation (corresponding to the preoperative plan). Additionally, indications can be superimposed onto the scene to provide the surgeon with information related to the final broach setting. This can be shown by changing the color around the broach or by giving the surgeon an indication of whether the broach is fully seated and what percentage or number the size is of the correct "final" component size.

[0294] Options can be provided to the surgeon performing a trial reduction to display combinations of leg length and offset increase based on the implant set being used (e.g., in the form of a chart). For example, the chart can list the combinations of leg length and offset changes for each implant combination. Alternatively, there can be an option to superimpose the proposed / changed components onto the patient's anatomy to show the surgeon what the resulting implant positioning will be if the neck offset / femoral head length is changed (e.g., from STD to high-offset neck, +0 to +4 femoral heads). The surgeon can select the appropriate implant from the first trial and perform the implant step. Multiple trial steps, i.e., the standard of care, may not be necessary.

[0295] Surface reconstruction techniques can also be improved through AR. When performing a surface reconstruction procedure, the HMD can superimpose an axis indicating the ideal position and orientation of the guide wire. In some embodiments, the software allows the surgeon to adjust this axis, and the HMD can superimpose a cross-section of the femoral neck or other views to show the surgeon how thick the bone will be at a certain location when inserting the implant (one of the most common complications from surface reconstruction surgery is inserting the component in varus, which can lead to femoral neck fracture). Giving the surgeon the ability to adjust this axis can enable optimization of the performance of the implant in vivo.

[0296] In some embodiments, this traditional femoral surface reconstruction technique can be replaced by a burring-only technique. In this exemplary technique, the surgeon prepares the proximal femur entirely by drilling. Bone maps can be overlaid on the bone to indicate how much bone remains to be removed. This can also be indicated by colors on the map. The surgeon can be provided with a variety of cutting instruments to reduce the total amount of time required to cut the bone into the desired shape.

[0297] Any suitable tracking system can be used to track surgical objects and patient anatomy in the operating room. For example, a combination of infrared and visible light cameras can be used in an array. Various light sources (such as infrared LED light sources) can illuminate the scene so that three-dimensional imaging can be performed. In some embodiments, this can include stereo, triscopic, tetracopic, etc. imaging. In addition to the camera array fixed to the cart in some embodiments, additional cameras can be placed throughout the operating room. For example, a handheld tool or a headpiece worn by an operator / surgeon can include imaging capabilities that can send images back to a central processor to correlate those images with the images acquired by the camera array. This can provide a more robust image for an environment modeled using multiple perspectives. Additionally, some imaging devices can have a suitable resolution on the scene or have a suitable perspective to pick up information stored in a QR code or a barcode. This helps to identify specific objects that are not manually registered with the system.

[0298] In some embodiments, the surgeon can manually register specific objects with the system preoperatively or during the surgery. For example, by interacting with a user interface, the surgeon can identify the starting position of a tool or a bone structure. By tracking fiducial markers associated with the tool or the bone structure, or by using other conventional image tracking methods, the processor can track the tool or the bone as it moves through the environment in a three-dimensional model.

[0299] In some embodiments, certain markers such as fiducial markers for identifying individuals, important tools, or bones in the operating room can include passive or active identifiers that can be picked up by a camera or a camera array associated with the tracking system. For example, an infrared LED can flash a pattern that conveys a unique identifier to the source of the pattern, thus providing a dynamic identification marker. Similarly, one-dimensional or two-dimensional optical codes (barcodes, QR codes, etc.) can be fixed to objects in the operating room to provide passive identification that can occur based on image analysis. If these codes are placed asymmetrically on the object, they can also be used to determine the orientation of the object by comparing the position of the identifier with the extent of the object in the image. For example, a QR code can be placed in the corner of a tool tray, thus allowing the tracking of the orientation and identification of the tray. Other tracking methods will be described throughout the text. For example, in some embodiments, surgeons and other personnel can wear augmented reality headpieces to provide additional camera angles and tracking capabilities.

[0300] In addition to optical tracking, certain features of an object can be tracked by registering physical properties of the object and associating them with an object that can be tracked, such as fiducial markers fixed to a tool or bone. For example, a surgeon can perform a manual registration process whereby a tracked tool and a tracked bone can be manipulated relative to each other. By striking the tip of the tool against the surface of the bone, a three-dimensional surface can be mapped for the bone, which is associated with the position and orientation of a reference frame relative to the fiducial markers. By optically tracking the position and orientation (pose) of the fiducial markers associated with the bone, a model of the surface can be tracked in the environment by extrapolation.

[0301] Figure 20 Examples of parallel processing platforms 2000 that can be used to implement the surgical computer 150, the surgical data server 180, or other computing systems used in accordance with the present invention are provided. The platform 2000 can be used, for example, in embodiments where machine learning and other processing-intensive operations benefit from parallel processing tasks. The platform 2000 can be implemented, for example, with NVIDIA CUDA TM or a similar parallel computing platform. The architecture includes a host computing unit (“host”) 2005 and a graphics processing unit (“device”) 2010 connected via a bus 2015 (e.g., a PCIe bus). The host 2005 includes a central processing unit or “CPU” ( Figure 20 (not shown in the figure) and a host memory 2025 accessible by the CPU. The device 2010 includes a graphics processing unit (GPU) and its associated memory 2020, which is referred to herein as device memory. The device memory 2020 can include various types of memory, each optimized for different memory uses. For example, in some embodiments, the device memory includes global memory, constant memory, and texture memory.

[0302] The parallel portions of the big data platform and / or the big simulation platform can be executed on the platform 2000 as “device kernels” or simply as “kernels”. A kernel includes parameterized code configured to perform a specific function. The parallel computing platform is configured to execute these kernels on the platform 2000 in an optimal manner based on parameters, settings, and other selections provided by the user. Additionally, in some embodiments, the parallel computing platform can include additional functionality to allow the kernels to be automatically processed with a minimum amount of input provided by the user in an optimal manner.

[0303] The processing required for each core is performed by a grid of thread blocks (described in more detail below). Using concurrent kernel execution, streams, and lightweight event synchronization, platform 2000 (or a similar architecture) can be used to parallelize portions of the machine learning-based operations performed during training or to utilize the intelligent editing processes discussed herein. For example, parallel processing platform 2000 can be used to execute multiple instances of a machine learning model in parallel.

[0304] Device 2010 includes one or more thread blocks 2030 representing the computing units of device 2010. The term thread block refers to a group of threads that can cooperate and synchronize their execution via shared memory to coordinate memory access. For example, threads 2040, 2045, and 2050 operate within thread block 2030 and access shared memory 2035. Depending on the parallel computing platform used, thread blocks can be organized in a grid structure. Computations or a sequence of computations can then be mapped onto this grid. For example, in an embodiment utilizing CUDA, computations can be mapped onto a one-dimensional, two-dimensional, or three-dimensional grid. Each grid contains multiple thread blocks, and each thread block contains multiple threads. For example, thread block 2030 can be organized in a two-dimensional grid structure of m + 1 rows and n + 1 columns. Generally, threads in different thread blocks of the same grid cannot communicate or synchronize with each other. However, thread blocks within the same grid can run simultaneously on the same multiprocessor within the GPU. The number of threads in each thread block may be limited by hardware or software constraints.

[0305] Continuing reference Figure 20 Registers 2055, 2060, and 2065 represent fast memory available to thread block 2030. Each register can only be accessed by a single thread. Thus, for example, register 2055 can only be accessed by thread 2040. In contrast, shared memory is allocated per thread block, so all threads within the block can access the same shared memory. Thus, shared memory 2035 is designed to be accessed in parallel by each of threads 2040, 2045, and 2050 within thread block 2030. A thread can access data in shared memory 2035 that has been loaded by other threads within the same thread block (e.g., thread block 2030) from device memory 2020. Device memory 2020 is accessed by all blocks of the grid and can be implemented using, for example, dynamic random access memory (DRAM).

[0306] Each thread can have one or more memory access levels. For example, in platform 2000, each thread can have three memory access levels. First, each thread 2040, 2045, 2050 can read from and write to its corresponding register 2055, 2060, and 2065. Registers provide the fastest memory access to the thread because there are no synchronization issues and registers are typically located near the multiprocessor executing the thread. Second, each thread 2040, 2045, 2050 in thread block 2030 can read from and write data to the shared memory 2035 corresponding to that block 2030. Typically, because access between all threads in the thread block needs to be synchronized, the time required for a thread to access shared memory exceeds the time required to access registers. However, like registers in the thread block, shared memory is typically located near the multiprocessor executing the thread. The third level of memory access allows all threads on device 2010 to read from and / or write to device memory 2020. Device memory requires the longest access time because the access must be synchronized between the thread blocks operating on the device.

[0307] Embodiments of the present disclosure can be implemented with any combination of hardware and software. For example, in addition to the parallel processing architecture presented in Figure 20 , a standard computing platform (e.g., a server, a desktop computer, etc.) can be specially configured to execute the techniques discussed herein. Additionally, embodiments of the present disclosure can be included in a manufactured article (e.g., one or more computer program products) having, for example, a computer-readable non-transitory medium. The medium may contain computer-readable program code for providing and facilitating the mechanisms of embodiments of the present disclosure. The manufactured article can be included as part of a computer system or sold separately.

[0308] Apply statistical models to optimize preoperative or intraoperative planning based on patient activity

[0309] There is a need for a simple and processor - efficient planning tool for surgical staff to perform patient - specific pre - operative or intra - operative planning. Due to the surgical volume and time constraints on engineers and surgeons, the pre - operative planning phase for joint replacement surgery should be computationally and labor - efficient, while the intra - operative planning phase imposes more computational restrictions on any simulated data because there is no time to wait for simulations in the operating room. With the rise of inexpensive tablets or mobile devices (generally, lower - powered computing systems such as cart workstations in the operating room), there is an opportunity to provide low - processor - overhead applications that can assist surgical staff in collecting data and planning surgical procedures, which utilize more powerful systems connected over a network that maintains a data store of simulated or real - world data from past patient cases. These backend systems can maintain, manipulate, create, and mine large amounts of data, thus allowing lower - powered devices in the operating room to utilize this treasure trove of information. The ideal planning application should assist surgical staff in collecting data and planning surgical procedures and accessing a data store of simulated or real - world data from past patient cases.

[0310] For example, in some embodiments, an operating room device or mobile device having an intuitive touch - screen interface, wireless network capabilities, and a camera makes them particularly suitable for helping to create or modify surgical plans that can be used with CASS during the pre - operative phase or during surgery. Its interface can be useful in collecting and interacting with new information (e.g., imaging or force characteristics of a joint captured during surgery) that can be used to improve the surgical plan. Devices such as tablets, laptops, or cart - based computers typically lack a powerful processor, which can be improved by leveraging a server or database with past simulations or clinical results, thus providing the benefits of patient - specific simulations to devices in the operating room without having to run simulations locally or on - demand. Some embodiments of the surgical planning tool utilize a network interface to allow a remote database or server processor to offload some data storage or processing from the operating room device or mobile device. By looking up or learning from past similar simulations, these databases provide an opportunity to efficiently and immediately estimate simulation results for given patient data, and these databases are particularly suitable for low - processor - overhead applications.

[0311] Some embodiments recognize that the patient goals for surgery are often unique and personalized. One patient may simply want to return to a pain-free, more sedentary life, while another patient may desire to return to an active life of golfing, cycling, or running. By leveraging big data repositories of similar patient simulations or past outcomes, preoperative data can help surgeons optimize the surgical plan towards these activity-specific goals. The characteristics of each of various common activities (e.g., walking, stair climbing, squatting, bending, golfing, etc.) can be motion curves that account for the actual motion the joint will experience during an exemplary repetitive motion associated with that activity, as well as the expected stresses on the implant and soft tissues. Repetitive motion curve simulations can be performed for each activity for a variety of patient joint geometries to populate a database of simulation results. An application on the CASS or the user computing device can then request selection of the activity for which the patient's surgical plan is to be optimized. The surgical plan can then focus on the simulation results associated with those motion curves, while ignoring or weighting less severe simulation results associated with unselected activities.

[0312] An exemplary embodiment of a surgical application or CASS that utilizes past simulation results is a knee replacement planning tool. It should be noted that these techniques can also be applied to other resection surgeries, such as partial knee replacement, hip replacement, shoulder replacement, ankle replacement, spinal resection, etc., or any procedure where the patient geometry affects the performance of the prosthetic implant. In an example of a knee replacement planning tool, the planning tool can assist the surgeon in selecting the appropriate size, position, and orientation of the implant, as well as the type of implant to use when implanting the prosthesis, to maximize mobility and minimize the likelihood of failure due to premature wear, impingement, dislocation, or unnecessary loading of the implant or ligaments during the expected activities.

[0313] In the context of THA, current guidelines for the abduction and anteversion angles of acetabular cup placement using the thumb rule (e.g., guidelines using the Lewinnek "safe zone") may not be sufficient to minimize the risk of hip dislocation during patient recovery. For example, recent studies have shown that more than 50% of postoperative dislocations occur in implants placed within the Lewinnek "safe zone". The studies also show that the spinal pelvic mobility of the patient can directly affect the appropriate acetabular cup placement, which may not be considered in traditional guidelines. Therefore, appropriate implant position and orientation can benefit from a planning tool that utilizes simulation results to account for patient-specific risk factors.

[0314] One embodiment of the surgical planning tool can be an application running on a desktop computer, server, laptop, tablet, mobile phone, or cart-based workstation. The exemplary application mainly considers geometric shape data within X-rays (or other medical image data, such as CT, MRI, ultrasound, etc.) that can minimize the effects of X-ray / image distortion. In some embodiments, the image processing software can estimate the positions and distances of prominent points within the image, or a touchscreen (or other) interface can allow the user to easily manipulate these points and measurements by dragging and measuring on the image until the surgeon is satisfied with the accuracy of their placement relative to the patient's anatomy. For example, in PKA / TKA, the anatomical axes of the femur and tibia can be extracted from the image to determine the varus / valgus angle; the center points and radii of the distal and posterior portions of the medial and lateral condyles can be extracted; the medial and lateral gaps between the tibial plateau and the corresponding condyles can be measured from images at various flexion degrees. The shape of the patellar groove can be determined from anterior / posterior images at various flexion degrees, while the depth can be determined from lateral images or MRI. The surgeon can also estimate the tension or laxity of the ligaments by applying forces to the knee at a predetermined flexion degree to supplement the image data. Some embodiments use a combination of estimates of the positions of these points that are completed by automated image processing (guided by searching for prominent features learned from a past human-selected training set) and refined by the user using a touchscreen or computer (which can be learned from past interactions with the surgeon user when the surgeon user places these points on the image). In some embodiments, images of extension and knee flexion at two or more positions (e.g., lateral and AP X-rays or MRI) are considered. In some embodiments, X-rays (or other images) of the standing patient and their sitting X-rays (or other images, such as MRI) are considered. This can provide the system with an estimate of the pelvic tilt change between standing and sitting, which can be used to estimate patient mobility issues. In other embodiments, X-rays of the patient in a challenging position, such as a flexed sitting position or hyperextension while standing, can be considered. Some embodiments also utilize a motion capture system to provide information about the patient's existing joint mobility, such as pelvic mobility limitations.

[0315] Once the image is marked to identify geometric features and relationships of the patient's anatomy (either automatically through image analysis or manually through touchscreen / UI manipulation), the simulation model can also incorporate any additional patient conditions, such as spinal or hip joint movement issues. (For example, in the case of THA, the conditions can include a specific range of motion in the sagittal plane and a measure of stiffness. This is important for positioning the implant device within the patient to reduce the incidence of edge loading and malpositioning for the patient's expected activity level.) The planning application can then perform a lookup of the results of a previously performed anatomical simulation or perform calculations based on transfer functions extracted from multiple past anatomical simulation results (of various patient geometries and properties) to generate curves for ligament impingement risk, ligament stress, and condylar compartment clearance, and patellar tracking (for PKA / TKA) or the center of pressure and range of motion between the femoral head and acetabular cup (THA) during repetitive motion curves for each of various selected activities over the entire range of motion. In some embodiments, the various activities to be considered can be guided by the patient's lifestyle and activity level and the overall results of the simulation of each activity. Simple user interface options can then be presented to the user to vary the position and orientation of the distal and posterior incisions, and the patellar attachment point / fill (for PKA / TKA) or the abduction and anteversion angles (THA) to observe how changing the position and orientation of the implant affects these characteristics of the resulting joint. Different implant options that affect the performance of the resulting joint can also be presented to the user, such as target gap size, artificial cartilage thickness, implant size and model, tibial and femoral implant depth, laxity, etc. For THA, these can include, but are not limited to: femoral head diameter, liner type (neutral, capped, anteverted, ceramic, constrained, dual mobility, or surface replacement), standard / high offset, femoral head offset, implant style / series (i.e., flat taper wedge versus fit and fill), implant depth in bone, implant size, etc. Since the results can be quickly generated algorithmically through table lookup or using transfer functions from past simulation results, the user obtains seemingly instantaneous feedback in the interface regarding the effects of his or her choices in implant type and position / orientation. This is particularly useful in the operating room when new information about the patient is collected (such as joint force loading, soft tissue laxity, etc.) or when the surgeon requests a change to an existing preoperative plan, and provides quick feedback on how the changes or new information affect the expected outcome and joint performance.

[0316] In some embodiments, a button is also presented to the user that allows the system to automatically suggest optimized distal and posterior cutting planes, tibial implant depth, patellar alignment and filling, and implant type and size to minimize deviations in the tibiofemoral joint space, as well as ligament tension and laxity throughout the range of motion, while also minimizing the amount of ligament release required during surgery to achieve an acceptable balance (for TKA / PKA). The results of these selections or optimizations can then be easily added to the surgical plan in CASS or shared among colleagues as at-a-glance information on range of motion and center of pressure. Optimization can be performed using any suitable algorithm, such as by gradually adjusting angles or implant types and re-running database searches or transfer function calculations until a local or global maximum is reached. In some embodiments, optimizations are performed a priori for every possible combination of patient geometries (or within every combination within a reasonable range), searching for distal and posterior cutting alignments or implant types for that combination to minimize deviations in compartment space or ligament strain / laxity (PKA / TKA), or for valgus and abduction angles or implant types for that combination to minimize the risk of edge loading or dislocation (for THA).

[0317] In some embodiments, all reasonable combinations or a subset of all combinations of patient geometries are simulated to create a simulated database for subsequent real-world patient implant planning. Once a sufficient number of simulations have been performed, the algorithm can search for transfer functions that approximate the results of these simulations, allowing interpolation of simulation maps for different combinations of implant type and position / orientation without actively simulating each combination. Since performing each anatomical simulation can take several minutes, determining the transfer function from all or a subset of past simulations can speed up the process of determining a plan for a specific patient. In some embodiments, a subset of implant orientations can be simulated and the implant angles optimized to populate the resulting database. Then, a suitable machine learning algorithm can be applied to create a learned outcome model that can be applied to estimate the results for additional combinations of parameters. This allows for the dynamic generation of good estimates of flexion and extension gaps and ligament tension (PKA / TKA) or edge loading or dislocation stress (THA) for various activities and implant orientations through the outcome model without fully simulating motion using the anatomical model (which is processor-intensive each time) for every combination. Exemplary algorithms that can be applied individually or in combination to develop and train the learned outcome model or optimize angle selection include, for example, linear or logistic regression, neural networks, decision trees, random forests, K-means clustering, K-nearest neighbors, or suitable open-source deep learning libraries. The transfer function can be identified through regression analysis, neural network creation, or any suitable AI / analytics algorithm that can estimate the parameters of the transfer function.

[0318] By running many different simulations for each activity with different implant orientations and types, and simulating the loads, stresses, and range of motion in the joint, a database or model created by analyzing these results can act as a statistical model that describes the output of a multi-body system for a given combination of outputs for each activity. This can be achieved by simulating all possible inputs for a model with only a few degrees of freedom, as a transfer function for fitting a multi-dimensional expression that closely estimates the response of the system after hundreds or thousands of simulations that explore a subset of possible parameter combinations, or by applying conventional machine learning software systems to create an AI-driven model that approximates the system response of a multi-body model of the joint based on a large number of simulations. In different embodiments, any of these statistical models can be used to model joint behavior in a manner that approaches the accuracy of on-demand simulations of a patient-specific model for a given activity, without the impracticality of the performance of on-demand simulations of a detailed multi-body model of the patient's joint.

[0319] These same concepts can be applied to selecting the position and orientation of other implantable devices, such as total or partial knee replacements, allowing the user to easily mark X-ray / image data to manipulate the implant position and orientation of the hardware, and quickly obtain at-a-glance feedback on how those positions and orientations affect implant-related risk factors based on the patient's activity level, and easily create or modify a surgical plan. Each anatomical simulation uses a specific patient anatomical geometry with a specific implant position and orientation that performs a given repetitive motion associated with a given activity, such as climbing stairs. Each simulation creates a data point in the database that can be accessed when attempting to later optimize the implant plan for a similar patient. By repeating this simulation for hundreds to thousands of other geometric combinations and activities, the database can be used for a wide combination of native patient geometries, implant postures, and alternative activities.

[0320] Figure 21FIG. 2100 is a system diagram of a system 2100 that is an exemplary embodiment of a surgical planning tool that can be used in a stand-alone system or as part of a CASS to generate or modify a surgical plan. In some embodiments, a low-computation-overhead client-server method using pre-populated data storage of complex simulation results allows surgeons to perform immediate optimization and adjustment, thus allowing selection of various postoperative patient activities and changes to implant factors. In the case of hip replacement, this can include the acetabular cup anteversion and abduction angles, bearing type (polyethylene, hard material, dual mobility, constrained, surface replacement), femoral head size and offset, femoral stem offset (standard, high, valgus), femoral stem style, femoral stem depth and orientation (form). In the case of knee replacement, the implant factors include anterior-posterior / lateral-medial placement or cut angles, distal depth, specific orientation of the distal and posterior cuts of the condyles to ensure that the compartmental clearances are consistent throughout the range of motion that occurs during the selected activity and that the ligaments are neither overstrained nor too lax, and the geometry of the patella relative to femoral features (PKA / TKA). The user device 2102 can be a personal computing device such as a mobile device, tablet, surgical workstation / cart, or laptop / desktop computer. For the purposes of this example, the user device 2102 can be described as a tablet device. In general, it is assumed that the user device 2102 has low processing power relative to a computer that could otherwise be used to run simulations of the patient's anatomy. The system 2100 is particularly suited to a system in which it is computationally impractical to perform immediate simulations of a given patient geometry and implant orientation and position for each selected activity. An application 2103 that guides the surgical planning process resides in the memory of the user device 2102. The application 2103 includes a user interface (UI) 2104 and a data and input storage area 2106 for user data. The application 2103 requests certain inputs from the user regarding a given patient. In some embodiments, the application 2103 can communicate with a medical record server that includes patient records to provide some of this information.

[0321] Exemplary information requested by application 2103 via UI 2104 and stored in database 2106 includes pictures of one or more images captured using the camera of user device 2102 of a patient's X-ray / medical image (or providing a device, if the application is used during surgery, from which the user can upload previously captured X-ray / CT / MRI images from medical records or currently captured images during surgery), and inputting important information about the patient, such as height, weight, age, physical development, and activity level. In some embodiments, the X-ray is the primary medical image, but in some embodiments, MRI, CT, or ultrasound images may be used. The UI also allows the user to select information about the prosthesis to be implanted and select various activities the patient wishes to participate in postoperatively (e.g., running, playing golf, climbing stairs, etc.). In some embodiments, the ability of the user to input this information using the touchscreen of the UI and the device camera simplifies the application such that it does not need to communicate with electronic medical records, which may pose additional regulatory issues and require additional security and software modules, etc.

[0322] User device 2102 communicates with a server or cloud service 2110 via the Internet 2108. Server 2110 provides backend processing and resources for user device 2102, allowing application 2103 to be a lightweight application while user device 2102 can be almost any available user device, such as a tablet or an existing surgical workstation. Server 2110 maintains a model database 2112 that contains simulation results of various implant patient geometries that perform a predefined motion curve associated with each patient activity. Database 2112 can be pre-populated with additional simulations or continuously updated by a suitable processor, such as a multi-core processor of server 2110. In some embodiments, an additional computer or cluster (not shown) populates this model database, allowing server 2110 to process incoming requests from multiple user devices.

[0323] In addition to the simulation results, model database 2112 may also include guidelines to assist the user in understanding the appropriate range for selecting an appropriate implant posture. In the case of TKA / PKA, this may include selecting an appropriate filling for the patella and other implant features. For THA, this may include various spinal pelvic motions or sacral tilt angles to assist the user in selecting an appropriate anteversion and abduction angle, and other implant features. Model database 2112 may also include optimal recommended implant information, such as the optimal implant posture, optimal implant size or type, or the optimal range that will work with a given patient's anatomical geometry performing a given activity for a given patient geometry and activity. In some embodiments, the simulation results also include simulations of patients with a given geometry and additional obstacles, such as orthopedic or neurological conditions that were not fully captured when viewing sitting and standing images.

[0324] Once the user has uploaded X-ray images (or other medical images) and manipulated those images to identify certain points and angles within those images (or the image processing software has automatically identified or estimated those angles from the images), patient characteristics, desired patient activities, and optional starting point implant characteristics (e.g., starting pose, implant size, bearing type, etc.), the server 2110 can consult the model database 2112 to find the entry that best matches the user input and the medical imaging geometry. In some embodiments, a large number of independently adjustable variables can make a complete database of all possible combinations impractical. In these embodiments, the database can include a subset of the possible combinations of patient characteristics, X-ray / imaging geometry, and implant characteristics, and the server 2110 can interpolate specific results from the surrounding entries that most closely match the particular user selection. In some embodiments, the closest match can be provided as the result without interpolation.

[0325] In some embodiments, once various simulations have been performed for various combinations of patient characteristics and geometries, the processor can optimize the transfer function to closely match the results of the simulations using any of the conventional means described above. By fitting the transfer function to the results of many simulations, the transfer function can be provided to the server 2110 for quickly calculating the results of various activities for a given user input of X-ray and implant characteristics and patient characteristics, regardless of whether that particular combination has been previously simulated. This can enable the server 2110 to quickly process requests from multiple users without having to run potentially tens of thousands or more combinations in simulations to populate the model database 2112. The model database 2112 can be used to store the transfer function, allowing the server 2110 to calculate results rather than search the model database 2112. In some embodiments, learning algorithms are used to train patient response models for each activity for a given geometry to allow for quick estimation of responses and determination of optimized implant positions and orientations at the server, similar to the use of the transfer function.

[0326] Exemplary simulations can utilize a variety of simulation tools, including LIFEMOD available from LIFEMODELER INC., a subsidiary of SMITH AND NEPHEW, INC. of Memphis, Tennessee TMAnatomical modeling software. Exemplary simulations are explained in U.S. Patent Application No. 12 / 234,444, commonly owned by Otto et al., which is incorporated herein by reference. The anatomical modeling software can utilize a multi-body physics model of the human anatomy including bone and soft tissue elements that accurately simulate a human joint of a given geometry. Specific physics-based biomechanical models can be customized for patient-specific information such as height, weight, age, gender, bone segment lengths, range of motion and stiffness curves for each joint, balance, posture, prior surgeries, lifestyle expectations, etc. Test designs are created to simulate various anatomical geometries and implant angles that perform various predetermined motions, each predetermined motion being related to a different alternative activity.

[0327] Figure 22 Various theoretical angles that can be extracted from X-ray imaging using a hip joint geometry model are shown. Model 2120 is a model of the geometry of the hip joint in the standing position, while model 2122 is a model of the hip joint in the sitting position. Various angles that can be extracted from the geometry of these models include the dynamic sacral tilt or inclination (ST, 45° standing, 20° sitting), static pelvic incidence (PI, 55° standing and sitting), dynamic pelvis-femur angle (PFA, 180° standing, 125° sitting), dynamic anteversion angle (AI, 35° standing, 60° sitting). Figure 22 Theoretical angles that can be extracted and used but are not shown in may include a combined sagittal index, which is defined as the sum of the anteversion angle and the pelvis-femur angle. The static sacral acetabular angle (SAA) at the intersection of the lines that produce the ST and AI angles in this model is also not shown, as explained below.

[0328] Figure 23A is an X-ray of various points, lines, and angles that can be extracted from an X-ray image 2130 of the hip of a standing person. The user can manipulate points 2151 to 2156 using a user interface / touch screen, or these points can be automatically generated by an image processing algorithm that can be improved through machine learning of multiple iterations of user input when the application is developed and used by real-world users. Note that in some embodiments, other image types can be used, including MRI, CT, or ultrasound image data. Once points 2151 to 2156 are placed in the image, lines 2132 to 2144 can be automatically placed on the image, allowing for Figure 22 the various angles described to be automatically calculated by the processor of the user device. The points include between the upper / rear S1 endplate 2151, the lower / front S1 endplate 2152, the center point 2153 between the hip joint centers, the posterior acetabulum 2154, the medial acetabulum 2155, and the femoral axis point 2156.

[0329] The lines include a horizontal line 2132 originating from point 2151, a line 2136 (which runs between the upper / rear S1 endplate point 2151 and the lower / front S1 endplate point 2152), a line 2134 (which automatically generates a line 2136 perpendicular to the bisector of points 2152 and 2151), a line 2138 (defined by the positions of points 2153 and 2155), a line 2140 (defined by the intersection of lines 2134 and 2136 with point 2155), and a line 2142 (defined by points 2155 and 2156), and a horizontal line 2144 (from point 2154). Once points 2151 to 2156 are added or extracted from the image, these lines can be automatically generated. This allows the processor to determine the ST (between lines 2132 and 2136), PI (between lines 2134 and 2140), SAA (between lines 2136 and 2138), PFA (between lines 2140 and 2142), and AI (between lines 2138 and 2144) angles.

[0330] As Figure 23B shown, each of points 2151 to 2156 (discussed in Figure 23A ) is manipulated and extracted from an X-ray image 2158 of a patient's hip joint in a sitting position. From these points, lines 2132 - 2144 can be extracted by the processor of the user device / tablet. Once these lines are extracted, various angles can be calculated. If the static PI and SAA angles are inconsistent between sitting and standing, an average value can be used. In some environments, the inconsistency between these angles can result in errors or user requests for additional input. In some embodiments, the inconsistency between these angles can be used to scale or adjust other angles or refine the placement of points in the sitting and standing X-ray images. The server can use these angles to find appropriate simulation results as these angles define the geometry of the patient's anatomy.

[0331] Figure 24is a flowchart of an exemplary process 2160 by which a user manipulates landmarks in an X-ray (or other medical image) to determine feature positions, sizes, angles, and spacings that will be used for patient geometry to determine joint performance for various patient activities using pre-populated statistical models of joint behavior for each activity. This can typically be done during the pre-operative phase, but can be updated in response to images captured during surgery or based on new data obtained during surgery, such as using a probe tracked by CASS to capture bone features and their relative positions. For example, for knee replacement surgery, the performance can include measures of changes in the condyles or compartment spaces, tension or laxity of ligaments, and the degree of patellar tracking in the patellar groove of the femur. At step 2162, an application on the user device loads a patient anatomical image into memory. This can be achieved by capturing an image from a film or screen using a camera, by connecting to a stored image in a local data memory or medical record system, or by instantaneously creating an image using a local imaging device (e.g., an X-ray taken during surgery). In some embodiments, the image can be an X-ray, ultrasound, CT, MRI, or other medical image. The file format of this image can be any suitable file format, such as PDF, JPEG, BMP, TIF, raw image, etc. Once the user device loads this image, at step 2164, the image is displayed and optionally analyzed using image recognition software. This optional analysis step can utilize any common image processing software that identifies prominent features in the image that can be used to identify landmarks. For example, the image processing software can be trained to look for certain geographical features, geometric markers, or can be trained through any suitable AI process to identify the likely positions of landmarks for determining patient geometry. At step 2166, the results of this analysis step are overlaid on the displayed image, placing marker points at the pixel positions (e.g., Figure 23A-23B the positions shown in) that the analysis software deems most likely to be the positions of these marker points. It should be understood that the process described in steps 2162 to 2172 need not be limited to medical images and can be done automatically without a person interacting with the processor. During operation, additional geometric and position data can be obtained in CASS by various means, including imaging or by a surgeon using a robotic arm or a pointing probe to draw the surface of relevant features and registering their positions with a surgical tracking system. This provides an additional level of improvement in the geometric extraction of relevant features that can be used to optimize the surgical plan beyond what can be achieved by preoperative imaging alone.

[0332] In some embodiments, image recognition software is useful and can be complementary to / partially or fully replace the analysis by an experienced surgeon or technician. In some embodiments, at step 2168, the user of the software has the opportunity to manipulate the precise placement of anatomical features such as the femoral axis, sacral geometry, femoral head and neck features, acetabular cup geometry, condyle center and radius, patellar groove, patellar dome, and tendon and ligament contact points. The exact method by which the user manipulates these points will depend on the interface of the computing device. For example, on a tablet computing device with a touch screen, the user can use his or her finger to manipulate and drag the precise placement of these points (which have been automatically placed at step 2166). In some embodiments, in the case where suitable image processing software has not been trained, the user can create these points from scratch at step 2168 by tapping on the location of these points on the image after being prompted by the system. In response to moving each point, at step 2170, the display creates and updates a model of the anatomical geometry (e.g., the relationship of the condyle features to the tibial and patellar features, and information suitable for determining the strain on the ligaments and tendons, such as the quadriceps angle). (In some embodiments, this step can be put on hold temporarily until the user indicates that all points have been moved and the user is satisfied with their placement.) The process continues at step 2172, where the software and the user determine whether all points and distances have been correctly manipulated and placed, and the manipulation is repeated for each point and distance.

[0333] Once all the points have been moved and the user is satisfied, at step 2174, the user is given the option to select and load additional images. For example, when the planned surgical procedure is a knee replacement, suitable images can include the anterior-posterior and medial-lateral images of the patient's knee in the flexed position, as well as images in the extended position or any additional poses required to determine the geometry of the relevant features. For a hip replacement, suitable images can include at least the lateral image of the patient in the standing position and the image in the sitting position. In a knee replacement, suitable images can include the lateral or anterior view of the knee fully extended and bent at a certain predetermined angle, such as 90°. Additional images can include the patient in the sitting position with the knee flexed or in the extended position while standing. If additional images such as the posterior / anterior view or medial / lateral view of the joint or images from different positions are available, the images can be loaded again at step 2162.

[0334] Once all the images are loaded and analyzed, the placement of the landmark points can be manipulated by the user (or software image processing AI). The relevant angles and distances can be calculated from the positioning of these points, and the method can proceed to step 2176. At step 2176, the user can select the appropriate activities that the patient wishes to enjoy postoperatively. These activities can be adjusted, for example, by the patient's relative activity level, age, other debilitating conditions of the patient, or mobility issues. Exemplary activities can include standing, walking, climbing stairs or descending stairs, cycling, golfing, low-impact yoga, squatting, sitting cross-legged, gardening / kneeling, etc. It should be understood that some of these activities may be relevant to most or all patients, while some activities may be relevant only to a subset of younger or specific patients. For example, perhaps only younger or more active patients can select the running activity.

[0335] At step 2178, the selection of the activity and the patient-specific geometry calculated from manipulating points on the patient's image can be loaded into the server (or set into the memory accessible by a processor that is adapted to) apply a model created from existing simulation data to calculate the appropriate results related to the surgical procedure. For example, for hip replacement, the results can include a map of the center of pressure within the acetabular cup and the range of motion of the joint, which is expressed in a polar plot of the femoral head relative to the acetabular cup. For knee replacement, the results can include the range of motion between the tibial plateau and the femoral condyles, a map of the contact points, and the stress experienced by the relevant ligaments, a plot of the lateral and medial condylar clearances and ligament tensions for a range of flexion angles or discrete angles, or a graphic highlighting the patellar tendon or ligament within the range of motion, or the way the patellar dome sits within the patellar groove during that range. For this first pass of step 2178, the default implementation characteristics of the prosthetic implant (e.g., a varus angle of 3 degrees and a condylar clearance of 9 - 10 mm) can be used. In some embodiments, prior to performing step 2178, the user also selects the starting implant position / orientation for a given implant for this initial calculation.

[0336] At step 2180, the processor that applies the statistical model to the anatomical information and activity selection sends this result to the user device in the implant variables for the results for display. In embodiments using a client / server model, the step can include sending the results to the user device over a network. In some embodiments, the processor that performs the analysis using the statistical model can be the same processor as the user device. In these embodiments, the statistical model is typically optimized enough to be stored and analyzed locally. The exact display of these results can be in any suitable graphical manner, such as those interfaces described in subsequent figures.

[0337] At step 2182, the user has the opportunity to choose to manually manipulate the implant characteristics (position, orientation and in some embodiments size and type) or request that the processor automatically optimize the implant geometry. In some cases, the user may choose to have the processor automatically optimize and then manually refine the exact resection for implantation to suit the surgeon's preference or give greater emphasis to certain activities over others. For example, the surgeon may attempt to optimize the implant such that the patient can resume playing golf but still emphasize the patient's ability to perform daily activities such as climbing stairs and sitting comfortably. In some embodiments, a weighting of activities may be provided in the patient profile and this weighting may be used by the optimization algorithm to perform the automation.

[0338] At step 2186, if the user has requested that the processor optimize the implant angle, the optimization algorithm is run by the processor (either at the server or on the user device in a non-client / server embodiment) based on a statistical model. This optimization algorithm may include any search algorithm that searches through a statistical database to find a local or global maximum implantation angle that provides the best performance or analyzes the extracted transfer function to find a maximum or minimum result. The criteria for this search may vary depending on the type of implant being implanted. For example, in a knee replacement, the algorithm may aim for a reasonable range of condylar chamber clearances for the medial and lateral compartments, identify the most consistent clearance through the flexion range according to guidelines while maintaining ligament tension within a suitable range, and identify the femoral implant orientation and patellar filling and attachment constraints that best allow the patellar dome to track in the patellar groove with minimal soft tissue strain. For a hip replacement, the anteversion and abduction angles that provide the lowest risk of edge loading or dislocation in terms of pressure center distribution and range of motion distribution can be identified by minimizing the amount of pressure near the acetabular cup edge and minimizing the amount of range of motion that risks impinging on the cup edge for each selected activity.

[0339] In some embodiments, this optimization procedure may include iteratively changing the resection placement and orientation until a suitable or optimal result is achieved. Once the processor that processes the statistical model has completed this optimization, the result can be received and displayed by the user device at step 2180. If the user wishes to manually change the implant angle, at step 2184, the user can update the selected implant angle via the user interface. Then, at step 2178, the selection can be uploaded to the processor that processes the statistical model. In some embodiments, the user may also change the implant bearing type or liner option. For example, the user can switch from a 0-degree neutral XLPE liner to a 20-degree anteversion XLPE liner, thus allowing the user to experiment with range of motion and pressure center. Similarly, the user can switch from a conventional bearing surface (XLPE) to a dual-mobility system that provides an increased head size and jump distance to allow for greater stability in the joint.

[0340] Once the user is satisfied with the results of the manipulation or automatic optimization, at step 2188, the user device updates the surgical plan presented to the user or within the CASS, allowing the robotic surgical system to prepare for implementing the selected implant location and orientation based on method 2160. In embodiments using a cutting guide, step 2188 may include sending the surgical plan (including a specific map of the relevant patient bone surfaces and the specific locations of the resection incisions relative to those surfaces) to a manufacturing system that fabricates patient-specific cutting guides, and requesting 3-D printing of these cutting guides prior to surgery. In some embodiments, in cases where the robotic arm will position or hold a non-patient-specific cutting guide, the surgical plan may include requesting that an appropriate cutting guide be provided to the surgeon for the surgery, and programming the robotic arm to place the cutting guide at a specific predetermined location and orientation.

[0341] Figure 25 Exemplary Table 2200, which may be presented to the user to assist in understanding how the preoperative anatomy compares to the expected ranges for a healthy patient. This can help guide the surgeon in understanding the appropriate surgical plan to use with the CAS system. Table 2200 shows the results of calculating various angles in the seated and standing images using image analysis or manipulation of points as described above for planning a hip replacement. Column 2202 lists the various angles calculated for each image, including ST, PI, PFA, and AI. Column 2204 shows the results of analyzing the angles based on points identified in the standing image. Column 2206 is an analysis performed by the server or user device based on a list of acceptable ranges for these angles, such as Figure 21 the guidelines shown in Table 2114 in. In this example, the sacral slope has been identified as normal, while the pelvic incidence, pelvic femoral angle, and anti-tilt angle have been identified as abnormal, outside the expected normal range for a healthy patient. Column 2208 shows the angles that have been calculated based on points identified in the seated X-ray image. Column 2210, like column 2206, is an identification of whether each angle is within the acceptable range for a healthy patient. Section 2212 is a comparison of the various angles between the standing and seated postures (2204 and 2208), including differences in sacral slope and anti-tilt. These increments are then used to determine whether the spinal-pelvic mobility falls within the normal range and the position of the spinal-pelvic balance. This information can be used to guide the selection of the implant orientation to improve mobility and balance the patient's hip geometry. This information can be considered by the surgical user or provided on demand as a teaching tool. As discussed, the standing and seated angles are used by the patient model database in combination with the anteversion and abduction (or other implant characteristics) of the acetabular cup to determine the range of motion and center of pressure results from the statistical database of simulation results.

[0342] Figure 26AThis is the exemplary user interface 2220, which is used to select a single activity related to a given patient and display results from a statistical database based on previous user inputs regarding the patient, including X-ray marker mapping for the patient. The user can select from various individual activities 2222 and can manipulate the abduction angle 2224 and the anteversion angle 2226 for the implant (note that these exemplary angles may differ from the real-world values for a given implant). The center of pressure heat map 2228 is an aggregation of all selections of the individual activity 2222 based on the patient geometry from X-ray imaging and the manipulation of the abduction angle 2224 and the anteversion angle 2226. (Options for changing the liner and bearing characteristics are not shown and may be available in some embodiments.) Based on a statistical model, manipulating the abduction angle 2224 or the anteversion angle 2226 will move the heat map relative to a circle that represents the range of the acetabular cup. Placing the pressure too close to any edge of the acetabular cup may result in edge loading, which may cause premature wear or failure of the hip implant or, in some cases, dislocation. Similarly, the range of motion map 2230 is an aggregation of the patient's range of motion for each individual activity and how that motion translates into the interaction between the femoral head and the acetabular cup. The breadth of this range of motion should be limited to the circle representing the breadth of the acetabular cup. Any range of motion beyond these breadths will create a high risk of hip dislocation or impingement, which hinders the motion required for that activity.

[0343] In some embodiments, an optimized button 2232 is presented to the user, allowing the user device or server to automatically change the abduction and anteversion angles such that it optimizes the placement of the center of pressure and the range of motion within the circle. (In some embodiments, this automated recommendation may include implant selection, such as implant size, bearing type, liner type, femoral head characteristics, etc.) This can be done iteratively using any suitable algorithm to adjust these angles to improve the pressure in the central region and the range of motion that falls within the range of the acetabular cup circle. The heuristics used may include maximizing the distance between the pressure point for maximizing the central pressure and the edge of the acetabular cup, and maximizing the distance between the breadth of the range of motion within the edge of the acetabular cup circle, maximizing the average distance, maximizing the minimum distance, etc.

[0344] In some environments, a graphic 2234 is presented to the user, which changes as the abduction and anteversion angles are manipulated to provide the user with visual feedback on how these angles affect the acetabular cup placement. Buttons are not shown in the interface 2220 that, in some embodiments, exist to finalize and save the abduction and anteversion angles and load these angles into the surgical plan of the CAS system.

[0345] Figure 26BShows how the user interface changes when only a single active subgroup 2236 is selected. Instead of a large heat map or a large range of motion, the center of pressure map 2238 shows a more limited center of pressure attributable to only these activities. Similarly, the range of motion map 2240 shows the range of motion used by only these activities. Comparing these figures with FIGS. 2228 and 2230 ( Figure 26A ) illustrates that a wider range of activities results in a wider heat map of the center of pressure and a denser range of motion. For some patients whose mobility is restricted by lifestyle or other factors, only certain activities may be important, thus allowing the user to more easily optimize implant selection and acetabular placement (in hip replacement applications) to ensure that the user can successfully perform these activities. For some patients, other constraints given to place the acetabular cup in a way that all activities are possible may not be feasible. This may be due to abnormal hip joint geometry determined from (X-ray) images or due to external mobility problems such as spinal fixation. In some embodiments, the surgeon or user can recommend to their patients which activities or positions may pose a risk of impingement, dislocation, or excessive wear to their artificial hip joint.

[0346] Figure 26C Is a user interface showing an embodiment, whereby hovering or otherwise temporarily selecting a single activity 2242 in a set of activities can be used to highlight the contribution of that activity to the center of pressure map 2228 and the range of motion map 2230. By selecting going downstairs, the center of pressure map is associated with that single activity, and the range of motion map associated with that single activity can be highlighted within the corresponding map. In this example, the heat map 2244 is temporarily highlighted on the center of pressure heat map 2228, which shows the concentrated portion of the map attributable to going downstairs. Similarly, the range of motion curve 2246 can be temporarily highlighted within the range of motion map 2230 to show the contribution to the range of motion map that can be attributed to this activity.

[0347] Figure 26D Is an illustration of how the user interface changes when manipulating the abduction angle 2224 and the anteversion angle 2226. In this example, all activities are selected, and the abduction angle is decreased by 12°, while the anteversion angle is decreased by 2°. This results in an expanded heat map 2248 of the central pressure, which creates a risk of increased edge loading of the acetabular cup, while the range of motion map 2249 of these activities is shifted from the center of the acetabular cup, which poses a greater risk of dislocation or injury. Figure 26D The results in Figure 26A are not as ideal as

[0348] Figure 27 Shows the effect of adding multiple activities together to produce an aggregated center of pressure heat map and range of motion map. In this example, climbing stairs and going downstairs result in a wider center of pressure and a wider range of motion than the single activity maps.

[0349] Figure 28 is a flowchart illustrating an exemplary creation of a statistical model database that can be queried using patient anatomy and implant characteristics and a selection of patient activities for estimating implant performance. This statistical model database is created by performing (typically) hundreds to thousands of simulations using a joint anatomy model (e.g., multibody simulation), where the model simulates the behavior of each component of the joint as the joint moves through a motion curve associated with at least one common motion that will occur in a patient joint when the patient engages in a given activity. In some embodiments, the motion curve can be simulated by using motion capture techniques on a sample subject performing the given activity. By capturing the motion of the individual (or individuals) performing the activity, an accurate model of the motion that each joint will experience during the activity can be created. Then, the model of the individual joint motion can be reviewed to identify exemplary repetitive motions that a person might experience when performing the activity. This motion curve model can then be used to guide each individual simulation of the activity, where the anatomical geometry and implant characteristics are varied to produce an experimental design that includes a range of anatomical geometries and implant characteristics.

[0350] In some embodiments, method 2260 begins at step 2262, where motion capture is used for a single participant while the individual performs an exemplary task associated with each activity to be simulated. For example, reflective markers can be placed on the model's body when the model climbs and descends stairs in front of one or more cameras. At step 2264, the processor can then extract the motion that these markers experience during the activity. By using an anatomical model and when these markers are associated with the joints of the individual, the motion curve that each joint experiences during the activity can be extracted. Any suitable conventional motion capture technique can be used for this step. For example, in the case where hip or knee motion curves are being generated, as the individual moves, at least two cameras can capture the reflective markers on the model's leg, iliac crest, torso, femur, tibia, patella, ankle, medial and lateral condyles, etc. The hip and knee curves can be created simultaneously with sufficient markers. As the individual moves their leg during the activity, lifting and placing their foot, the degree of motion and rotation within each degree of freedom can be calculated as the hip and knee move. The processor can then use this to estimate the relative degree of motion of the various components of the joint.

[0351] At step 2266, the processor performing the simulation of the anatomical model will load the anatomical model. An exemplary anatomical model can be a multi-body model of bone and soft tissue components that accurately simulates the behavior of each of these components of an anatomical joint. By using a multi-body model, the simulation time can be accelerated by a finite element analysis model. In some embodiments, a finite element analysis model can also be used. Exemplary multi-body simulation tools used with the modeled joint include LIFEMOD TM software available from LIEMODELER INC. Once this anatomical model is loaded, it can be customized for a given geometry. For example, the component sizes can be adjusted to achieve any joint geometry to be simulated.

[0352] At step 2268, sample joint parameters for the next simulation are selected (such as anatomical geometry and implant size, type, location, and orientation). The selection of this joint geometry can be based on a pre-planned experimental design or can be randomly assigned in a Monte Carlo style simulation. Then, the components in the model can be appropriately sized to match the geometry of this given experimental simulation. For example, in the context of TKA, the selected geometry can include any reasonable combination of distal and posterior incisions, as well as other implant features for implanting the sample prosthesis. In some embodiments, the selected geometry can include additional patient information, such as abnormal motion constraints due to other debilities or deformities, or other common medical comorbidities. In some embodiments, age and weight can be added to the model to account for variations in various components whose response can vary with the patient's age or weight (e.g., less compliant or thinner soft tissues).

[0353] At step 2270, the simulation of the model sized according to the experimental parameters is performed using a joint motion curve for a given activity. This produces a number of quantifiable results, such as soft tissue indentation and tension, e.g., the pressure between components such as the femoral head and acetabular cup (for hip replacement) and the condylar chamber gap, and patellar tracking over the range of motion between components (in knee replacement). The simulation profile used to run the simulation in step 2270 can define which results should be generated and recorded. At step 2272, once the simulation has been completed, the results for a given combination of anatomical and implant features and motion curve can be stored.

[0354] At step 2274, the processor determines whether additional parameters, such as anatomical geometry, implant design, orientation, and position, should be simulated. In many embodiments, a simulation profile that defines an experimental design or a broad Monte Carlo simulation will define that hundreds to thousands of different simulations should be performed. Thus, the loop of steps 2268 to 2274 should be repeated multiple times. Once a sufficient number of simulations have been completed, method 2260 can proceed to step 2276, where the processor mines the results of these multiple simulations to create a transfer function or model of joint behavior based on various geometries. This model or transfer function creates a statistical model for the activity and the joint, which accurately estimates performance based on the patient's anatomy and the geometry of the implanted prosthesis. In some embodiments, this can be achieved by performing a statistical fit of a polynomial function that maps the input implant values to output values. In some embodiments, a neural network or other AI strategy can be used to create a heuristic model of implant characteristics versus performance output. As additional simulation results are added to the database, step 2276 can be repeated. In some embodiments, as additional implant surgeries are clinically performed and monitored, measured real-world performance values can be added to the database and mined similar to the simulation results.

[0355] At step 2278, the processor stores and maintains the statistical model of joint performance as a function of anatomical geometry and implant characteristics (position, orientation, and type of the selected implant). This stored model can be updated when additional simulation data becomes available. In some embodiments, the resulting model can be computationally lightweight (e.g., a transfer function), allowing this model to be mined and manipulated by any processor in the system, at the server, or on the user device. At step 2280, the processor mines this statistical model to identify the best implant values for optimizing performance based on a predetermined heuristic. For example, in a model of hip replacement surgery, plots of the center of pressure and range of motion can be optimized to minimize the risk of edge loading or dislocation, minimizing the range of motion and degree of pressure that falls near the edge of the acetabular cup for a given activity. For example, in a knee replacement surgery model, the heuristic may include the gap, patellar tracking, and ligament tension being close to normal anatomical values during a given simulated activity. This optimization can be performed by any statistical or AI method, including finding the best fit to the most ideal anatomical model. Then, at step 2278, these optimized values can be added to the maintained model.

[0356] Once the model has been stored and optimizations have been performed to assist in identifying the best implant characteristics (location, orientation, implant type, implant size) for a given patient's anatomy and activities, at step 2282, the processor can query the model in response to user interaction, such as during a pre-operative or intra-operative planning phase. This allows the surgical user to access the statistical database model to develop a surgical plan that can then be used by CASS or displayed to the user.

[0357] Figure 29 FIG. 2400 is a flow chart of an exemplary method for creating a pre-operative plan using a simulation database and patient-specific geometry extracted from images and (optionally) motion capture. The same method can be used to optimize pre-operative plans for hip (or other) joint replacements, but will be discussed in the context of knee replacement. At step 2402, the surgical planning system collects pre-operative imaging, such as MRI, CT scans, X-rays, and ultrasound images. These can be done at any time prior to surgery. The image files can be loaded into the patient-specific database for feature extraction. In some embodiments, at step 2404, motion capture technology can be used during a pre-operative visit to capture the patient's gait by attaching markers to various points on the patient's leg and observing the relative movement of the markers as the patient performs various movements. This can be used to provide complementary details about the geometry of the patient's knee in its pre-operative state. At step 2406, the imaging is analyzed using geometric techniques for image analysis (and any motion capture data is analyzed based on a motion model of the human anatomy). Software performing the image analysis can use any suitable feature extraction technique to identify predefined features in the patient's image and create a three-dimensional model of the patient's knee based on a plurality of images including information such as sacral / pelvic geometry, femoral head / neck and acetabular cup geometry, femoral and tibial axes, condyle centers and sizes, existing condyle gaps, patellar size, and its existing relationship to pre-operative soft tissue tension.

[0358] In parallel, and typically prior to these steps, at step 2408, a statistical model is created that takes into account a wide range of possible patient geometries for a wide range of patients. Hundreds to thousands of simulations for various possible patient geometries can be performed offline to generate the statistical model. The simulation data can then be mined to create a transfer function or simplified model for a given geometry, allowing the performance to be determined or estimated for any given patient geometry. At step 2410, when the anatomical geometry of the individual patient has been determined (step 2406), the statistical model is loaded.

[0359] At step 2412, a statistical model is used to explore possible corrections to the given anatomical geometry of the patient. This can be done using various suitable methods known in the art, such as applying artificial intelligence search and matching algorithms to identify incremental improvements to the implant geometry to correct the patient's debilitating condition. For example, AI can be used to identify test designs to identify possible candidate changes that improve the mechanics of the patient's joint. Similarly, Monte Carlo simulations can be used to study random variations in the implant geometry using a statistical model to identify the best performing options for implanting a TKA prosthesis. In some embodiments, at step 2412, many (e.g., dozens or thousands) of variations are used to identify the best implant solution. Since this step is preoperative, the processing time or overhead can be quite large if needed. Various attributes can be varied, including the orientation of the tibial and femoral components, patellar filling, and dome geometry, etc., to find a solution that optimizes the performance of the patient's knee, which takes into account the tibiofemoral and patellofemoral joints, rather than treating the patella as an afterthought, as is often the case with existing surgical plans. In some embodiments, it can be simply based on the starting patient geometry, e.g., by using a transfer function with an AI model to query the statistical model created at step 2408 to find the best implant posture. In some embodiments, the simulated variations performed at step 2412 can be specific to different patient activities, allowing activity-specific optimization for individual patients.

[0360] In some embodiments, the purpose of step 2412 also aims to create a patient-specific model that takes into account the imprecise nature of the preoperative data. For example, preoperative images and motion capture can produce estimates of the patient's geometry that are not exact. Data collected during the surgery can be used later to refine the model of the patient's anatomical geometry. By considering multiple variations not only in the implant orientation but also in the patient's anatomical geometry (within a certain range), a patient-specific model can be created such that the surgical plan can be modified instantaneously during the surgery based on additional data or on a request from the surgeon to change the implant plan.

[0361] Once multiple variations of the patient geometry and implant orientation are considered, a patient-specific model can be stored at step 2414. This model can be stored in non-volatile memory, allowing CASS to access it during the surgery. This will be referred to Figure 30 discussed. Any suitable amount of information or format can be used, with the aim of simplifying any processing or simulation done during the patient's surgery such that changes can be processed instantaneously without slowing down the surgery.

[0362] At step 2416, the processor may create an optimized surgical plan based on the optimization of the implant pose created at step 2412 and the patient profile. For PKA / TKA, this plan may include the implant poses for the femoral and tibial components (including the resections required to achieve these poses), and a plan for any changes to the patella filling and dome or patellar button that may be required to achieve the relationship between the patella and femoral components as part of the joint replacement. At step 2420, this optimized pre-operative plan is provided to the surgeon and the CASS to prepare for the surgery.

[0363] In some embodiments, an additional step, step 2418, may be performed, whereby changes to the pre-operative plan are created to account for possible variations in the patient's anatomical geometry that may be discovered during the surgery, as well as any reasonable deviations from the surgical plan that the surgeon may make during the surgery. These changes may be associated with the expected performance outcomes of the surgically modified joint. This may make it easier for the CASS to provide recommendations during the surgery based on additional patient data observed in the operating room or based on requests from the surgeon. Effectively, this may result in a very low computational load for providing recommendations or calculating the expected performance impact of additional data or decisions during the surgery.

[0364] Figure 30 An exemplary method 2430 for updating the surgical plan or providing recommendations to the surgeon during the surgery is shown. Once the pre-operative plan is created, additional information that can be used to update this plan can be collected in the operating room. Similarly, surgeons often rely on their own experience and expertise and may adjust the surgical plan based on what they discover during the surgery or if it is inconsistent with the AI-generated recommendations. At step 2432, the CASS collects intraoperative imaging and probe data. Intraoperative imaging may include any conventional medical imaging, such as ultrasound data. This intraoperative imaging may supplement the pre-operative imaging. This may provide additional details or updates to the model of the patient's geometry. Similarly, once the patient's tissue is opened, probes can be used to "map" the various surfaces of the patient's bone, as explained throughout. This may provide additional details about the exact 3-D nature of the patient's tissue surface, which may be more accurate than the models created through two-dimensional or three-dimensional imaging. At step 2434, the processor analyzes the intraoperative imaging and probe data to supplement the model of the patient's geometry by identifying specific features observed in the operating room and comparing them to the existing model. Once the geometric model of the patient's anatomy is updated, the surgical user may request an updated plan at step 2436 (or forego the request to update and jump to step 2444).

[0365] In response to this new data and the user request, at step 2440, the processor selects the best plan based on the new anatomical geometry model. This is achieved by first loading the patient-specific model or plan from the memory at step 2438. Due to the time-critical nature of intraoperative recommendations, in some embodiments, the model and plan loaded at step 2438 are patient-specific, such as those created in Figure 27 . This limits the range of possible geometric variations to the most relevant ones based on the preoperative patient anatomical model to accelerate the processing of recommendations for plan changes. At step 2440, the best plan can be selected based on the observed patient anatomy through any conventional computational method from the available plans and models. At step 2442, this updated recommendation of the plan can be presented to the user through the user interface of CASS and presented to CASS to update the plan it will assist in implementing.

[0366] At step 2444, the processor can start monitoring the user actions or requests for plan updates. The user actions can be monitored through CASS, such as by monitoring the actual resection performed by the surgeon. For example, deviations from the surgical plan during resection may require recommended changes to other resections to limit the impact of the deviation. The user can also manually change the surgical plan based on expertise and experience, such as when the surgeon intentionally deviates from the recommended plan. These deviations from the plan will be recorded by the processor, and the processor will provide feedback to the user. If the user wants recommendations for plan changes at any time, the user interface can be used to request recommendations, which repeats step 2436.

[0367] At step 2446, the processor estimates the performance impact of the deviation from the best plan provided at step 2440 and provides feedback to the user based on this estimate at step 2448. For example, for PKA / TKA, changes in the patella / implant posture that occur during surgery may require changes to the patella. At step 2448, the GUI can cause the indicator of the patella to blink or change color to indicate a potential problem. For hip revision / THA, the hip component can blink or change color.

[0368] AI-Enhanced Cutting Guide

[0369] Some embodiments utilize the simulation databases and AI-guided planning processes described herein to improve robotic or computer-enhanced surgical planning. However, such surgical systems can be limited or unavailable. Therefore, in some embodiments, patient-specific cutting guides can be manufactured according to the same concept - which is guided in i...

Claims

1. A method for improving a robotic surgical system, comprising: Training one or more machine learning models based on historical state data obtained for a computer - assisted surgical system during each of a plurality of time periods in a plurality of historical knee replacement surgical procedures; Applying one or more of the machine learning models to initial state data of a current knee replacement surgical procedure to generate robotic commands required to achieve one or more future states of the computer - assisted surgical system, wherein the initial state data includes a surgical plan, and wherein the robotic commands are commands for manipulating one or more surgical tools of the computer - assisted surgical system to achieve one or more future states of the computer - assisted surgical system and thereby execute at least a portion of the surgical plan; Applying one or more of the machine learning models to an expected future action sequence to show the impact of planned future state data on the surgical plan; And Obtaining approval before manipulating the one or more surgical tools of the computer - assisted surgical system based on the robotic commands.

2. The method according to claim 1, wherein the initial state data further includes one or more of patient anatomy data, implant data, or one or more surgeon preferences.

3. The method according to claim 1 or 2, further comprising training the machine learning models further based on patient data associated with one or more of the historical knee replacement surgical procedures.

4. The method according to claim 1 or 2, further comprising outputting a plurality of visualizations of the future states of the computer - assisted surgical system to a display device.

5. The method according to claim 4, wherein the visualizations collectively depict aspects of the entire current knee replacement surgical procedure.

6. The method according to claim 4, further comprising modifying one or more of the visualizations based on the application of one or more of the machine learning models to obtain planned future state data for the current knee replacement surgical procedure.

7. The method according to claim 6, wherein one or more of the visualizations are modified to illustrate the impact of the planned future state data on the surgical plan.

8. The method according to claim 4, wherein the visualizations include a plurality of images corresponding to one or more aspects of the current knee replacement surgical procedure during one or more of the plurality of time periods corresponding to one or more of the future states.

9. The method according to claim 8, wherein the visualizations collectively depict one or more changes in the patient anatomy on which the current knee replacement surgical procedure is performed over one or more of the plurality of time periods.

10. The method according to claim 8, further comprising transitioning between one or more of the images on the display device in response to a received input.

11. The method according to claim 8, wherein each of the future states corresponds to one of the plurality of time periods.

12. A non - transitory computer - readable medium having stored thereon instructions including executable code for improving a robotic surgical system, the executable code causing the processor to implement the method according to any one of claims 1 - 11 when executed by one or more processors.

13. A surgical calculation device, comprising: A memory including programming instructions stored on the memory for improving a robotic surgical system; and one or more processors coupled to the memory and configured to execute the stored programming instructions to implement the method according to any one of claims 1 - 11.

14. A computer program product having instructions including executable code for improving a robotic surgical system, the executable code causing the processor to implement the method according to any one of claims 1 - 11 when executed by one or more processors.

Images