Method and system for registering three-dimensional bone model

Abstract

And registering the three-dimensional bone model. Various examples relate to methods and related systems that identify surface features of rigid structures visible in a video stream and use the surface features to register three-dimensional models for computer-aided navigation of surgical procedures. In some examples, the surface features are determined using non-contact techniques based on known or calculated motion of the camera. In other examples, the surface features are collected using a contact probe that is not itself tracked directly. In other examples, the three-dimensional model may be registered by using a patient-specific instrument coupled to the rigid structure in only one orientation.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 457,908, filed April 7, 2023, entitled “Methods and Systems of Registering a Three-Dimensional Bone Model”. That provisional application is incorporated herein by reference as if it were reproduced authentically below. Background Technology

[0003] Arthroscopic surgery is a minimally invasive surgical procedure in which the surgical site inside the body is accessed through small holes or ports inserted through the patient's skin. Various tissues within the surgical site are visualized using an arthroscope placed through the port, and the internal scene is displayed on an external display device. Tissues can be repaired or replaced through the same or additional ports. In computer-assisted surgery (e.g., anterior cruciate ligament (ACL) replacement, reduction of femoral-acetabular impingement), the position of various objects relative to the surgical site can be tracked relative to the bone using images captured through the arthroscope and a three-dimensional model of the bone. Attached Figure Description

[0004] For a detailed description of the exemplary embodiments, reference will now be made to the accompanying drawings, in which:

[0005] Figure 1 An anterior view or anterior elevation of the right knee is shown, in which the patella has been removed;

[0006] Figure 2 A rear view or rear elevation view of the right knee is shown;

[0007] Figure 3 This shows a view of the femur in the intercondylar fossa as seen from below;

[0008] Figure 4 A surgical system according to at least some embodiments is shown;

[0009] Figure 5 A conceptual diagram of a surgical site according to at least some embodiments is shown, wherein various objects within the surgical site are tracked;

[0010] Figure 6 The exemplary video shown according to at least some embodiments illustrates the various parts of the femur and has bone references visible therein;

[0011] Figure 7 A method according to at least some embodiments is shown;

[0012] Figure 8is an exemplary video display during a registration process according to at least some embodiments, showing portions of a femur and bone fiducials;

[0013] Figure 9 shows a method according to at least some embodiments;

[0014] Figure 10 shows a method according to at least some embodiments;

[0015] Figure 11 shows a front or elevational view of a right knee according to at least some embodiments, with the patella removed and a patient-specific instrument installed; and

[0016] Figure 12 shows a computer system according to at least some embodiments.

[0017] Definitions

[0018] Various terminology is used to refer to particular features of the system. Different companies may refer to components by different names - this document does not intend to distinguish between components that differ in name but not in function. In the following discussion and in the claims, the terms "including" and "comprising" are used in an open-ended way to mean "including, but not limited to...". Also, the term "couple" or "couples" is intended to mean either an indirect or direct connection. Thus, if a first device couples to a second device, that connection can be through a direct connection, or through an indirect connection via other devices and connections.

[0019] An endoscope having a "single optical path" through the endoscope shall mean that the endoscope is not a stereoscopic endoscope having two different optical paths separated by an interpupillary distance at the light collection end of the endoscope. The fact that the endoscope has two or more optical members (e.g., glass rods, optical fibers) that form a single optical path shall not preclude status as a single optical path. DETAILED DESCRIPTION

[0020] The following discussion relates to various embodiments of the application. Although one or more of these embodiments can be preferred, the disclosed embodiments should not be construed or otherwise used as limiting the scope of the disclosure, including the claims. Additionally, those skilled in the art will recognize that the following description has broad applicability and is only intended to be illustrative of one embodiment and not restrictive of the scope of the disclosure, including the claims.

[0021] Various examples relate to methods and systems that register a three-dimensional model of a rigid structure, such as a bone. More particularly, various examples relate to methods and related systems that identify surface features of a rigid structure visible in a video stream and use the surface features to register a three-dimensional model for use in computer-aided navigation of a surgical procedure. In some examples, the surface features are determined using non-contact techniques based on known or computed motion of a camera. In other examples, the surface features are collected using a contact probe that is not itself directly tracked; rather, the pose of the contact probe, and thus the location of the distal tip of the contact probe that contacts the bone, can be determined by segmenting frames of the video stream and pose estimation. In still other examples, the three-dimensional model can be registered by using a patient-specific instrument that is coupled to the rigid structure in only one orientation; thus, a fiducial coupled to the patient-specific instrument, or in some cases the patient-specific instrument itself without a fiducial, can be used to register the three-dimensional bone model.

[0022] Various examples were developed in the context of anterior cruciate ligament (ACL) repair, and thus the discussion below is based on that development context. In this case, the rigid structure is a bone, and the three-dimensional model is a three-dimensional bone model. However, the techniques are applicable to any suitable rigid anatomical structure, such as a tooth. Moreover, the various techniques can be applicable to many types of surgical procedures, such as repairs associated with the knee, hip, shoulder, wrist, or ankle. The techniques can be applicable not only to ligament repairs (e.g., medial collateral ligament repair, lateral collateral ligament repair, and posterior cruciate ligament repair), but also to surgical procedures that plan and place anchors to reattach soft tissue (e.g., to reattach the labrum, rotator cuff, or meniscal root of a hip joint), as well as surgical procedures that address femoral acetabular impingement. Thus, the description and development context should not be read as limiting the applicability of the teachings. To orient the reader, the description first turns to a description of the knee.

[0023] Figure 1 A front view or elevational view of a right knee is shown, with the patella removed. In particular, a lower portion of the femur 100 is visible, including the lateral or outside condyle 102 and the medial or inside condyle 104. The femur 100 and condyles 102 and 104 are in operative relationship with the tibia 106, which includes the tibial tuberosity 108 and the Pellegrini-Stieda tubercle 110. The lateral meniscus 112 and the medial meniscus 114 are disposed between the femoral condyles 102 and 104 and the tibia 106. In Figure 1 Figure 1 A number of ligaments are also visible in the view, such as the ACL 116, which extends from the lateral side of the femoral notch to the medial side of the tibia 106. Conversely, the posterior cruciate ligament 118 extends from the medial side of the femoral notch to the tibia 106. The fibula 120 is also visible, as are a number of additional ligaments, which are not specifically numbered.

[0024] Figure 2 A posterior view or elevational view of a right knee is shown. In particular, a lower portion of the femur 100 is visible, including the lateral or outside condyle 102 and the medial or inside condyle 104. The femur 100 and condyles 102 and 104 are in operative relationship with the tibia 106, which includes the tibial tuberosity 108 and the Pellegrini- Stieda tubercle 110. The lateral meniscus 112 and the medial meniscus 114 are disposed between the femoral condyles 102 and 104 and the tibia 106. In​Figure 2 The lower portion of the femur 100 is visible, including the lateral condyle 102 and the medial condyle 104. The femur 100 and the femoral condyles 102 and 104 are again in operative relationship with the tibia 106, and the lateral meniscus 112 and the medial meniscus 114 are disposed between the femoral condyles 102 and 104 and the tibia 106. Figure 2 Also shown is the ACL 116 extending from the lateral aspect of the femoral fossa to the medial aspect of the tibia 106, but its attachment point to the tibia 106 is not visible. The posterior cruciate ligament 118 extends from the medial aspect of the femoral fossa to the tibia 106, but its attachment point to the femur 100 is not visible. Similarly, several additional ligaments, not specifically numbered, are shown.

[0025] The most common ACL injury is a complete tear of the ligament. Treatment involves reconstructing the ACL by placing an alternative graft (e.g., an autologous graft from the patellar tendon, quadriceps tendon, or hamstring tendon). The graft is placed in a prepared tunnel within the femoral 100 and tibial 106. The current standard of care for ACL repair is to position the tunnel so that the tunnel entrance point of the graft is located at the anatomical attachment site of the natural ACL. This tunnel placement at the attachment site of the natural ACL attempts to reconstruct the original knee kinematics. In arthroscopic surgery, the location of the tunnel through the tibial 106 is relatively easy to reach, especially when the knee is flexed or bent. However, the tunnel through the femoral 100 is located in the intercondylar fossa. Depending on the patient's body size and the surgeon's choice of location for the port through the skin, reaching the attachment site of the natural ACL to the femoral 100 can be difficult.

[0026] Figure 3 This shows a view of the femur in the intercondylar fossa from below. Specifically, in... Figure 3 The lateral condyle 102 and medial condyle 104 are visible. The femoral fossa 200 is defined between the femoral condyles 102 and 104. A femoral tunnel can define an internal aperture 202 within the femoral fossa 200, which is closer to the lateral condyle 102 and displaced into the posterior portion of the femoral fossa 200. The femoral tunnel extends through the femur 100 and forms an external aperture on the lateral or outer surface of the femur 100 (the external aperture is located on...). Figure 3 (Not visible in the middle). Figure 3 An exemplary drill wire 204 is shown that can be used to create an initial tunnel or guide hole. Once the surgeon verifies that the guide hole is closely aligned with the planned tunnel path, the femoral tunnel is created by drilling or reaming with another instrument (e.g., a reamer) that can use the drill wire 204 as a guide. In some cases, a countersunk hole or inlet is created on the intercondylar fossa side to accommodate the width of the graft extending into the bone, and this countersunk hole can also be created using another instrument (e.g., a reamer) that can use the drill wire 204 as a guide.

[0027] Figure 4A surgical system (not to scale) is shown in accordance with at least some embodiments. In particular, the example surgical system 400 includes a tower or device cart 402, an example mechanical resection instrument 404, an example plasma-based ablation instrument (hereinafter ablation instrument 406), and an endoscope in the example form of an arthroscope 408 and attached camera head or camera head 410. In the example system, the arthroscope 408 is a rigid device unlike endoscopes used for other surgeries, such as the upper endoscope. The device cart 402 can include a display device 414, a resection controller 416 and camera head control unit (CCU), and an endoscope light and video controller 418. In the example case, the combined CCU and video controller 418 not only provides light to the arthroscope 408 and displays images received from the camera head 410, but also implements various additional aspects, such as registering a three-dimensional bone model with the bone visible in the video images, and providing computer-aided navigation during the surgery. Thus, the combined CCU and video controller is referred to hereinafter as the surgical controller 418. However, in other cases, the CCU and video controller can be separate and distinct systems from the controller, which handle registration and computer-aided navigation, but the separate devices are still operatively coupled.

[0028] The example device cart 402 also includes a pump controller 422 (e.g., a single or dual peristaltic pump). Fluid connections of the mechanical resection instrument 404 and the ablation instrument 406 to the pump controller 422 are not shown to avoid overcomplicating the figure. Similarly, fluid connections between the pump controller 422 and the patient are not shown to avoid overcomplicating the figure. In the example system, both the mechanical resection instrument 404 and the ablation instrument 406 are coupled to the resection controller 416, which is a dual-function controller. However, in other cases, there can be a mechanical resection controller that is separate and distinct from the ablation controller. The example devices and controllers associated with the device cart 402 are merely examples, and other examples include a vacuum pump, a patient positioning system, a robotic arm holding various instruments, an ultrasonic cutting device and related controller, a patient positioning controller, and a robotic surgical system.

[0029] Figure 4 Additional instruments that can be present during arthroscopic surgery are also shown. In particular, Figure 4An exemplary contact probe 424, drill guide or scope 426, and bone fiducial 428 are shown. The contact probe 424 can be used during the surgical procedure to provide information to the surgical controller 418, such as information to register the three-dimensional bone model to the underlying bone visible in the images captured by the arthroscope 408 and camera head 410. The scope 426 can be used as a guide for placing a drill wire and drilling with the drill wire to create an initial or pilot tunnel through the bone. The bone fiducial 428 can be fixed or rigidly attached to the bone and used as an anchor location for the surgical controller 418 to know the orientation of the bone (e.g., after registration of the three-dimensional bone model). There will be additional tools and instruments, such as drill wires, various reamers for forming through holes and counterbores for tunnels through the bone, and various tools for suturing and anchoring grafts. These additional tools and instruments are not shown to further complicate the figure. The description now turns to an exemplary workflow for an anterior cruciate ligament repair.

[0030] The surgical procedure can begin with a planning phase. An exemplary anterior cruciate ligament repair can begin with imaging (e.g., X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI)) of the patient's knee, including the relevant anatomy, such as the lower portion of the femur, the upper portion of the tibia, and the articular cartilage. The imaging can be preoperative imaging, hours or days before the intraoperative repair, or the imaging can be performed within the surgical environment shortly before the intraoperative repair. The following discussion assumes MRI imaging, but again many different types of imaging can be used. The image slices can be segmented from the MRI imaging, such that a volumetric model or three-dimensional model of the anatomy is created. Any suitable currently available or later developed segmentation technique can be used to create the three-dimensional model. More specifically, for the example of an anterior cruciate ligament repair, a three-dimensional bone model of the lower portion of the femur, including the femoral condyles, is created.

[0031] Using the three-dimensional bone model, a surgical plan is created that includes selecting a planned tunnel path through the femur, including the location of holes in the bone that define the ends of the tunnel. For an exemplary inside-out repair, the hole in the femoral fossa is the entry location for drilling, and the hole on the lateral surface of the femur is the exit location. For an outside-in repair, the entry and exit locations for drilling are swapped. Still assuming an inside-out repair, the entry location can be selected to be the same as or close to the natural anterior cruciate ligament's attachment location to the femur. In some cases, selecting the entry location in the femoral fossa can involve using the Bernard & Hertel quadrant or grid placed on a fluoroscopy image, or placing the Bernard & Hertel quadrant on a simulated fluoroscopy image created from the three-dimensional bone model. Based on the use of the Bernard & Hertel quadrant, the entry location for the tunnel is selected. For an inside-out repair, the selection of the exit location is less constrained, not only because the portion of the tunnel near the exit location is used to place the anchor for the graft, but also because the exit location is roughly centered in the femur (considering from front to back), so the issue of bone wall thickness at the exit location is less important. In some cases, a three-dimensional bone model of the proximal end of the tibia is also created, and the surgeon can likewise select a planned tunnel path(s) through the tibia.

[0032] The results of the planning can include: a three-dimensional bone model of the distal end of the femur; a three-dimensional bone model of the proximal end of the tibia; the entry and exit locations for the planned tunnel path through the femur and thus the femur; and the entry and exit locations for the planned tunnel path through the tibia and thus through the tibia. Other surgical parameters can also be selected during the planning, such as tunnel through-hole diameter, tunnel counterbore diameter and depth, desired post-repair flexion, etc., but these additional surgical parameters are omitted to avoid overcomplicating the specification.

[0033] The specification now turns to intraoperative aspects. The intraoperative aspects include steps and procedures for setting up the surgical system to perform various repairs. However, it should be noted that some of the intraoperative aspects (e.g., optical system calibration) can occur before any ports or incisions through the patient's skin are made, and indeed before the patient is wheeled into the operating room. However, these steps and procedures can be considered intraoperative in that they occur in the surgical environment and using the surgical equipment and instruments used to perform the actual repair.

[0034] The exemplary ACL repair is performed arthroscopically and is computer-aided in the sense that the surgical controller 418 is used for arthroscopic navigation within the surgical site. More specifically, in the exemplary system, the surgical controller 418 provides computer-aided navigation during ligament repair by tracking the positions of various objects within the surgical site, such as the position of bone in the three-dimensional coordinate space of the arthroscopic view, and the positions of various instruments (e.g., burrs) in the three-dimensional coordinate space of the arthroscopic view. The specification turns to a brief description of such tracking techniques.

[0035] Figure 5 A conceptual diagram of a surgical site containing various objects is shown. Specifically, in... Figure 5 The distal end of the arthroscopy 408, a portion of the bone 500 (e.g., the femur), the bone reference 428 within the surgical site, and the contact probe 424 are visible. Each is described in turn.

[0036] The arthroscopic 408 uses visible light to illuminate the surgical site. Figure 5 In the example, illumination is indicated by arrow 508. The illumination provided to the surgical site is reflected by various objects and tissues within the surgical site, and the reflected light returning to the distal end enters the arthroscope 408, propagates along the optical path within the arthroscope 408, and finally enters the camera 410. Figure 4 The images detected by the capture array within the camera 410 are electronically transmitted to the surgical controller 418. Figure 4 ) and in display device 414 ( Figure 4 As shown on the image. In one example, arthroscopy 408 is monocular or has a single optical path through the arthroscopy for capturing images of the surgical site, but the single optical path may consist of two or more optical components (e.g., glass rods, optical fibers). That is, in the exemplary system and method, computer-aided navigation provided by arthroscopy 408, camera 410, and surgical controller 418 is provided with arthroscopy 408, which is not a stereoscopic endoscope having two different optical paths separated by the interocular distance at the distal endoscope.

[0037] During surgical procedures, surgeons select an arthroscopy with a line of sight that is advantageous for the planned surgical procedure. The line of sight is the center of the angle between the outer edge or periphery of the view from the endoscope. Some arthroscopy have their line of sight aligned with the longitudinal central axis of the arthroscopy, and such arthroscopy are called “zero-degree” arthroscopy (e.g., the angle between the line of sight and the longitudinal central axis of the arthroscopy is zero degrees). Other arthroscopy have a line of sight that forms a non-zero angle with the longitudinal central axis of the arthroscopy. For example, for a 30° arthroscopy, the line of sight forms a 30° angle with the longitudinal central axis of the arthroscopy, which is measured as an obtuse angle extending distal to the arthroscopy. In many cases of ACL repair, surgeons select either a 30° or 45° arthroscopy based on the location of the port created through the patient's skin. Figure 5 In the example, the angle 510 of the arthroscope 408 forms a non-zero angle with the longitudinal central axis 512 of the arthroscope 408.

[0038] Still referencing Figure 5 Within the view of arthroscopy 408, there is a portion of bone 500 (e.g., within the intercondylar fossa), and an exemplary bone reference 428 and an exemplary contact probe 424. The exemplary bone reference 428 is a multi-faceted element, wherein a reference is set or created on each facet or facet. However, the bone reference does not need to have multiple faces and can actually take any shape, as long as that shape can be tracked within the video image. The bone reference (e.g., bone reference 428) can be attached to bone 500 in any suitable form; in this example, it is fastened by a screw portion of the bone reference 428 (in... Figure 5 Invisible in the middle, but in Figure 4 (See image). The reference pattern on each face is designed to provide information about the orientation of the bone reference 428 in the three-dimensional coordinate space of the view of the arthroscope 408. More specifically, the pattern is selected so that it can be adjusted according to the arthroscope 408 and the attached camera (…). Figure 4 The captured images determine the orientation of bone reference 428.

[0039] Contact probe 424 is also shown as partially visible in the view of arthroscopy 408. As discussed more below, contact probe 424 can be used to identify multiple surface features on bone 500 as part of registering bone 500 to a three-dimensional bone model. Alternatively, although not specifically shown, aiming device 426 ( Figure 4) can carry their own unique fiducials, such that their respective poses can be computed from one or more fiducials present in the video stream. However, in other cases, and as shown, the medical instrument used to assist in the registration of the three-dimensional bone model, whether it be the contact probe 424, the scope 426, or any other suitable medical device, can omit carrying a fiducial. In other words, in such examples, the medical instrument does not have a fiducial marker. In such cases, the pose of the medical instrument can be determined by a machine learning model, which is discussed in greater detail below.

[0040] Images captured by the arthroscope 408 and attached camera are subject to optical distortion in many forms. For example, the field of view between the distal end of the arthroscope 408 and the bone 500 within the surgical site is filled with fluid, such as body fluids and saline used to distend the joint. Many arthroscopes have one or more lenses at the distal end that widen the field of view, and the wider field of view results in a “fisheye” effect in the captured images. In addition, optical elements within the arthroscope (e.g., rod lenses) can have optical aberrations inherent to the manufacturing and / or assembly process. Furthermore, the camera can have various optical elements used to focus the received images onto the capture array, and the various optical elements can have aberrations inherent to the manufacturing and / or assembly process. In an example system, prior to use in each surgical procedure, the endoscopic optical system is calibrated to account for the various optical distortions. The calibration creates a characterization function that characterizes the optical distortions, and further analysis of frames of the video stream can use the characterization function to compensate prior to further analysis.

[0041] The next example step of the intraoperative procedure is to register the bone model created during the planning phase. During the intraoperative repair, the three-dimensional bone model is obtained by or provided to the surgical controller 418. Again using the example of an anterior cruciate ligament repair, and specifically for computer-aided navigation of the tunnel path through the femur, a three-dimensional bone model of the lower portion of the femur is obtained by or provided to the surgical controller 418. Thus, the surgical controller 418 receives the three-dimensional bone model, and assuming the arthroscope 408 is inserted into the knee by way of a port through the patient’s skin, the surgical controller 418 also receives video images of a portion of the lower end of the femur. To relate the three-dimensional bone model to the images received by way of the arthroscope 408 and the camera 410, the surgical controller 418 registers the three-dimensional bone model to the images of the femur received by way of the arthroscope 408 and the camera 410.

[0042] To perform the registration, and in accordance with an example method, a bone fiducial 428 is attached to the femur. The bone fiducial placement is such that the bone fiducial is within the field of view of the arthroscope 408, but at a location spaced from the intended tunnel entry / exit point through the lateral condyle. More particularly, in the example case, the bone fiducial 428 is placed within the intercondylar notch, above the intended location of the tunnel through the lateral condyle.

[0043] Figure 6 is an example video display frame showing portions of the femur and the bone fiducial. This display can be shown, for example, on a display device 414 Figure 4 associated with the device cart 402 Figure 4 . In particular, the femoral notch or intercondylar notch 600, a portion of the lateral condyle 102, a portion of the medial condyle 104, and the example bone fiducial 428 are visible in Figure 6 . In this example, the bone fiducial 428 has a machine-readable pattern or fiducial on each outer face, and in some cases, each fiducial is unique. In other cases, the upper face of the bone fiducial 428 opposite the face from which the threaded portion extends can have an attachment feature and thus not carry a fiducial. In still other cases, the bone fiducial can have only a single face with a fiducial when the viewing direction of the arthroscope 408 is relatively constant. Regardless of the precise arrangement of the bone fiducial 428, once placed, the bone fiducial 428 represents a fixed location on the outer surface of the bone in the view of the arthroscope 408, even as the position of the arthroscope 408 moves and changes relative to the bone fiducial 428. Initially, the surgical controller 418 does not know the position of the bone fiducial 428 relative to the three-dimensional bone model, and thus needs to register the three-dimensional bone model.

[0044] To relate or register the bone visible in the video image to the three-dimensional bone model, the surgical controller 418 Figure 4 determines a plurality of surface features of the outer surface of the bone. Identifying surface features can take several forms, including a contact-based registration using the contact probe 424 that does not carry a fiducial, a non-contact registration technique that identifies surface features after resolving the motion of the arthroscope 408 and camera relative to the bone fiducial 428, and a third technique that uses a patient-specific instrument. Each will be described in turn.

[0045] In an example contact-based registration, the surgeon can use the contact probe 424 Figure 4) contacting multiple locations. In some cases, particularly when portions of the outer surface of the bone are exposed to view, receiving multiple surface features of the outer surface of the bone can involve the surgeon "painting" the outer surface of the bone. "Painting" is a technical term that does not involve the application of color or pigment, but rather means the motion of the contact probe 424 as the distal end of the contact probe 424 contacts the bone. In this example, the contact probe 424 does not carry or have visible to the arthroscope 408 and camera 410 fiducials. Subsequently, the pose of the contact probe 424 and the location of the distal tip of the contact probe 424 need to be determined in order to collect the surface features for the purpose of registering the three-dimensional bone model.

[0046] Figure 7 A method according to at least some embodiments is shown. The example method can be implemented in software within a computer system, such as the surgical controller 418. In particular, the example method starts (block 700) and includes obtaining a three-dimensional bone model (block 702). That is, in the example method, what is obtained is a three-dimensional bone model that can be created by segmenting a plurality of non-invasive images (e.g., CT, MRI) taken preoperatively or intraoperatively. In the case where the bone is segmented from images or within images, a three-dimensional bone model can be created. The three-dimensional bone can take any suitable form, such as a computer aided design (CAD) model, a point cloud of data points relative to an arbitrary origin, or a parametric representation of a surface expressed using analytical mathematical equations. Thus, the three-dimensional bone model is defined relative to an origin and on any suitable orthogonal basis.

[0047] A next step in the example method is to capture video images of a bone fiducial attached to the bone (block 704). This capture is performed intraoperatively. In the example case of an arthroscopic anterior cruciate ligament repair, the capture of the video images is by means of the arthroscope 408 and camera 410. Other endoscopes can be used, such as an endoscope with the array of capture located at the distal end of the device (e.g., a chip-on-the-tip device). However, in open surgery where the skin is cut and pulled back, thereby exposing the bone to open air, the capture can be performed by any suitable camera device (e.g., one or both cameras of a stereo camera system) or portable computing device (e.g., a tablet or smartphone device). The video images can be provided to the surgical controller 418 in any suitable form.

[0048] A next step in the example method is to determine a position of a distal tip of the medical instrument visible within the video images (block 706), where the distal tip contacts the bone in at least some frames of the video images, and the medical instrument does not have a fiducial. Determining the position of the distal tip of the medical instrument can take any suitable form. In one example, determining the position can include segmenting the medical instrument in frames of the video images (block 708). The segmentation can take any suitable form, such as applying the video images to a segmentation machine learning algorithm. The segmentation machine learning algorithm can take any suitable form, such as a neural network or convolutional neural network trained with a training data set showing the medical instrument in a plurality of known orientations. The segmentation machine learning algorithm can produce segmented video images, in which the medical instrument is identified or highlighted in some way (e.g., a box, increased brightness, removal of other objects).

[0049] With the segmented video images, the example method can estimate a plurality of poses of the medical instrument within a respective plurality of frames of the video images (block 710). Estimating the poses can take any suitable form, such as applying the video images to a pose machine learning algorithm. The pose machine learning algorithm can take any suitable form, such as a neural network or convolutional neural network trained to perform six-dimensional pose estimation. For at least some frames of the video images, the result of the pose machine learning algorithm can be an estimated pose of the medical instrument in a reference frame of the video images and / or in a reference frame provided by the bone fiducial. That is, the result of the pose machine learning algorithm can be a plurality of poses, each for at least some frames of the segmented video images. While in many cases a pose can be determined for each frame, in other cases pose estimation can not be possible for at least some frames due to video quality issues, such as motion blur caused by electronic shutter operation.

[0050] A next step in the example method is to determine a position based on the plurality of poses (block 712). In particular, for each frame for which a pose can be estimated, based on a model of the medical device, a position of the distal tip can be determined in the reference frame of the video images and / or the bone fiducial. Thus, the result is a set of positions, at least some of which represent positions of the outer surface of the bone.

[0051] Figure 7An exemplary three-step process for determining the location of the distal tip of a medical device is shown. However, this method is merely an example, and many variations are possible. For example, a single machine learning model, such as a convolutional neural network (CNN), can be built and trained to perform all three steps as a single, holistic process, although the CNN may have many hidden layers. That is, the CNN can segment the medical device, perform six-dimensional pose estimation, and determine the location of the distal tip in each frame. In this case, the training dataset would include a dataset in which each frame has the segmented medical device, the identified six-dimensional pose, and the identified location of the distal tip. The output of determination step 706 can be a segmented video stream different from the video image captured at step 704. In such cases, later method steps can use both the segmented video stream and the video image to perform additional tasks. In other cases, location information can be combined with the video image, for example, embedded in the video image, or added as metadata to each frame of the video image.

[0052] Figure 8 This is an exemplary video display showing the various parts of the femur and the bone reference during the registration process. For example, this display can be shown on a display device 414 associated with the device trolley 402 or at any other suitable location. Specifically, in Figure 8 The intercondylar fossa 600, a portion of the lateral condyle 102, a portion of the medial condyle 104, and an exemplary bone reference 428 are visible in the main portion of the display. A depiction of bone is shown in the upper right corner of the exemplary display, which may be a rendering 800 of bone created from a three-dimensional bone model. A recommended region 802 is shown on the rendering 800; the recommended region 802 consists of portions of the bone surface that will be “painted” as part of the registration process. A depiction of bone is shown in the lower right corner of the exemplary display, which again may be a rendering 804 of bone created from a three-dimensional bone model. Multiple surface features 806 on the bone model, identified as part of the registration process, are shown on the rendering 804. A progress indicator 808 is also shown in the lower right corner of the exemplary display, indicating the progress of providing and receiving positions on the bone. The exemplary progress indicator 808 is a horizontal bar with a length proportional to the number of positions received, but any suitable graphical or numerical display showing progress (e.g., 0% to 100%) can be used.

[0053] Referring to both the primary display and the lower right rendering, as the surgeon contacts the outer surface of the bone within the image captured by the arthroscope 408 and camera 410, the surgical controller 418 receives surface features on the bone and can display each location as a point or location 806 within the primary display and within the rendering shown in the lower right corner. More specifically, the example surgical controller 418 superimposes an indication of the recognized surface features 806 on the display of the image captured by the arthroscope 408 and camera 410 and, in the example case shown, also superimposes an indication of the recognized surface features 806 on the rendering 804 of the bone model. In addition, as the number of recognized locations 806 increases, the surgical controller 418 also updates a progress indicator 808.

[0054] Still referring to Figure 8 , despite the surgeon's diligence, not all of the locations recognized by the surgical controller 418 based on the surgeon's movement of the contact probe 424 result in valid locations on the surface of the bone. In the example case shown, as the surgeon moves the contact probe 424 from the inner surface of the lateral condyle 102 to the inner surface of the medial condyle 104, the surgical controller 418 receives a number of locations 810 based on the example six-dimensional pose estimation that can represent locations where the distal end of the contact probe 424 is not in contact with the bone. Figure 8

[0055] Returning to Figure 7 , the plurality of surface features 806 can be a registration model relative to the bone fiducials 428 or the example surgical controller 418 can generate a registration model relative to the bone fiducials (block 714). The registration model can take any suitable form, such as a computer-aided design (CAD) model or a point cloud of data points based on any suitable orthogonal basis. Regardless of the form, the registration model can have fewer total data points or less "structure" than the bone model created by non-invasive computer imaging (e.g., MRI). However, the goal of the registration model is to provide a basis for a coordinate transformation and scaling used to relate the bone model to the registration model and relative to the bone fiducials 428. Thus, the next step in the example method is to register the bone model relative to the location of the bone fiducials based on the registration model (block 716). The registration can conceptually involve testing a plurality of coordinate transformation and scaling values to find a correlation having a sufficiently high correlation or confidence factor. Once a correlation having a sufficiently high confidence factor is found, the bone model is said to be registered to the location of the bone fiducials. Thereafter, the example registration method can end (block 718); however, the surgical controller 418 can then use the registered bone model to provide computer-aided navigation regarding a procedure involving the bone.

[0056] ​In the example presently discussed, the registration of the bone model involves a contact-based registration technique using a contact probe 424 that does not carry fiducials. However, other registration techniques are possible, and the specification now turns to a non-contact registration technique. The example non-contact registration technique again relies on the placement of bone fiducials 428, as shown in Figure 6 FIG. 4. As previously mentioned, the bone fiducials can have fewer faces with corresponding fiducials when the viewing direction of the arthroscope 408 is relatively constant. Once placed, the bone fiducials 428 indicate fixed locations on the outer surface of the bone in the view of the arthroscope 408, even as the position of the arthroscope 408 moves and changes relative to the bone fiducials 428. Again, to correlate or register the bone visible in the video images with the three-dimensional bone model, the surgical controller 418 Figure 4 ) determines a plurality of surface features of the outer surface of the bone, and in this example, determines the plurality of surface features based on a non-contact registration technique in which the surface features are identified based on the motion of the arthroscope 408 and the camera 410 relative to the bone fiducials 428.

[0057] Figure 9 An example non-contact registration method is shown. The example method can be implemented in software within a computer system, such as the surgical controller 418. In particular, the example method begins (block 900) and includes obtaining a three-dimensional bone model (block 902). Much like the contact-based registration technique, in the non-contact registration technique, what is obtained is a three-dimensional bone model that can be created by segmenting a plurality of non-invasive images (e.g., MRI) taken pre-operatively or intra-operatively.

[0058] The next step in the example method is to capture video images of bone fiducials attached to the bone (block 904). Here again, the capturing can be performed intra-operatively. In the example case of an arthroscopic anterior cruciate ligament repair, the capturing of the video images is performed with the aid of the arthroscope 408 and the camera 410. However, in open surgery where the skin is cut and pulled back, thereby exposing the bone to open air, the capturing can be performed by any suitable camera device (e.g., one or both cameras of a stereo camera system) or portable computing device (e.g., a tablet or smartphone device). The video images can be provided to the surgical controller 418 in any suitable form.

[0059] A next step in the example method is to resolve the motion of the camera relative to the bone fiducial based on the video images (block 906). In particular, the bone fiducial 428 not only represents a fixed position relative to the bone, but the bone fiducial also directly or indirectly provides dimensional and scaling information. For example, fiducials visible on each face of the bone fiducial can be of a predetermined size or can define a predetermined size. By determining the aspect ratio and changes in size of the fiducials in frames of the video images, the surgical controller 418 can determine the motion of the arthroscope 408 and the camera 410 frame-by-frame. Over the course of multiple frames, the motion of the camera relative to the bone fiducial can be determined. In other words, for at least some frames of the video images, the pose of the arthroscope 408 relative to the bone fiducial 428 can be determined. Changes in pose frame-by-frame represent the motion of the arthroscope 408 relative to the bone fiducial 428.

[0060] A next step in the example method is to identify a plurality of surface features on the bone in the video images based on the motion of the camera (block 908). More specifically, in the example non-contact registration technique, the plurality of surface features is identified from frames of the video stream without using a medical instrument that physically contacts the bone in the video images. More precisely, in the example non-contact registration technique, the plurality of surface features is identified from frames of the video stream without using a contact probe, a scope, or other medical instrument that physically contacts the bone in the video images. Determining the surface features can take any suitable form. In one example, determining the surface features can include applying the video images and an indication of the motion of the camera to a machine learning algorithm trained to identify optical textures, e.g., spatial variations in luminance intensity, even in the absence of physical features, or optical textures caused by surface features such as contours, ridges, and / or valleys. The machine learning algorithm can take any suitable form, such as a neural network or convolutional neural network trained with a training data set identifying surface features for a plurality of different poses of the camera and a plurality of different bones. Variations of the identification are discussed in more detail below.

[0061] A subsequent step in the example method is to generate a registration model of the bone relative to the bone fiducial (block 910), and to register a bone model relative to the bone fiducial (block 912). The generation of the registration model and the registration of the bone model are similar or identical to those discussed above with respect to the contact-based registration technique of Figure 6 For the sake of not unduly prolonging the specification, the generation of the registration model and the registration of the bone model based on the registration model are not repeated here. Thereafter, the example registration method can end (block 914); however, the surgical controller 418 can then use the registered bone model to provide computer-aided navigation with respect to a procedure involving the bone. The discussion now turns to a more detailed description of identifying the plurality of surface features.

[0062] One of the problems faced in identifying multiple surface features on a bone is that the bone and surrounding tissue are low-textured. Moreover, in the limited field of view of the arthroscope 408, the outer surface of the bone can have smoothly varying features. Thus, detecting surface features by finding and tracking salient features is a non-trivial task.

[0063] There are photogrammetry techniques for simultaneously extracting three-dimensional structure and camera motion from multiple frames of two-dimensional video images. More specifically, within the subset of computer vision are techniques known as Structure from Motion (SfM) techniques. In related art Structure from Motion techniques, a computer system identifies salient features within video images and tracks these features from frame to frame. However, in the context of identifying multiple surface features on a bone in video images captured by the arthroscope 408 and camera 410, related art Structure from Motion techniques are inadequate. Also, in the context of an anterior cruciate ligament repair, the bone and surrounding tissue are low-textured surfaces, which present difficulties to Structure from Motion methods. In more detail, arthroscopic sequences are particularly challenging not only due to poor texture but also due to the presence of deformable tissue, complex lighting, shadows, highlights, fog, condensation, and very close range acquisition. Additionally, the camera is handheld, the lens range rotates, the procedure is performed in a wet medium, and the surgeon frequently switches camera entry. Attempts to use visual SLAM pipelines reported to work in laparoscopy were fruitless and revealed that additional visual aids are needed to achieve the robustness required for a true clinical application. Thus, related art Structure from Motion techniques are unable to consistently select salient features of the outer surface of a bone, nor are they able to find the correct correspondence between features in consecutive frames, thus making it difficult to simultaneously solve for camera and three-dimensional structure motion.

[0064] Exemplary non-contact registration techniques at least partially address the problems of related art Structure from Motion techniques by using artificial features in the form of bone fiducials 428. In particular, in exemplary methods and systems, the bone fiducials 428 are visible in video images as the arthroscope 408 and camera 410 move. Because the bone fiducials 428 represent known or knowable dimensions and fixed positions, the motion of the arthroscope 408 and camera 410 can be precisely determined or resolved. When motion is resolved with high accuracy, it becomes an easier computational task to identify the three-dimensional structure represented in two-dimensional video images. That is, with the motion of the arthroscope 408 and camera 410 resolved with high accuracy, identifying surface features on a bone becomes a more deterministic and easier computational task, thereby improving the operation of a computer system in the form of the surgical controller 418.

[0065] In particular, with known camera motion, several techniques can be used to determine the three-dimensional structure represented in the two-dimensional video images. For example, one technique can be to perform feature extraction, matching, and reconstruction on each incoming frame. That is, for each incoming frame, features are extracted and explicitly matched, and then reconstruction is performed. As another example, features are extracted in one frame and tracked in subsequent frames (i.e., locating the features in the subsequent frames does not involve feature extraction). Further, a dense tracking approach can be used, where tracking is performed on most or all pixels in the image. Tracking some or all pixels provides a dense scene reconstruction. Yet another approach is a deep learning approach, such as using a neural radiance field to perform feature extraction, matching, and reconstruction simultaneously in an implicit manner. For all of these approaches, the known camera motion is used to constrain the problem, and a 3D model of the bone can additionally be used to provide further constraints. These solutions can not only provide a reconstruction of the scene, but also provide differential information associated with each reconstructed point (e.g., a normal vector).

[0066] Unlike related techniques that rely on feature extraction and matching for structure and camera motion retrieval from motion, various examples utilize the pose of the bone fiducial 428 in each frame to resolve the motion aspect. The structure of the bone fiducial 428 can be used directly—using known or predetermined features of the bone fiducial 428, including visible fiducials. In other cases, the surgical controller 418 can read the visible fiducials, and based on that reading, obtain dimensional information directly from the fiducials or by accessing a local or internet-based database.

[0067] As noted above, the bone in the limited field of view of the arthroscope 408 is low-textured, and for many patients, the bone has a smoothly varying character. While in some cases, the example non-contact registration technique operates directly, in other cases, one or more additional steps can be included to ensure that a sufficient sample set of surface features on the bone are identified. The additional steps can be conceptually divided into an illumination strategy, applying a coloration to the bone, and creating an optical texture. Each will be described in turn.

[0068] Bone is a matrix of calcium and collagen, with collagen acting as a mechanical support for bone cells. Calcium and collagen can have different absorption and / or reflectance coefficients depending on the wavelength of the photons incident on the surface of the bone. In some cases, surface features of the bone can be highlighted by illuminating the bone with photons having a wavelength outside the visible range and receiving a video image based on the illuminating photons accordingly. More specifically, in one example, the bone can be illuminated by light in the near-infrared range (e.g., wavelengths between and including 780 nanometers (nm) and 2500 nm). In other cases, the bone can be illuminated with light having wavelengths between and including 488 nm and 620 nm. The difference in absorption and / or reflectance of collagen and calcium in the matrix can make optical textures in the video image more prominent. Frames of the video image can be used directly by the surgical controller 418 to resolve motion and identify surface features. That is, the image capture array of the camera 410 of the camera 410 of the cameras can be selected to be responsive to photons in the near-infrared range, and the surgical controller 418 can resolve motion of the arthroscope 408 and the camera 410 and identify surface features based on near-infrared photons reflected or emitted from the bone.

[0069] A second conceptual additional step is to apply a surface tint to the bone prior to identifying surface features. In particular, in these examples, a chemical compound is applied to the bone to add optical textures to the bone within the video image. In other words, in these examples, a chemical compound is applied to the bone to highlight optical textures of the bone. In one example, a biocompatible dye can be applied to the surface of the bone. The dye can take any suitable form, such as a dye similar to a food coloring or a butcher’s ink. Thus, the dye imparts a color to the bone, such as a blue color, which makes it easier for the surgical controller 418 to identify surface features. The dye can be placed on large areas of the surface area of the bone visible in the video image (e.g., covering 50% or more of the surface area, or 75% or more of the surface area). In other cases, the dye can be placed on the surface of the bone in periodic or irregularly placed dots or spots.

[0070] As another example of coloring, a fluorescent protein, such as green fluorescent protein (GRFP), can be applied to the surface of the bone. Green fluorescent protein is a protein that emits bright green fluorescence in the presence of ultraviolet light. Proteins that emit fluorescence in other colors are now or will become available. Like the dye, the green fluorescent protein imparts a color to the bone, here green, that makes it easier for the surgical controller 418 to identify surface features. The fluorescent protein can be placed on large areas of the surface area of the bone that are visible in the video image (e.g., covering 50% or more of the surface area, or covering 75% or more of the surface area). In other cases, the green fluorescent protein can be placed on the surface of the bone in periodic or irregularly placed dots or spots. Thus, this example combines the application of an illumination strategy with coloring to achieve the goal of identifying surface features. That is, the light source of the arthroscope can be designed and constructed to produce ultraviolet light, and the combined ultraviolet and visible light provided to the arthroscope 408 and the surgical site.

[0071] A third conceptual additional step that aids in identifying surface features is to create optically textured regions on the surface of the bone. That is, in many cases, the bone is smooth and has smoothly varying features. The additional step in these examples involves adding an optical texture that aids in identifying surface features by adding an optical texture, but that does not significantly impact the structural integrity of the bone. In one example, the optical texture can be added to non-load bearing locations. In the example of an ACL repair, the optical texture can be added on the inner surface of the intercondylar notch, the inner surface of the lateral condyle (e.g., above the lateral meniscus 112), and / or the inner surface of the medial condyle (e.g., above the medial meniscus 114).

[0072] Adding an optical texture can take many forms. In one example, the optical texture can be added by a mechanical cutting instrument with a rotating cutting element or burr, but where the burr is designed and constructed to limit the amount of bone removed. For example, the burr can be designed and constructed to mill or create grooves that are no more than 0.1 millimeters (mm) in depth. In other cases, the optical texture can be created by a counterbore drill bit that is designed and constructed to limit the depth of the drill hole, such as by a shoulder region that limits the depth of the drill hole to no more than 0.1 mm. For example, the surgeon can place a plurality of countersinks in the bone along the non-load bearing surface. Further, the surgeon can use any suitable instrument, such as the aiming device 426 Figure 4 ) to perform mechanical abrasion of the surface of the bone, such as pulling the aiming device along the surface of the bone and thereby creating surface indentations.

[0073] The examples discussed so far for non-contact registration techniques separate the parsing of camera motion relative to the bone fiducials from the identification of surface features. In other cases, however, the parsing and identification can be an integrated programming approach in which the bone fiducials are found in each frame, the motion is parsed frame-by-frame, and the identification of surface features occurs substantially simultaneously with the parsing of the frame-by-frame motion.

[0074] The description now turns to a third technique for registering a bone model to a bone, namely a patient-specific instrument. In the contact-based registration techniques and non-contact registration techniques discussed so far, a registration model is created and the registration model is used to register the bone model to the bone visible in the video images. Conceptually, the registration model is used to determine the coordinate transformation and scaling to align the bone model with the actual bone. If, however, the orientation of the bone in the video images is known or can be determined, then the use of a registration model can be omitted and the coordinate transformation and scaling can be calculated directly.

[0075] Figure 10 A method according to at least some embodiments is shown. The example method can be implemented in software within one or more computer systems, such as the surgical controller 418, in part. In particular, the example method begins (block 1000) and includes obtaining a three-dimensional bone model (block 1002). Much like the previous techniques, in the patient-specific instrument registration technique, what is obtained is a three-dimensional bone model that can be created by segmenting a plurality of non-invasive images (e.g., MRI) taken pre-operatively or intra-operatively.

[0076] The next step in the example method is to generate a patient-specific instrument that has features designed to couple to a bone represented in the bone model in only one orientation (block 1004). Generating the patient-specific instrument can first involve selecting a location at which the patient-specific instrument is to be attached. For example, a device or computer system can analyze the bone model and select the attachment location. In various examples, the attachment location can be a unique location in the sense that the patient-specific instrument will not couple to the bone at any other location if the patient-specific instrument is coupled to the unique location. In the example case of an ACL repair, the selected location can be at or near the superior or superior portion of the intercondylar notch. If the bone model shows another location with a unique feature, such as a bone spur or other raised or recessed surface anomaly, then such unique location can be selected as the attachment location for the patient-specific instrument.

[0077] Further, forming the patient-specific instrument can take any suitable form. In one example, the device or computer system can directly print the patient-specific instrument, e.g., using a 3D printer. In other cases, the device or computer system can print a model of the attachment location, which model can then serve as a mold for creating the patient-specific instrument. For example, the model can be a mold for injection molding plastic or casting techniques. In some examples, the patient-specific instrument carries one or more fiducials, but as noted above, in other cases the patient-specific instrument itself can be tracked, and thus does not carry fiducials.

[0078] A next step in the example method is coupling the patient-specific instrument to the bone, in some cases with the patient-specific instrument having a fiducial coupled to the outer surface (block 1006). As noted previously, the attachment location of the patient-specific instrument is chosen to be unique, such that the patient-specific instrument is coupled to the bone in only one location and in only one orientation. In the example case of arthroscopic ACL repair, the patient-specific instrument can be inserted under arthroscopy. That is, the attachment location can be chosen such that the physical dimensions of the patient-specific instrument enable insertion through a port in the patient's skin. In other cases, the patient-specific instrument can be made or constituted of a flexible material that enables the patient-specific instrument to be deformed for insertion into the surgical site, but return to a predetermined shape for coupling to the attachment location. However, in open surgery where the skin is cut and pulled open, exposing the bone to open air, the patient-specific instrument can be a rigid device with fewer size constraints.

[0079] A next step in the example method is capturing video images of the patient-specific instrument (block 1008). Here again, the capturing can be performed intraoperatively. In the example case of arthroscopic anterior cruciate ligament repair, the capturing of the video images is performed by the surgical controller 418 by way of the arthroscope 408 and the camera 410. However, in open surgery where the skin is cut and pulled open, exposing the bone to open air, the capturing can be performed by any suitable camera device (e.g., one or both cameras of a stereo camera system) or portable computing device (e.g., a tablet or smartphone device). In such cases, the video images can be provided to the surgical controller 418 in any suitable form.

[0080] The next step in the example method is to register the bone model based on the position of the patient-specific instrument (block 1010). That is, given that the patient-specific instrument is coupled to the bone in only one position and in only one orientation, the position and orientation of the patient-specific instrument is directly related to the position and orientation of the bone, so the coordinate transformation and scaling for registration can be directly calculated. Thereafter, the example method can end (block 1012); however, the surgical controller 418 can then use the registered bone model to provide computer-assisted navigation regarding the surgical task or surgical procedure involving the bone.

[0081] For example, using the registered bone model, the surgical controller 418 can provide guidance regarding the surgical task of the surgical procedure. The specific guidance depends on the surgical procedure being performed and the stage of the surgical procedure. A non-exhaustive list of guidance includes: changing the drill path entry point; changing the drill path exit point; aligning an aiming device along the planned drill path; displaying the location of the cut and / or resected bone; reaming the bone a certain depth along a certain direction; placing a device (suture, anchor, or other) at a certain location; placing a suture at a certain location; placing an anchor at a certain location; displaying areas of bone to contact and / or avoid; and identifying areas and / or landmarks of the anatomy. In still other cases, the guidance can include: highlighting within a version of the video image displayed on a display device, which can be an arthroscopic display or a fluoroscopic display; or communicating with a virtual reality device or a robotic tool.

[0082] Figure 11 A front or anterior elevation view of the right knee is shown, with the patella removed, and with an example patient-specific instrument installed. In particular, a patient-specific instrument 1102 is shown installed in the intercondylar notch 600 of the femur 100. Figure 1 The lower portion of the femur 100 is visible in

[0083] In the example of Figure 11 installed within the intercondylar notch 600 is a patient-specific instrument 1102. In particular, Figure 11 The patient-specific instrument 1102 of

[0084] In the example of Figure 11In the example of FIG. 11, the patient-specific instrument 1102 includes a feature 1104 that is designed and configured to couple to the bone in only one orientation. Since the example patient-specific instrument 1102 is shown installed, only a portion of the feature 1104 is visible in Figure 11 FIG. 11. However, the feature 1104 is actually a negative image of the bone at the attachment location.

[0085] The example patient-specific instrument 1102 also includes a fiducial 1106 disposed on or coupled to an outer surface of the patient-specific instrument 1102. The example fiducial 1106 is shown in the form of a quick response (QR) code, but any machine-readable code (e.g., one-dimensional barcode, two-dimensional barcode) can be used. While the example patient-specific instrument 1102 contains only a single fiducial, in other cases, the patient-specific instrument 1102 can contain two or more fiducials disposed on respective faces. In still other cases, the patient-specific instrument itself can be tracked and thus does not carry a fiducial.

[0086] Figure 12 An example computer system 1200 is shown. In one example, the computer system 1200 can correspond to the surgical controller 418, a device that creates patient-specific instruments, a tablet device within the operating room, or any other system that implements any or all of the various methods discussed in this specification. The computer system 1200 can be connected (e.g., networked) to other computer systems in a local area network (LAN), an intranet, and / or an extranet (e.g., device cart 402 network), or to an internet, in some cases. The computer system 1200 can be a server, a personal computer (PC), a tablet computer, or any device that is capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while a single computer system is shown, the term “computer” shall also be taken to include any collection of computer processes or

[0087] The computer system 1200 includes a processing device 1202, a main memory 1204 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory 1206 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 1208, which communicate with each other via a bus 1210.

[0088] The processing device 1202 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, or the like. More particularly, the processing device 1202 can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processing device 1202 can also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 1202 is configured to execute instructions for performing any operations and steps discussed herein. The processing device 1202, and more generally the computer system 1200, becomes a specially programmed device in that once programmed with particular instructions, the processing device 1202 and thus the entire computer system 1200 becomes a specially programmed device, such as the surgical controller 418.

[0089] The computer system 1200 can also include a network interface device 1212 for communicating with any suitable network, such as the device cart 402 network. The computer system 1200 can also include a video display 1214 (e.g., the display device 414), one or more input devices 1216 (e.g., microphone, keyboard, and / or mouse), and one or more speakers 1218. In one illustrative example, the video display 1214 and the input device(s) 1216 can be combined into a single component or device (e.g., an LCD touch screen).

[0090] The data storage device 1208 can include a computer-readable storage medium 1220 on which is stored instructions 1222 embodying any one or more of the methodologies or functions described herein (e.g., implementing any of the methods and any of the functions performed by any of the devices and / or components described herein). The instructions 1222 can also reside completely, or at least partially, within the main memory 1204 and / or within the processing device 1202 during execution thereof by the computer system 1200. Thus, the main memory 1204 and the processing device 1202 also constitute computer-readable media that can store instructions 1222. In some cases, the instructions 1222 can be received, sent, or accessed via a network by the network interface device 1212.

[0091] While the computer-readable storage medium 1220 is shown in an illustrative example as a single medium, the term“computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more sets of instructions. The term“computer-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by a machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term“computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0092] The above discussion is meant to be illustrative of the principles and various embodiments of the present application. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be construed to cover all such variations and modifications.

Claims

1. A method of registering a bone model, the method comprising: obtaining, by a surgical controller, the bone model; capturing, by the surgical controller, a video image of a bone fiducial affixed to a bone, the capturing by way of a camera that is moving; resolving, by the surgical controller, motion of the camera relative to the bone fiducial based on the video image; identifying, by the surgical controller, a plurality of surface features on the bone in the video image based on the motion of the camera; generating, by the surgical controller, a registration model of the bone relative to the bone fiducial; and registering, by the surgical controller, the bone model relative to the bone fiducial based on the registration model.

2. The method of claim 1, further comprising providing, by the surgical controller, navigation information regarding a procedure involving the bone.

3. The method of claim 1, wherein identifying a plurality of surface features on the bone further comprises illuminating the bone with a sub-photonic wavelength that highlights optical texture.

4. The method of claim 3, wherein illuminating the bone with a sub-photonic wavelength that highlights optical texture further comprises illuminating with a light having a wavelength between and including 780 nanometers (nm) and 2500 nm.

5. The method of claim 3, wherein illuminating the bone with a sub-photonic wavelength that highlights optical texture further comprises illuminating the bone with a wavelength between and including 488 nanometers (nm) and 620 nm.

6. The method of claim 3, wherein illuminating the bone with a sub-photonic wavelength that highlights optical texture further comprises illuminating the bone with a wavelength for which the reflectivity of collagen is higher than the reflectivity of calcium.

7. The method of claim 1, further comprising applying a surface stain to the bone prior to identifying the plurality of surface features.

8. The method of claim 7, wherein applying the surface stain further comprises applying a dye to the bone.

9. The method of claim 7, wherein applying the surface stain further comprises applying a green fluorescent protein to the bone.

10. The method of claim 1, further comprising creating a textured region on the bone prior to identifying the plurality of surface features.

11. The method of claim 10, wherein creating the textured region further comprises creating the textured region at a non-load bearing location.

12. The method of claim 11, wherein creating the textured region at a non-load bearing location further comprises creating the textured region in an intercondylar notch for an anterior cruciate ligament procedure.

13. The method of claim 10, wherein creating the textured region further comprises performing mechanical abrasion.

14. The method of claim 1, wherein identifying the plurality of surface features further comprises applying the video image and an indication of the motion of the camera to a machine learning algorithm trained to identify optical texture of bone.

15. The method of claim 1, wherein the camera is affixed to a rigid endoscope. ​ 16. The method of claim 15, wherein the endoscope is monoscopic.

17. The method of claim 1, wherein obtaining the bone model further comprises: segmenting a plurality of non-invasive images taken preoperatively or intraoperatively; and generating the bone model based on the segmentation.

18. A surgical controller comprising: a processor; a memory communicatively coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to: receive a bone model; capture video images of a bone fiducial affixed to a bone, the video images being captured while a camera is moving; resolve motion of the camera relative to the bone fiducial based on the video images; identify a plurality of surface features on the bone in the video images based on the motion of the camera; generate a registration model of the bone relative to the bone fiducial; and register the bone model relative to the bone fiducial based on the registration model.

19. The surgical controller of claim 18, wherein the instructions further cause the processor to provide navigation information regarding a procedure involving the bone.

20. The surgical controller of claim 18, wherein when the processor identifies the plurality of surface features on the bone, the instructions further cause the processor to instruct a light source to illuminate the bone with a light wavelength that highlights optical textures.

21. The surgical controller of claim 20, wherein when the processor instructs the light source to illuminate the bone with a light wavelength that highlights optical textures, the instructions cause the processor to instruct the light source to illuminate with a light having a wavelength between and including 780 nanometers (nm) and 2500 nm.

22. The surgical controller of claim 20, wherein when the processor instructs the light source to illuminate the bone with a light wavelength that highlights optical textures, the instructions cause the processor to instruct the light source to illuminate the bone with a light wavelength for which the reflectivity of calcium is higher than the reflectivity of collagen.

23. The surgical controller of claim 20, wherein when the processor instructs the light source to illuminate the bone with a light wavelength that highlights optical textures, the instructions cause the processor to instruct the light source to illuminate the bone with a light wavelength for which the reflectivity of collagen is higher than the reflectivity of calcium.

24. The surgical controller of claim 18, wherein when the processor identifies the plurality of surface features, the instructions cause the processor to apply the video images and an indication of the motion of the camera to a machine learning algorithm trained to identify optical textures of bone.

25. A method of registering a bone model, the method comprising: obtaining, by a surgical controller, the bone model; capturing, by the surgical controller, video images of a bone fiducial affixed to a bone; determining, by the surgical controller, a location of a distal tip of a medical instrument visible within the video images, the distal tip contacting the bone in at least some frames of the video images, and the medical instrument not having a fiducial; ​ generating, by the surgical controller, a registration model of the bone relative to the bone fiducial based on the position of the distal tip of the medical instrument; and registering, by the surgical controller, the bone model relative to the bone fiducial based on the registration model.

26. The method of claim 25, further comprising providing, by the surgical controller, navigation information regarding a procedure involving the bone.

27. The method of claim 25, wherein determining the position of the distal tip of the medical instrument further comprises: segmenting, by the surgical controller, the medical instrument in the video images; estimating, by the surgical controller, a plurality of poses of the medical instrument within a respective plurality of frames of the video images; and determining the position based on the plurality of poses.

28. The method of claim 27, wherein estimating the plurality of poses further comprises applying the video images to a machine learning algorithm trained to perform six- dimensional pose estimation.

29. The method of claim 28, wherein applying to a machine learning algorithm trained to perform six-dimensional pose estimation further comprises applying the video images to a convolutional neural network trained to perform the six-dimensional pose estimation.

30. The method of claim 25, wherein the medical instrument is at least one selected from the group consisting of: a contact probe; a scope; and a drill guide.

31. The method of claim 25, wherein the camera is fixed to a rigid endoscope.

32. The method of claim 31, wherein the endoscope is monopurpose.

33. The method of claim 25, wherein obtaining the bone model further comprises: segmenting a plurality of non-invasive images taken preoperatively or intraoperatively; and generating the bone model based on the segmenting.

34. A surgical controller, comprising: a processor; a memory communicatively coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to: obtain a bone model; capture video images of a bone fiducial fixed to a bone; determine a position of a distal tip of a medical instrument visible within the video images, the distal tip contacting the bone in at least some frames of the video images, and the medical instrument not having a fiducial; generate a registration model of the bone relative to the bone fiducial based on the position of the distal tip of the medical instrument; and register the bone model relative to the bone fiducial based on the registration model.

35. The surgical controller of claim 34, wherein the instructions further cause the processor to provide navigation information regarding a procedure involving the bone.

36. The surgical controller of claim 34, wherein when the processor determines the position of the distal tip of the medical instrument, the instructions cause the processor to: segment the medical instrument in the video images; estimate a plurality of poses of the medical instrument within a respective plurality of frames of the video images; and determine the position based on the plurality of poses.

37. The surgical controller of claim 36, wherein when the processor estimates the plurality of poses, the instructions cause the processor to apply the video images to a machine learning model trained to perform six-dimensional pose estimation.

38. The surgical controller of claim 37, wherein when the processor applies the video images to the machine learning model, the instructions cause the processor to apply the video images to a convolutional neural network trained to perform six-dimensional pose estimation.

39. The surgical controller of claim 34, wherein the medical instrument in the video images is at least one selected from the group consisting of: a contact probe; a scope; and a drill guide.

40. A method of registering a bone model, the method comprising: obtaining, by a device, the bone model; generating, by the device, a patient-specific instrument having a feature configured to couple to a bone represented in the bone model in only one orientation; coupling the patient-specific instrument to the bone; and then capturing, by a surgical controller, a video image of the patient-specific instrument, the capturing being by way of a camera; and registering, by the surgical controller, the bone model based on a position of the patient-specific instrument.

41. The method of claim 40, further comprising directing, by the surgical controller, a surgical task of a surgical procedure, the directing being based on the bone model after registration.

42. The method of claim 41, wherein directing the surgical task further comprises at least one selected from the group consisting of: changing a drill path entry point; changing a drill path exit point; aligning a scope along a planned drill path; displaying a location to cut the bone; and highlighting an area of the bone to avoid within a version of a video image displayed on a display device.

43. The method of claim 41, wherein the surgical procedure is an open surgery.

44. The method of claim 41, wherein the surgical procedure is an arthroscopic surgical procedure.

45. The method of claim 44, wherein the camera is fixed to a rigid arthroscope.

46. The method of claim 45, wherein the arthroscope is monopurpose.

47. The method of claim 44, wherein the arthroscopic surgical procedure is to a joint selected from the group consisting of: knee; hip; shoulder; wrist; and ankle.

48. The method of claim 44, wherein the arthroscopic surgical procedure is at least one selected from the group consisting of: anterior cruciate ligament repair; reduction of femoral acetabular impingement; and repair of rotator cuff.

49. The method of claim 40, wherein obtaining the bone model further comprises: segmenting a plurality of non-invasive images taken preoperatively or intraoperatively; and generating the bone model based on the segmenting.

50. A surgical controller comprising: a processor; a memory communicatively coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to: obtain a bone model; capture a video image of a fiducial of a patient-specific instrument, the capturing being by way of a camera; and ​ registering the bone model based on the position of the patient-specific instrument by the surgical controller.

51. The surgical controller of claim 50, wherein the instructions further cause the processor to guide a surgical task of a surgical procedure, the guiding based on the bone model after registration.

52. The surgical controller of claim 51, wherein the guiding is at least one selected from the group consisting of: changing a drill path entry point; changing a drill path exit point; aligning an aiming device along a planned drill path; displaying a location to cut and / or resect the bone; reaming the bone a certain depth along a certain direction; placing a device (suture, anchor, or other) at a certain location; placing a suture at a certain location; placing an anchor at a certain location; displaying an area of the bone to contact and / or avoid; and identifying an area and / or landmark of an anatomical structure.

53. The surgical controller of claim 52, wherein the guiding is provided by highlighting within a version of a video image displayed on a display device, the display device can be an arthroscopic display or a fluoroscopic display, or the guiding is provided by communicating with a virtual reality device or a robotic tool.

54. The surgical controller of claim 51, wherein the guiding is at least one selected from the group consisting of: changing a drill path entry point; changing a drill path exit point; aligning an aiming device along a planned drill path; displaying a location to cut the bone; and highlighting within a version of a video image displayed on a display device an area of the bone to avoid.

55. The surgical controller of claim 52, wherein the surgical procedure is an open surgical procedure.

56. The surgical controller of claim 52, wherein the surgical procedure is an arthroscopic surgical procedure.

57. The surgical controller of claim 52, wherein the surgical procedure is a procedure selected from the group consisting of: knee; hip; shoulder; wrist; and ankle.

58. The surgical controller of claim 52, wherein the surgical procedure is at least one selected from the group consisting of: anterior cruciate ligament repair; reduction of femoral acetabular impingement; and repair of rotator cuff.

Images