Method and system for registering a three-dimensional bone model

Abstract

The purpose is to register a three-dimensional bone model. Various embodiments relate to methods and associated systems for identifying surface features of a rigid structure visible in a video stream and using these surface features to register a three-dimensional model for use in computer-aided navigation of surgical procedures. In some embodiments, the surface features are determined using touchless technology based on known or calculated camera movements. In other embodiments, the surface features are collected using a touch probe that is not directly tracked itself. In further embodiments, the three-dimensional model may be registered by the use of a patient-specific instrument that is coupled to the rigid structure in only one orientation.

Description

Technical Field

[0001] Cross - reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 457,908, filed on April 7, 2023, entitled "METHODS AND SYSTEMS OF REGISTERING A THREE - DIMENSIONAL BONE MODEL". This provisional application is incorporated herein by reference as if the full text thereof were reproduced below.

Background Art

[0003] Arthroscopic surgical procedures are minimally invasive surgical procedures that access surgical sites within the body through small keyholes, or ports, that pass through the patient's skin. Various tissues within the surgical site are visualized by an arthroscope placed through the port, and the internal view is displayed on an external display device. Tissues can be repaired or replaced through the same port or through additional ports. In computer - assisted surgical procedures (e.g., anterior cruciate ligament (ACL) replacement, reduction of femoroacetabular impingement), the positions of various objects with respect to the surgical site can be tracked relative to the bone by images captured by the arthroscope and by a three - dimensional bone model.

[0004] For a detailed description of exemplary embodiments, reference is now made to the accompanying drawings.

Brief Description of the Drawings

[0005] [Figure 1] FIG. 1 shows a front view, i.e., a front cross - sectional view, of the right knee with the patella removed. [Figure 2] FIG. 2 shows a rear view, i.e., a rear cross - sectional view, of the right knee. [Figure 3] FIG. 3 shows a view of the intercondylar notch of the femur looking up from below. [Figure 4]Figure 4 shows a surgical system according to at least some embodiments. [Figure 5] Figure 5 shows a conceptual diagram of a surgical site according to at least some embodiments, in which various objects within the surgical site are tracked. [Figure 6] Figure 6 is an exemplary video display according to at least some embodiments, showing a portion of the femur and having a visible bone reference inside. [Figure 7] Figure 7 shows a method relating to at least some embodiments. [Figure 8] Figure 8 is an exemplary video display relating to at least some embodiments, showing a portion of the femur and bone reference during the registration procedure. [Figure 9] Figure 9 shows a method relating to at least some embodiments. [Figure 10] Figure 10 shows a method relating to at least some embodiments. [Figure 11] Figure 11 shows a frontal or frontal cross-sectional view of the right knee, in which the patella has been removed and a patient-specific device has been placed, according to at least some embodiments. [Figure 12] Figure 12 shows a computer system according to at least some embodiments. [Modes for carrying out the invention]

[0006] definition Various terms are used to refer to specific system components. Different companies may use different names to refer to the same component, but this document is not intended to distinguish between components that have different names but no difference in function. In the following discussion and claims, the terms “includes” and “equipped with” are used in an open-ended manner and should therefore be interpreted as “includes, but not limited to.” Also, the terms “couple” or “couples” are intended to mean either an indirect or direct connection. Thus, when a first device is coupled to a second device, that connection may be a direct connection or an indirect connection via other devices or other connections.

[0007] An endoscope having a "single optical path" means that the endoscope is not a stereoscopic endoscope having two separate optical paths separated by the interpupillary distance at the focusing end of the endoscope. The fact that an endoscope has two or more optical components (e.g., glass rods, optical fibers) that form a single optical path does not negate the state of having a single optical path.

[0008] Detailed description of the invention The following discussion covers various embodiments of the present invention. While one or more embodiments in these embodiments may be preferred, the disclosed embodiments should not be construed as limiting the scope of the disclosure, including the claims, nor should they be used in other forms. In addition, those skilled in the art will understand that the following description has broad applications, and that any discussion of any embodiment is intended to be merely illustrative of that embodiment and not to imply that the scope of the disclosure, including the claims, is limited to that embodiment.

[0009] Various examples relate to methods and systems for registering three-dimensional models of rigid structures such as bone. More specifically, various embodiments relate to methods and related systems for identifying surface features of a rigid structure visible in a video stream and using these surface features to register a three-dimensional model for use in computer-aided navigation of surgical procedures. In some embodiments, surface features are determined using touchless techniques based on known or calculated camera movements. In other embodiments, surface features are collected using a touch probe, rather than being directly tracked itself, where the pose of the touch probe, and therefore the position of the distal tip of the touch probe in contact with the bone, can be determined by segmenting the video stream frames and pose estimation. In further embodiments, the three-dimensional model may also be registered by the use of a patient-specific instrument coupled to the rigid structure in only one orientation, and thus a three-dimensional bone model can be registered using a criterion coupled to the patient-specific instrument, or in some cases, without a criterion, using the patient-specific instrument itself.

[0010] The various embodiments were developed in the context of anterior cruciate ligament (ACL) repair, and therefore the following discussion is based on this developmental context. In this context, the rigid structure is bone, and the three-dimensional mode is a three-dimensional bone model. However, the technique is applicable to any suitable rigid anatomical structure, such as teeth. Furthermore, the various techniques may be applicable to many types of surgical procedures, such as repairs related to the knee, hip, shoulder, wrist, or ankle. The technique is applicable not only to ligament repairs (e.g., medial collateral ligament repair, lateral collateral ligament repair, and posterior cruciate ligament repair), but also to surgical procedures for planning and positioning anchors to reattach soft tissues (e.g., reattaching the labrum, rotator cuff, or meniscal root of the hip), and to address femoroacetabular impingement. Therefore, this specification and the circumstances leading up to its development should not be construed as limiting the applicability of this teaching. To direct the reader, this specification begins with a description of the knee.

[0011] Figure 1 shows an anterior view, or frontal cross-section, of the right knee with the patella removed. Particularly visible in Figure 1 is the lower portion of the femur 100, including the lateral condyle, or lateral condyle 102, and the medial condyle, or medial condyle 104. The femur 100 and the condyles 102 and 104 are in a working relationship with the tibia 106, which includes the tibial tuberosity 108 and Geldi's tubercle 110. The lateral meniscus 112 and medial meniscus 114 are located between the femoral condyles 102 and 104 and the tibia 106. Several ligaments, such as the ACL 116, which extends from the lateral side of the femoral notch to the medial side of the tibia 106, are visible in Figure 1. Conversely, the posterior cruciate ligament 118 extends from the medial side of the femoral notch to the tibia 106. Additionally, fibula 120 and several other ligaments that are not specifically labeled are also visible.

[0012] Figure 2 shows a posterior view, or dorsal cross-section, of the right knee. In particular, what is visible in Figure 2 is the lower portion of the femur 100, including the lateral condyle 102 and the medial condyle 104. The femur 100 and the femoral condyles 102 and 104 are in a working relationship with the tibia 106, and the lateral meniscus 112 and the medial meniscus 114 are located between the femoral condyles 102 and 104 and the tibia 106. Figure 2 further shows the ACL 116 extending from the lateral side of the femoral notch to the medial side of the tibia 106, but its attachment point to the tibia 106 is not visible. The posterior cruciate ligament 118 extends from the medial side of the femoral notch to the tibia 106, but its attachment point to the femur 100 is not visible. In this case as well, several additional ligaments, which are not specifically labeled, are shown.

[0013] The most common type of ACL injury is a complete rupture of the ligament. Treatment involves reconstructing the ACL by placing an alternative graft (e.g., an autologous graft from either the patellar tendon, quadriceps tendon, or hamstring tendon). The graft is placed inside tunnels prepared within the femur 100 and tibia 106. Current standard treatment for ACL repair involves positioning the tunnel so that the tunnel entry point for the graft is at the anatomical attachment site of the natural ACL. Such tunnel placement to the natural ACL attachment site attempts to replicate the original kinematic characteristics of the knee. In arthroscopic surgery, the tunnel location through the tibia 106 is relatively easy to access, especially when bending or flexing the knee. However, the tunnel through the femur 100 lies within the intercondylar notch. Depending on the patient's body size and how the surgeon selects the location of the port through the skin, reaching the natural ACL attachment site through the femur 100 can sometimes be difficult.

[0014] Figure 3 shows a view of the intercondylar notch of the femur from below. In particular, the lateral condyle 102 and medial condyle 104 are visible in Figure 3. The femoral notch 200 is defined between the femoral condyles 102 and 104. The femoral tunnel may define a medial opening 202 within the femoral notch 200, the medial opening 202 being close to the lateral condyle 102 and displaced in the posterior portion of the femoral notch 200. The femoral tunnel extends through the femur 100 and forms a lateral opening on the outer or lateral surface of the femur 100 (the lateral opening is not visible in Figure 3). Figure 3 shows an exemplary drill wire 204 that may be used to create an initial tunnel, i.e., a pilot hole. After the surgeon has confirmed that the pilot hole is precisely aligned with the planned tunnel path, the femoral tunnel is created by drilling or reaming using another instrument (e.g., a reamer) that can use the drill wire 204 as a guide. In some cases, a socket or counterbore is created in the intercondylar notch to accommodate the width of the graft extending into the bone, and this counterbore may also be created using another instrument (e.g., a reamer) that can use the drill wire 204 as a guide.

[0015] Figure 4 shows surgical systems (not to exact scale) according to at least some embodiments. In particular, exemplary surgical system 400 includes a tower or device cart 402, exemplary mechanical excision instruments 404, exemplary plasma-based ablation instruments (hereinafter simply referred to as ablation instruments 406), and exemplary endoscopes consisting of arthroscopy 408 and an attached camera head or camera 410. In exemplary systems, arthroscopy 408 is a rigid device, unlike endoscopes for other procedures such as upper endoscopy. Device cart 402 may include a display device 414, an excision controller 416, and a camera control unit (CCU), together with an endoscopic light source and a video controller 418. In exemplary cases, the combined CCU and video controller 418 not only provide light to arthroscopy 408 and display images received from camera 410, but also implement various additional embodiments such as registering a three-dimensional bone model to bones visible in video images and providing computer-assisted navigation during surgery. Therefore, hereafter, the combined CCU and video controller will be referred to as the surgical controller 418. However, in other cases, the CCU and video controller may be separate systems distinct from the controllers handling registration and computer-assisted navigation, although these separate devices will still be functionally combined.

[0016] The exemplary device cart 402 further includes a pump controller 422 (e.g., a single or dual peristaltic pump). Fluid connections of the mechanical excision instrument 404 and the ablation instrument 406 to the pump controller 422 are not shown to avoid overcomplicating the drawings. Similarly, fluid connections between the pump controller 422 and the patient are not shown to avoid overcomplicating the drawings. In the exemplary system, both the mechanical excision instrument 404 and the ablation instrument 406 are coupled to an excision controller 416, which is a dual-function controller. However, in other examples, a mechanical excision controller may be provided, separated and distinct from the ablation controller. The exemplary devices and controllers associated with the device cart 402 are illustrative only, and other embodiments include a vacuum pump, a patient positioning system, a robotic arm holding various instruments, an ultrasonic cutting device, and associated controllers, a patient positioning controller, and a robotic surgical system.

[0017] Figure 4 further illustrates additional instruments that may be present during arthroscopic surgical procedures. In particular, Figure 4 shows an exemplary touch probe 424, drill guide or sight 426, and bone reference 428. The touch probe 424 may be used during the surgical procedure to provide information to the surgical controller 418, such as information for registering a three-dimensional bone model in the underlying bone, which can be visualized in the images captured by the arthroscope 408 and camera head 410. The sight 426 may be used as a guide for positioning the drill wire to generate an initial tunnel, i.e., a pilot tunnel, that penetrates the bone, and as a guide for drilling with the drill wire. The bone reference 428 may be fixed and firmly attached to the bone and may function as an anchor position for the surgical controller 418 to sense the orientation of the bone (e.g., after registration to the three-dimensional bone model). Additional tools and instruments will be present, such as drill wires, various reamers for creating through-hole and counterbore configurations for tunnels penetrating the bone, and various tools for suturing and anchoring the graft. These additional tools and instruments are not shown to avoid further complicating the drawings. This specification now moves on to describing the workflow for an exemplary anterior cruciate ligament repair.

[0018] Surgical procedures can begin at the planning stage. An exemplary anterior cruciate ligament repair can begin with imaging (e.g., X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI)) of the patient's knee, including relevant anatomical structures such as the lower portion of the femur, the upper portion of the tibia, and articular cartilage. The imaging may be performed preoperatively, hours or days before the intraoperative repair, or the imaging may be performed within the surgical setting immediately prior to the intraoperative repair. In the following description, MRI is assumed, but again, many different types of imaging can be used in this case. Image slices from the MRI imaging may be segmented, thereby creating a volumetric model, or three-dimensional model, of the anatomical structure. Any suitable, currently available, or later developed segmentation technique can be used to create the three-dimensional model. To be more specific about an example of an anterior cruciate ligament repair, a three-dimensional bone model of the lower portion of the femur, including the femoral condyles, is generated. [[ID=!]]

[0019] Using a three-dimensional bone model, a surgical plan is developed, including selecting a planned tunnel path through the femur, including the location of a bone opening that defines the end of the tunnel. For an exemplary inside-out repair, the opening within the femoral notch is the entry point for the perforation, and the opening on the lateral surface of the femur is the exit point. For an outside-in repair, the entry and exit points for the perforation are reversed. Nevertheless, assuming an inside-out repair, the entry point may be selected to be identical to, or close to, the original anterior cruciate ligament attachment point to the femur within the femoral notch. In some cases, the selection of the entry point within the femoral notch may involve using the Bernard and Hertel Quadland method or grid placed on a fluoroscopic image, or placing the Bernard and Hertel Quadland method on a simulated fluoroscopic image generated from a three-dimensional bone model. Based on the use of the Bernard and Hertel Quadland method, the tunnel entry point is selected. In inside-out repairs, the selection of the exit site is not restricted not only because the portion of the tunnel adjacent to the exit site is used for graft anchor placement, but also because the exit site is located approximately in the center of the femur (considered to be from anterior to posterior), and therefore, concerns about the thickness of the bone wall at the exit site do not arise. In some cases, a three-dimensional bone model of the proximal end of the tibia is also generated, and the surgeon can similarly select the planned tunnel pathway(s) through the tibia.

[0020] The results of the plan may include a three-dimensional bone model relating to the distal end of the femur, a three-dimensional bone model relating to the proximal end of the tibia, the entry and exit points through the femur, and thus the planned tunnel path through the femur, and the entry and exit points through the tibia, and thus the planned tunnel path through the tibia. Other surgical parameters such as the tunnel through-hole diameter, the diameter and depth of the tunnel counterbore hole, the desired post-repair flexion, and similar parameters may also be selected during the planning stage, but these additional surgical parameters are omitted to avoid overcomplicating this specification.

[0021] In this specification, we now turn to the description of the intraoperative aspects. The intraoperative aspects include steps and procedures for setting up a surgical system to perform various repairs. However, note that some of the intraoperative aspects (e.g., calibration of the optical system) can actually be performed before the patient is brought into the operating room, before any ports or incisions are made through the patient's skin. Nevertheless, such steps and procedures are considered intraoperative because they are performed using the surgical instruments and devices used to perform the actual repair within the surgical setting.

[0022] An exemplary ACL repair is arthroscopic and is computer-assisted in that a surgical controller 418 is used for arthroscopic navigation within the surgical site. More specifically, in an exemplary system, the surgical controller 418 provides computer-assisted navigation during ligament repair by tracking the positions of various objects within the surgical site, such as the position of the bone in a three-dimensional coordinate space within the arthroscopic field of view and the positions of various instruments (e.g., drill wire) in a three-dimensional coordinate space within the arthroscopic field of view. In this specification, we turn to a brief description of such tracking techniques.

[0023] FIG. 5 shows a conceptual diagram of a surgical site in which various objects are present within the surgical site. In particular, what is visible in FIG. 5 is the distal end of the arthroscope 408, a portion of the bone 500 (e.g., femur), a bone reference 428 within the surgical site, and the touch probe 424. We will take each in turn.

[0024] The arthroscope 408 illuminates the surgical site with visible light. In the embodiment shown in Figure 5, the illumination is indicated by arrow 508. The illumination supplied to the surgical site is reflected by various objects and tissues within the surgical site, and the reflected light returning to the distal end enters the arthroscope 408, propagates along the optical channels within the arthroscope 408, and finally enters the capture array in the camera 410 (Figure 4). The image detected by the capture array in the camera 410 is electronically transmitted to the surgical controller 418 (Figure 4) and displayed on the display device 414 (Figure 4). In one embodiment, the arthroscope 408 is monocular or has a single optical path through the arthroscope for capturing images of the surgical site, even though a single optical path may be constructed from two or more optical members (e.g., glass rods, optical fibers). In other words, in the exemplary system and method, computer-assisted navigation provided by the arthroscope 408, camera 410, and surgical controller 418 is provided, and the arthroscope 408 is not a stereoscopic endoscope, having two separate optical paths separated by the interocular distance at the distal end of the endoscope.

[0025] During surgical procedures, surgeons select arthroscopes that have a field of view orientation that is beneficial for the planned surgical procedure. The field of view orientation refers to the straight line located at the center of the angle formed by the outer edge, or peripheral edge, of the endoscope's field of view. For some arthroscopes, the field of view orientation is aligned with the longitudinal central axis of the arthroscope, and such arthroscopes are referred to as "0-degree" arthroscopes (for example, the angle between the field of view orientation and the longitudinal central axis of the arthroscope is 0 degrees). For other arthroscopes, the field of view orientation forms a non-zero angle with respect to the longitudinal central axis of the arthroscope. For example, in the case of a 30° arthroscope, the field of view orientation forms a 30° angle with respect to the longitudinal central axis of the arthroscope, and this angle is measured as an obtuse angle beyond the distal end of the arthroscope. In many cases in ACL repair, surgeons select either a 30° or 45° arthroscope based on the location of the port to be created through the patient's skin. In the embodiment shown in Figure 5, the field of view angle 510 of the arthroscope 408 forms a non-zero angle with respect to the longitudinal central axis 512 of the arthroscope 408.

[0026] Referring again to Figure 5, within the field of view of the arthroscope 408, there is a portion of bone 500 (e.g., an intercondylar notch) along an exemplary bone reference 428 and an exemplary touch probe 424. The exemplary bone reference 428 is a multifaceted element, and each or more faces have references placed or generated on it. However, the bone reference does not need to have multiple faces and can actually take any shape as long as its shape can be tracked in the video image. The bone reference, such as the bone reference 428, can be attached to bone 500 in any preferred form, in this embodiment by fastening by the threaded portion of the bone reference 428 (not visible in Figure 5, but visible in Figure 4). The pattern of the bone reference on each face is designed to provide information about the orientation of the bone reference 428 in the three-dimensional coordinate space of the field of view of the arthroscope 408. More specifically, the pattern is selected so that the orientation of the bone reference 428 can be determined from images captured by the arthroscope 408 and the accompanying camera (Figure 4).

[0027] The touch probe 424 is also shown to be partially visible within the field of view of the arthroscope 408. The touch probe 424 may be used to identify multiple surface features on bone 500 as part of the registration of bone 500 into a three-dimensional bone model, as will be discussed in more detail below. Alternatively, although not specifically shown, a sighting device 426 (Figure 4) may be used as a device to assist in the registration process. In some cases, the touch probe 424 and / or sighting device 426 may carry their own unique criteria so that their respective orientations may be calculated from one or more criteria present in the video stream. However, in other cases, as shown, the medical device used to assist in the registration of the three-dimensional bone model, for example, the touch probe 424, sighting device 426, or any other suitable medical device may omit carrying a criterion. Otherwise, in such embodiments, the medical device will not have criterion markings. In such cases, the orientation of the medical device may be determined by a machine learning model, as will be discussed in more detail below.

[0028] Images captured by the arthroscope 408 and its accompanying camera are subject to optical distortion in various forms. For example, the field of view between the distal end of the arthroscope 408 and the bone 500 in the surgical site is filled with fluids such as body fluids and saline solution used to inflate the joint. Many arthroscopes have one or more lenses at the distal end to widen the field of view, which creates a "fisheye" effect in the captured image. Furthermore, optical elements within the arthroscope (e.g., rod lenses) may have optical aberrations inherent to the manufacturing and / or assembly processes. Moreover, the camera may have various optical elements for focusing the received image onto a capture array, and these various optical elements may have aberrations inherent to the manufacturing and / or assembly processes. In the exemplary system, the endoscopic optical system is calibrated to account for various optical distortions before use in each surgical procedure. Calibration generates a characterization function that characterizes the optical distortion, and further analysis of frames in the video stream may be corrected using the characterization function before further analysis.

[0029] The next exemplary step in the intraoperative procedure is the registration of the bone model generated during preoperative planning. During intraoperative repair, the three-dimensional bone model is obtained by and provided to the surgical controller 418. Again, in an example of anterior cruciate ligament repair, using specific computer-aided navigation for a tunneling pathway through the femur, the three-dimensional bone model of the lower portion of the femur is obtained by and provided to the surgical controller 418. Thus, the surgical controller 418 receives the three-dimensional bone model and, assuming that the arthroscope 408 is inserted into the knee through a port through the patient's skin, the surgical controller 418 also receives video images of the lower portion of the femur. To associate the three-dimensional bone model with the images received by the arthroscope 408 and camera 410, the surgical controller 418 registers the three-dimensional bone model with the images of the femur received by the arthroscope 408 and camera 410.

[0030] To perform registration and according to the exemplary method, bone reference 428 is attached to the femur. The placement of the bone reference is such that it is located within the field of view of the arthroscope 408, but spaced apart from the expected tunnel entrance / exit through the lateral condyle. More specifically, in the exemplary case, bone reference 428 is placed within the intercondylar notch, above the expected location of the tunnel through the lateral condyle.

[0031] Figure 6 is an exemplary frame of a video display showing a portion of the femur and a bone reference. The display may be shown, for example, on a display device 414 (Figure 4) associated with a device cart 402 (Figure 4), or in any other preferred location. In particular, visible in Figure 6 are the femoral notch or intercondylar notch 600, a portion of the lateral condyle 102, a portion of the medial condyle 104, and an exemplary bone reference 428. In some embodiments, each of the outer surfaces of the bone reference 428 has a machine-readable pattern or reference on it, and in some cases each reference is unique. In other cases, the upper surface of the bone reference 428 opposite the surface on which the threaded portion extends may have a mounting feature and therefore does not carry a reference. In yet another case, if the field of view of the arthroscope 408 is relatively constant, the bone reference may have only a single surface on which the reference is located. Regardless of the precise placement of the bone reference 428, once placed, the bone reference 428 represents a fixed position on the outer surface of the bone within the field of view of the arthroscope 408, even if the position of the arthroscope 408 is moved and altered relative to the bone reference 428. Initially, the position of the bone reference 428 relative to the three-dimensional bone model is unknown to the surgical controller 418; therefore, registration of the three-dimensional bone model is required.

[0032] For the purpose of relating or registering bones visible in video images to a three-dimensional bone model, the surgical controller 418 (Figure 4) provides or determines multiple surface features of the outer surface of the bone. Identifying surface features can take several forms, including touch-based registration using a touch probe 424 without a supported reference, touchless registration techniques in which surface features are identified after analyzing the movement of the camera relative to the arthroscope 408 and bone reference 428, and a third technique using patient-specific instruments. Each will be discussed in turn.

[0033] In an exemplary touch-based registration, the surgeon may use the touch probe 424 (Figure 4) to make contact with multiple locations. In some cases, if a portion of the outer surface of the bone is exposed to the field of view, receiving multiple surface features of the outer surface of the bone may involve the surgeon “painting” the outer surface of the bone. “Painting” is a technical term that does not involve the application of color or pigment, but rather implies the movement of the touch probe 424 as its distal end touches the bone. In this embodiment, the touch probe 424 does not carry or has reference points visible to the arthroscope 408 and camera 410. Therefore, the orientation of the touch probe 424 and the position of its distal tip must be determined in order to collect surface features for the purpose of registering a three-dimensional bone model.

[0034] Figure 7 shows a method according to at least some embodiments. The exemplary method may be implemented in software within a computer system, such as a surgical controller 418. In particular, the exemplary method includes initiating (block 700) and acquiring a three-dimensional bone model (block 702). That is, in the exemplary method, what is acquired is a three-dimensional bone model that may be generated by segmenting multiple non-invasive images (e.g., CT, MRI) taken preoperatively or intraoperatively. The three-dimensional bone model may be generated from or within images with bone segmented. The three-dimensional bone may take any preferred form, such as a computer-aided design (CAD) model, a point cloud of data points relative to an arbitrary origin, or a parametric representation of the surface expressed using analytical formulas. Thus, the three-dimensional bone model is defined relative to an origin and by any preferred orthogonal criteria.

[0035] The next step in the exemplary method is to capture video images of the bone reference attached to the bone (block 704). The capture is performed intraoperatively. In the exemplary case of arthroscopic anterior cruciate ligament repair, the video images are captured by arthroscope 408 and camera 410. Other endoscopes may be used, such as endoscopes where the capture array is located at the distal end of the device (e.g., a chip-on-chip device). However, in open procedures where the skin is cut and pulled away and the bone is exposed to open air, the capture may be by any suitable camera device, such as one or both cameras of a stereoscopic camera system, or by a portable computing device such as a tablet or smartphone device. The video images may be provided to the surgical controller 418 in any suitable form.

[0036] The next step in the exemplary method is to determine the position of the distal tip of a medical device visible in the video image (block 706), where the distal tip is in contact with bone in at least part of the frame of the video image, and the medical device has no reference point. Determining the position of the distal tip of the medical device can take any preferred form. In one embodiment, determining the position may include segmenting the medical device within the frame of the video image (block 708). Segmentation can take any preferred form, such as applying the video image to a segmentation machine learning algorithm. The segmentation machine learning algorithm can take any preferred form, such as a neural network or convolutional neural network, trained on a training dataset showing the medical device in multiple known orientations. The segmentation machine learning algorithm may generate a segmented video image in which the medical device is identified or highlighted in some way (e.g., a box, increased brightness, other objects removed).

[0037] Using segmented video images, the exemplary method can estimate multiple poses of a medical device within multiple frames of each video image (block 710). Pose estimation can take any preferred form, such as applying the video images to a pose machine learning algorithm. The pose machine learning algorithm can take any preferred form, such as a neural network or a convolutional neural network trained to perform six-dimensional pose estimation. The result of the pose machine learning algorithm may be the estimated pose of the medical device within a reference frame of the video image and / or a reference frame provided by a bone criterion, for at least a portion of the frames of the video image. That is, the result of the pose machine learning algorithm may be multiple poses, each of which may be for at least a portion of the frames of the segmented video image. In many cases, the pose can be determined for each frame, but in other cases, it may be impossible to perform pose estimation for at least some frames due to video quality issues, such as motion blur caused by electronic shutter operation.

[0038] The next step in the exemplary method is to determine the position based on multiple postures (block 712). In particular, for each frame in which posture can be estimated, the position of the distal tip can be determined within a video image and / or a reference frame of bone reference, based on a model of the medical device. Thus, the result is a set of positions in which at least a portion represents the position on the outer surface of the bone.

[0039] Figure 7 shows an exemplary three-step process for determining the position of the distal tip of a medical device. However, the method is merely an example, and many variations are possible. For example, a single machine learning model, such as a convolutional neural network, may have many hidden layers, but can be configured and trained to perform all three steps as a single overall process. That is, the convolutional neural network may segment the medical device, perform six-dimensional pose estimation, and determine the position of the distal tip within each frame. The training dataset in such a scenario would include a dataset where each frame has been segmented, its six-dimensional pose identified, and the position of the distal tip identified. The output of the determination step 706 may be a segmented video stream, distinct from the video images captured in step 704. In such cases, later method steps may use both the segmented video stream and the video images to perform further tasks. In other cases, the positional information may be combined with the video images, such as being embedded in the video images or added as metadata to each frame of the video images.

[0040] Figure 8 is an exemplary video display showing a portion of the femur and bone references during the registration procedure. The display may be shown, for example, on a display device 414 associated with the device cart 402, or in any other preferred location. In particular, visible in the main part of the display in Figure 8 are the intercondylar notch 600, a portion of the lateral condyle 102, a portion of the medial condyle 104, and an exemplary bone reference 428. Shown in the upper right corner of the exemplary display is a depiction of the bone, which may be a bone rendering 800 generated from a three-dimensional bone model. Shown on the rendering 800 is a recommended area 802, which is a portion of the bone surface to be "painted" as part of the registration process. Shown in the lower right corner of the exemplary display is another depiction of the bone, which may also be a bone rendering 804 generated from a three-dimensional bone model. Shown on the rendering 804 are several surface features 806 on the bone model identified as part of the registration process. Further shown in the lower right corner of the exemplary display is a progress indicator 808, which indicates the progress of providing and receiving positions on the bone. The exemplary progress indicator 808 is a horizontal bar whose length is proportional to the number of positions received, but any suitable graphic or numerical display (e.g., 0% to 100%) indicating progress may also be used.

[0041] Referring to both the main display and the rendering in the lower right, when a surgeon makes contact with the outer surface of a bone in the images captured by the arthroscope 408 and camera 410, the surgical controller 418 can receive surface features on the bone and display each location as a dot or position 806 in the main display and simultaneously in the rendering shown in the lower right corner. More specifically, the exemplary surgical controller 418 overlays the display of identified surface features 806 on the display of images captured by the arthroscope 408 and camera 410, and in the illustrated exemplary case, it overlays the display of identified surface features 806 on the rendering 804 of the bone model. Furthermore, as the number of identified positions 806 increases, the surgical controller 418 also updates the progress indicator 808.

[0042] Referring further to Figure 8, despite the surgeon's diligence, not all positions identified by the surgical controller 418 based on the surgeon's movement of the touch probe 424 are valid positions on the bone surface. In the embodiment of Figure 8, when the surgeon moves the touch probe 424 from the inner surface of the lateral condyle 102 to the inner surface of the medial condyle 104, the surgical controller 418 receives several positions 810 that, based on exemplary six-dimensional pose estimation, are likely to represent positions where the distal end of the touch probe 424 is not in contact with the bone.

[0043] Returning to Figure 7, multiple surface features 806 may be registration models for the bone criterion 428, or the exemplary surgical controller 418 may generate a registration model for the bone criterion 428 (block 714). The registration model may take any preferred form, such as a computer-aided design (CAD) model or a point cloud of data points of any preferred orthogonal reference. Regardless of its form, the registration model may have fewer overall data points or fewer “structures” than a bone model generated by non-invasive computer imaging (e.g., MRI). However, the purpose of the registration model is to provide a basis for coordinate transformations and scaling used to correlate the bone model to the registration model and to the bone criterion 428. Thus, the next step in the exemplary method is to register the bone model against the location of the bone criterion based on the registration model (block 716). Registration may conceptually involve testing multiple coordinate transformations and scaling values ​​to find a correlation with a sufficiently high correlation or confidence coefficient. When a correlation with a sufficiently high confidence coefficient is found, the bone model is said to be registered at the location of the bone criterion. The exemplary registration method may then be terminated (block 718), however, the surgical controller 418 may then use the registered bone model to provide computer-assisted navigation regarding bone-related procedures.

[0044] In the embodiments discussed up to this point, bone model registration involves a touch-based registration technique using a touch probe 424 without a supported reference. However, other registration techniques are also possible, and this specification now moves on to touchless registration techniques. An exemplary touchless registration technique again depends on the placement of bone references 428, as shown in Figure 6. As previously mentioned, when the field of view direction of the arthroscope 408 is relatively constant, the bone references may have fewer faces, each having a reference. Once placed, bone references 428 represent a fixed position on the outer surface of the bone within the field of view of the arthroscope 408, even if the position of the arthroscope 408 is moved and changed relative to the bone references 428. Again, in order to associate or register bones visible in the video image with a three-dimensional bone model, the surgical controller 418 (Figure 4) determines multiple surface features of the outer surface of the bone, and in this embodiment, determining multiple surface features is based on a touchless registration technique in which the surface features are identified based on the movement of the arthroscope 408 and the camera 410 relative to the bone references 428.

[0045] Figure 9 shows an exemplary touchless registration method. The exemplary method may be implemented in software within a computer system, such as a surgical controller 418. In particular, the exemplary method includes initiating (block 900) and acquiring a three-dimensional bone model (block 902). Similar to touch-based registration techniques, in touchless registration techniques, what is acquired is a three-dimensional bone model that may be generated by segmenting multiple non-invasive images (e.g., MRI) taken preoperatively or intraoperatively.

[0046] The next step in the exemplary method is to capture video images of the bone reference applied to the bone (block 904). Here again, the capture may be performed intraoperatively. In the exemplary case of arthroscopic anterior cruciate ligament repair, the video images are captured by the arthroscope 408 and camera 410. However, in open procedures in which the skin is cut and pulled away and the bone is exposed to open air, the capture may be by any preferred camera device, such as one or both cameras of a stereoscopic camera system, or by a portable computing device, such as a tablet or smartphone device. The video images may be provided to the surgical controller 418 in any preferred form.

[0047] The next step in the exemplary method is to analyze the camera's movement relative to the bone reference based on the video images (block 906). In particular, the bone reference 428 not only represents a fixed position relative to the bone, but also provides size and scale information directly or indirectly. For example, a reference visible on each face of the bone reference may be of a predetermined size or be definite. By determining the aspect ratio and size changes of the reference in the video image frames, the surgical controller 418 can determine the movement of the arthroscope 408 and the camera 410 between frames. The camera's movement relative to the bone reference may be determined over the course of multiple frames. Otherwise, the posture of the arthroscope 408 relative to the bone reference 428 may be determined for at least a portion of the video image frames. The change in posture between frames represents the movement of the arthroscope 408 relative to the bone reference 428.

[0048] The next step in the exemplary method is to identify multiple surface features on the bones in the video image based on the camera's movement (block 908). More specifically, in the exemplary touchless registration technique, the multiple surface features are identified from frames of the video stream without the use of a medical device that physically contacts the bones in the video image. More precisely, in an embodiment of the touchless registration technique, the multiple surface features are identified from frames of the video stream without the use of a touch probe, sight, or any other medical device that physically contacts the bones in the video image. Determining the surface features can take any preferred form. In one embodiment, determining the surface features may involve applying a display of the video image and the camera's movement to a machine learning algorithm trained to recognize optical textures, such as spatial variations in brightness intensity or optical textures caused by surface features such as contours, ridges, and / or valleys, even in the absence of physical features. The machine learning algorithm can take any preferred form, such as a neural network or convolutional neural network, trained on a training dataset that identifies surface features for multiple different camera orientations and multiple different bones. The differences in identification will be discussed in more detail below.

[0049] The next steps in the exemplary method are to generate a registration model of the bone against a bone reference (block 910) and to register the bone model against the location of the bone reference (block 912). The generation of the registration model and the registration of the bone model are similar to, or identical to, those discussed with respect to the touch-based registration technique in Figure 6 above. To avoid making the specification excessively long, the generation of the registration model and the registration of the bone model based on the registration model are not repeated here. The exemplary registration method may then be terminated (block 914), however, the surgical controller 418 may then use the registered bone model to provide computer-assisted navigation regarding bone-involved procedures. The discussion now moves to a more detailed description of the identification of multiple surface features.

[0050] One of the problems encountered when identifying multiple surface features on bone is the fact that bone and surrounding tissues have low texture. Furthermore, within the limited field of view of the arthroscope 408, the outer surface of bone may have smoothly changing features. Therefore, detecting surface features by finding and tracking prominent features is not a simple task.

[0051] There are various related photogrammetry techniques for simultaneously extracting three-dimensional structure and camera motion from multiple frames of two-dimensional video images. More specifically, within the subset of computer vision, there is a technique called Stoichiometric Reconstruction from Motion (SfM) technique. In Stoichiometric Reconstruction from Motion-related techniques, a computer system identifies prominent features in the video image and tracks those features frame by frame. However, Stoichiometric Reconstruction from Motion-related techniques are insufficient in the context of identifying multiple surface features on bone in video images captured by arthroscope 408 and camera 410. Again, in the context of anterior cruciate ligament repair, the bone and surrounding tissues have low texture surfaces, making the Stoichiometric Reconstruction from Motion approach difficult. More specifically, arthroscopic alignment is particularly difficult not only due to poor texture but also due to deformable tissue, complex lighting, shadows, highlights, haze, condensation, and acquisition at very close range. Furthermore, the camera is handheld, the lens scope rotates, the procedure is performed in moist media, and the surgeon often switches camera portals. Attempts to use a visual SLAM pipeline, which has been reported to work in laparoscopy, have proven impractical and revealed the need for additional visual aids to achieve the robustness required for actual clinical incorporation. Thus, techniques related to structural reconstruction from motion are unable to consistently select prominent features of the outer surface of bone, nor can they find the correct correspondence between features in a continuous frame, and therefore, it is difficult to simultaneously resolve the motion of the camera and the three-dimensional structure.

[0052] The exemplary touchless registration technology addresses the problems of structural restoration-related techniques from motion by, at least in part, using artificial features of the form of a bone reference 428. In particular, in the exemplary method and system, the bone reference 428 is visible in the video image as the arthroscope 408 and camera 410 move. Since the bone reference 428 represents a known or potentially known size and fixed position, the movement of the arthroscope 408 and camera 410 can be accurately determined or analyzed. When the movement is analyzed with high precision, the determination of the three-dimensional structure represented in the two-dimensional video image becomes easier to obtain. That is, when the movement of the arthroscope 408 and camera 410 is analyzed with high precision, identifying surface features on the bone becomes a more decisive and easier computer task, thereby improving the operation of the computer system in the form of a surgical controller 418.

[0053] In particular, using known camera movements, several techniques can be used to determine the three-dimensional structure represented in a two-dimensional video image. For example, one technique may involve performing feature extraction, matching, and reconstruction on each incoming frame. That is, for each incoming frame, features are extracted, explicitly matched, and then reconstructed. Another example is that features are extracted in one frame and tracked in subsequent frames (i.e., the positioning of features in subsequent frames does not involve feature extraction). Furthermore, a high-density tracking approach may be used, where tracking is performed on most or all of the pixels in the image. Tracking some or all of the pixels provides a high-density scene reconstruction. Yet another approach is a deep learning approach, such as using a neural radiance field to implicitly perform feature extraction, matching, and reconstruction simultaneously. For all these approaches, known camera movements are used to limit the problem, and further limitations may be provided by using a 3D model of bones. These solutions can provide not only a reconstruction of the scene, but also difference information related to each reconstructed point (e.g., the normal vector).

[0054] Unlike motion-based structural reconstruction techniques that rely on feature extraction and matching to simultaneously acquire structure and camera movement, various embodiments analyze the motion pattern using the posture of bone reference 428 within each frame. The structure of bone reference 428 can be used directly, using known or predetermined features of bone reference 428, including the visible reference. In other cases, the surgical controller 418 can read the visible reference and, based on the reading, obtain size information directly from the reference, or by accessing a local or internet-based database.

[0055] As described above, bone within the limited field of view of arthroscopy 408 has low texture, and in many patients, the bone has a smoothly changing characteristic. In some cases, the exemplary touchless registration technique works directly, but in others, one or more additional steps may be involved to ensure that a sufficient sample set of surface features is identified on the bone. These additional steps can conceptually be divided into illumination strategies, application of coloring to the bone, and creation of optical texture. Each of these will be addressed in turn.

[0056] Bone is a matrix of calcium and collagen, with collagen acting as a mechanical support for bone cells. Calcium and collagen may have different absorption and / or reflectance coefficients as a function of the wavelength of photons incident on the bone surface. In certain situations, bone surface features can be highlighted by irradiating the bone with photons having wavelengths outside the visible range and, accordingly, receiving a video image based on the irradiated photons. More specifically, in one embodiment, bone may be irradiated with light in the near-infrared range, such as wavelengths between 780 nanometers (nm) and 2500 nm (inclusive). In other cases, bone may be irradiated with wavelengths between 488 nm and 620 nm (inclusive). Differences in absorption and / or reflectance of collagen and calcium in the matrix can make the optical texture more pronounced in the video image. Frames of the video image can be used directly by the surgical controller 418 to analyze motion and identify surface features. In other words, the image capture array of the camera 410 within the camera may be selected to respond to photons in the near-infrared range, and the surgical controller 418 can analyze the movement of the arthroscope 408 and the camera 410 and identify surface features based on near-infrared photons reflected or emitted from the bone.

[0057] A second conceptual additional step is to apply surface coloring to the bone before identifying surface features. In particular, in these embodiments, a chemical compound is applied to the bone to add optical texture to the bone in the video image. Otherwise, in these embodiments, a compound is applied to the bone to enhance the optical texture of the bone. In one embodiment, a biocompatible dye may be applied to the surface of the bone. The dye may take any preferred form, such as a food coloring or a dye similar to butcher's ink. Thus, the dye imparts a color to the bone, such as blue, thereby facilitating the identification of surface features of the surgical controller 418. The dye may be placed over a large area of ​​the bone surface visible in the video image (e.g., covering more than 50% of the surface area or more than 75% of the surface area). In other instances, the dye may be placed in dots or spots that are periodically or irregularly arranged on the surface of the bone.

[0058] Another example of coloration is the application of fluorescent proteins, such as green fluorescent protein (GRFP), to the surface of bone. Green fluorescent protein is a protein that fluoresces bright green in the presence of ultraviolet light. Proteins that fluoresce in other colors may be available now or in the future. Similar to dyes, green fluorescent protein, here in green, colors the bone, thereby facilitating the identification of surface features by the surgical controller 418. The fluorescent protein may be placed over a large area of ​​the bone surface visible in the video image (e.g., covering more than 50% of the surface area or more than 75% of the surface area). In other cases, green fluorescent protein may be placed in periodically or irregularly arranged dots on the surface of the bone. Thus, this embodiment achieves the objective of identifying surface features by combining the illumination strategy with the application of coloration. That is, the arthroscopic light source may be designed and constructed to generate ultraviolet light and provide combined ultraviolet and visible light to the arthroscopic 408 and the surgical site.

[0059] A third conceptual additional step to aid in the identification of surface features is to generate optically textured areas on the bone surface. That is, bone often has smooth and smoothly changing features. The additional step in these embodiments involves adding optical texture, which aids in the identification of surface features by adding optical texture, but does not significantly affect the structural integrity of the bone. In one embodiment, the optical texture may be added to a non-bearing location. In an anterior cruciate ligament repair embodiment, the optical texture may be added within the intercondylar notch, on the medial surface of the lateral condyle (e.g., above the lateral meniscus 112), and / or on the medial surface of the medial condyle (e.g., above the medial meniscus 114).

[0060] The addition of optical texture can take many forms. In one embodiment, optical texture may be added by a rotary cutting element or a mechanical cutting instrument with burrs, the burrs being designed and constructed to limit the amount of bone removed. For example, burrs may be designed and constructed to break up or create grooves having a depth of 0.1 millimeters (mm) or less. In other cases, optical texture may be generated by a counterbore drill bit designed and constructed to limit the drill depth, such as in a shoulder area that limits the drill depth to 0.1 mm or less. For example, a surgeon may place multiple counterbores in the bone along a non-bearing surface. Furthermore, a surgeon may perform mechanical wear on the bone surface, such as by using any suitable instrument, such as a sight 426 (Figure 4), to pull the sight along the bone surface, thereby creating surface indentations.

[0061] The examples of touchless registration technology discussed at this point separate the analysis of camera motion relative to a bone reference from the identification of surface features. However, in other examples, the analysis and identification may be an integrated programmed method in which the bone reference is found within each frame, the motion is analyzed frame by frame, and the identification of surface features is performed substantially simultaneously with the inter-frame analysis of the motion.

[0062] Here, this specification moves on to a third technique for registering a bone model to bone, namely a patient-specific device. In both the touch-based and touchless registration techniques discussed up to this point, a registration model is generated and used to register the bone model to a bone visible in a video image. Conceptually, the registration model is used to determine the coordinate transformation and scaling to align the bone model to the actual bone. However, if the orientation of the bone in the video image is known or can be determined, the use of the registration model may be omitted, and instead, the coordinate transformation and scaling may be calculated directly.

[0063] Figure 10 shows a method according to at least some embodiments. The exemplary method may be implemented in part in software within one or more computer systems, such as a surgical controller 418. In particular, the exemplary method includes initiating (block 1000) and acquiring a three-dimensional bone model (block 1002). Similar to the prior art, in the patient-specific instrument registration technique, what is acquired is a three-dimensional bone model that may be created by segmenting multiple non-invasive images (e.g., MRI) taken preoperatively or intraoperatively.

[0064] The next step in the exemplary method is to generate a patient-specific device that has features designed to bond to the bone represented in the bone model in only one orientation (block 1004). Generating the patient-specific device may first involve selecting the location where the patient-specific device will be attached. For example, a device or computer system may analyze the bone model and select the attachment site. In various embodiments, the attachment site may be unique in the sense that the patient-specific device will not bond to the bone at any other location, if it is fabricated to bond to that unique location. In the exemplary case of anterior cruciate ligament repair, the selected site may be the upper or superior portion of the intercondylar notch, or near there. If the bone model shows another location with unique features, such as a bone spur, or other raised or depressed surface abnormality, such a unique location may be selected as the attachment site for the patient-specific device.

[0065] Furthermore, the formation of patient-specific devices can take any preferred form. In one embodiment, a device or computer system may directly print the patient-specific device, for example, using a 3D printer. In other embodiments, the device or computer system may print a model of the mounting location, which then becomes a mold for creating the patient-specific device. For example, the model may be a mold for injection-molded plastic or casting techniques. In some embodiments, the patient-specific device carries one or more criteria, but as mentioned above, in other embodiments, the patient-specific device may track itself and therefore not carry any criteria.

[0066] The next step in the exemplary method is to bond the patient-specific device to the bone, in some cases the reference is a patient-specific device bonded to the external surface (block 1006). As previously mentioned, the attachment site of the patient-specific device is selected to be unique so that the patient-specific device bonds to the bone in only one position and in only one orientation. In the exemplary case of arthroscopic ACL repair, the patient-specific device may be inserted arthroscopically; that is, the attachment site may be selected so that the physical size of the patient-specific device can be inserted through a port in the patient's skin. In other cases, the patient-specific device may be fabricated or constructed from a flexible material that allows the patient-specific device to deform for insertion into the surgical site, and still return to its predetermined shape to bond to the attachment site. However, in open procedures where the skin is cut and pulled apart and the bone is exposed to open air, the patient-specific device may be a rigid device with fewer size limitations.

[0067] The next step in the exemplary method is to capture video images of patient-specific instruments (block 1008). Here again, the capture may be performed intraoperatively. In the exemplary case of arthroscopic anterior cruciate ligament repair, video images are captured by the arthroscope 408 and camera 410 by the surgical controller 418. However, in open procedures in which the skin is cut and pulled away and the bone is exposed to open air, the capture may be by any preferred camera device, such as one or both cameras of a stereoscopic camera system, or by a portable computing device, such as a tablet or smartphone device. In such cases, the video images may be provided to the surgical controller 418 in any preferred form.

[0068] The next step in the exemplary method is to register a bone model based on the location of the patient-specific instrument (block 1010). That is, assuming that the patient-specific instrument attaches to the bone in only one position and only one orientation, the position and orientation of the patient-specific instrument are directly related to the position and origin of the bone, and therefore the coordinate transformation and scaling for registration can be calculated directly. The exemplary method can then be completed (block 1012), however, the surgical controller 418 may then use the registered bone model to provide computer-assisted navigation regarding surgical tasks or surgical procedures involving bone.

[0069] For example, using a registered bone model, the surgical controller 418 may provide guidance on the surgical task of a surgical procedure. Specific guidance depends on the surgical procedure being performed and the stage of the procedure. A non-exhaustive list of guidance includes changing the drill path entry point, changing the drill path exit point, aligning the sight along the planned drill path, indicating the location to cut and / or excise bone, reaming bone to a specific depth along a specific direction, positioning devices (sutures, anchors, or other) in specific locations, positioning sutures in specific locations, positioning anchors in specific locations, indicating areas of bone to contact and / or avoid, and identifying areas of anatomical structures and / or landmarks. In further cases, guidance may include being displayed on a display device, which may be an arthroscopic display or a see-through display, or being highlighted within a version of a video image by communicating with a virtual reality device or robotic tool.

[0070] Figure 11 shows a cross-sectional view of the right knee from the anterior (front) side, with the patella removed and an exemplary patient-specific device in place. In particular, what is visible in Figure 1 is the lower portion of the femur 100, including the lateral condyle, or lateral condyle 102, and the medial condyle, or medial condyle 104. The femur 100 and the condyles 102 and 104 are in a working relationship with the tibia 106, which includes the tibial tuberosity 108 and Geldi's tubercle 110. An intercondylar notch 600 is located between the lateral condyle 102 and the medial condyle 104.

[0071] In the embodiment shown in Figure 11, a patient-specific instrument 1102 is positioned within the intercondylar notch 600. Specifically, the patient-specific instrument 1102 in Figure 11 is designed and constructed to connect within the intercondylar notch 600, between the upper or apex of the intercondylar notch 600 and the medial condyle 104. In this way, the patient-specific instrument 1102 is visible in video images captured by the arthroscope 408 and camera 410, but generally does not interfere with or obstruct the anterior cruciate ligament repair tunnel, which is positioned through the lateral condyle 102.

[0072] In the embodiment shown in Figure 11, the patient-specific device 1102 includes feature 1104, which is designed and constructed to bond to the bone in only one orientation. Only a portion of feature 1104 is visible in Figure 11, as long as the exemplary patient-specific device 1102 is shown in a fitted state. However, feature 1104 is actually a negative image of the bone at the attachment site.

[0073] An exemplary patient-specific device 1102 further includes a criterion 1106, which is placed on or coupled to the outer surface of the patient-specific device 1102. While the exemplary criterion 1106 is shown in the form of a quick-response (QR) code, any machine-readable code (e.g., a one-dimensional barcode, a two-dimensional barcode) may also be used. Although the exemplary patient-specific device 1102 includes only a single criterion, in other cases the patient-specific device 1102 may include two or more criteria, each placed on its respective surface. In yet another case, the patient-specific device itself may be tracked and therefore not carry any criteria.

[0074] Figure 12 shows an exemplary computer system 1200. In one embodiment, the computer system 1200 may correspond to a surgical controller 418, a device for generating patient-specific instruments, an in-operating tablet device, or any other system implementing any or all of the various methods described herein. The computer system 1200 may be connected to other computer systems (e.g., network connections) on a local area network (LAN), intranet, and / or extranet (e.g., the device cart 402 network), or, at certain times, on the internet (e.g., when not being used in surgical procedures). The computer system 1200 may be a server, a personal computer (PC), a tablet computer, or any device capable of executing a set of instructions (sequential or otherwise) specifying actions to be performed by that device. Furthermore, although only a single computer system is illustrated, the term “computer” shall also be interpreted to include any collection of computers that individually or collectively execute a set of instructions (or multiple sets) to perform any one or more of the methods described herein.

[0075] The computer system 1200 includes a processing device 1202, main memory 1204 (e.g., read-only memory (ROM), flash memory, synchronous DRAM (SDRAM), and other dynamic random access memory (DRAM)), static memory 1206 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 1208, which communicate with each other via a bus 1210.

[0076] The processing device 1202 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device 1202 may be a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set, or a processor implementing a combination of instruction sets. The processing device 1202 may also be one or more application-specific processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 1202 is configured to execute instructions for performing any operations and steps described herein. Once programmed with specific instructions, the processing device 1202, and thus the entire computer system 1200, becomes an application-specific device, such as a surgical controller 418.

[0077] The computer system 1200 may further include a network interface device 1212 for communicating with any suitable network (e.g., the DeviceCart 402 network). The computer system 1200 may also include a video display 1214 (e.g., a display device 414), one or more input devices 1216 (e.g., a microphone, keyboard, and / or mouse), and one or more speakers 1218. In one exemplary embodiment, the video display 1214 and the input device(s) 1216 may be combined within a single component or device (e.g., an LCD touchscreen).

[0078] The data storage device 1208 may include a computer-readable storage medium 1220 on which instructions 1222 (for example, implementing any method and any function performed by any illustrated device and / or component described herein) are stored, embodying any one or more of the methodologies or functions described herein. The instructions 1222 may also reside, all or at least partially, in the main memory 1204 and / or in the processing device 1202 when executed by the computer system 1200. Thus, the main memory 1204 and the processing device 1202 also constitute computer-readable media. In certain cases, the instructions 1222 may also be transmitted or received over a network via the network interface device 1212.

[0079] In the exemplary embodiments, the computer-readable storage medium 1220 is shown as a single medium, while the term “computer-readable storage medium” shall be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term “computer-readable storage medium” shall also be interpreted to include any medium that causes a machine to execute any one or more of the methodologies of this disclosure, which may store, encode, or transmit a set of instructions for machine execution. Accordingly, the term “computer-readable storage medium” shall be interpreted to include, but not be limited to, solid-state memory, optical media, and magnetic media.

[0080] The above discussion is intended to illustrate the principles and various embodiments of the present invention. Many variations and modifications will be apparent to those skilled in the art once the above disclosure is fully understood. The following claims are intended to be construed as encompassing all such variations and modifications.

Claims

1. A method for registering a bone model, wherein the method is The bone model is obtained using a surgical controller, The surgical controller captures bone-referenced video images fixed to the bone, and the capture is performed using a moving camera. The surgical controller analyzes the movement of the camera relative to the bone reference based on the video image, The surgical controller identifies multiple surface features on the bone in the video image based on the movement of the camera, The surgical controller generates a registered model of the bone for the bone reference, A method comprising registering the bone model for the position of the bone reference based on the registered model using the surgical controller.

2. The method according to claim 1, further comprising providing navigation information relating to a procedure involving the bone using the surgical controller.

3. The method according to claim 1, further comprising identifying the plurality of surface features on the bone by irradiating the bone with a wavelength of photons that enhances the optical texture.

4. The method according to claim 3, further comprising irradiating the bone with the wavelength of photons that enhance the optical texture with light having a wavelength between 780 nanometers (nm) and 2500 nm (including both ends).

5. The method according to claim 3, further comprising irradiating the bone with the wavelength of light that enhances the optical texture with wavelengths between 488 nanometers (nm) and 620 nm (including both ends).

6. The method according to claim 3, further comprising irradiating the bone with a wavelength of light that enhances the optical texture, with a wavelength of light that has a higher reflectivity to collagen than to calcium.

7. The method according to claim 1, further comprising applying surface coloring to the bone before identifying the plurality of surface features.

8. The method according to claim 7, wherein applying the surface coloring further comprises applying a dye to the bone.

9. The method according to claim 7, wherein applying the surface coloring further comprises applying green fluorescent protein to the bone.

10. The method according to claim 1, further comprising generating a textured region on the bone before identifying the plurality of surface features.

11. The method according to claim 10, further comprising generating the texture region at a non-load-bearing position.

12. The method according to claim 11, further comprising generating the texture region at the non-load-bearing position for anterior cruciate ligament treatment, in the intercondylar notch.

13. The method according to claim 10, further comprising performing mechanical wear to create the texture region.

14. The method according to claim 1, further comprising applying the video image and the display of the camera's movement to a machine learning algorithm trained to recognize the optical texture of bone, in order to identify the plurality of surface features.

15. The method according to claim 1, wherein the camera is attached to a rigid endoscope.

16. The method according to claim 15, wherein the endoscope is monocular.

17. Obtaining the aforementioned bone model Segmenting multiple non-invasive images taken preoperatively or intraoperatively, The method according to claim 1, comprising generating the bone model based on the segmentation.

18. It is a surgical controller, Processor and A memory that is communicably coupled to the processor, wherein the memory contains an instruction that, when executed by the processor, the processor Receiving a bone model, The capture of bone-referenced video images fixed to a bone, wherein the video images are captured while the camera is moving. Based on the aforementioned video images, the movement of the camera relative to the bone reference is analyzed, Based on the movement of the camera, the video image is used to identify multiple surface features on the bone, To generate a registered model of the bone for the aforementioned bone criteria, A surgical controller comprising: a memory for storing instructions for registering the bone model for the position of the bone reference based on the registered model; and a memory for storing instructions for executing the above.

19. The surgical controller according to claim 18, wherein the instruction causes the processor to further provide navigation information relating to a procedure involving the bone.

20. The surgical controller according to claim 18, wherein when the processor identifies the plurality of surface features on the bone, the instruction further instructs the processor to cause a light source to illuminate the bone with wavelengths of light that enhance the optical texture.

21. The surgical controller according to claim 20, wherein when the processor commands the light source to irradiate the bone with the wavelength of light that enhances the optical texture, the command commands the processor to irradiate the bone with light having a wavelength between 780 nanometers (nm) and 2500 nm (including both ends).

22. The surgical controller according to claim 20, wherein the processor commands the light source to irradiate the bone with a wavelength of light that enhances the optical texture, and the command also commands the processor to cause the light source to irradiate the bone with a wavelength of light that has a higher reflectivity to calcium than to collagen.

23. The surgical controller according to claim 20, wherein the processor commands the light source to irradiate the bone with a wavelength of light that enhances the optical texture, and the command also commands the processor to cause the light source to irradiate the bone with a wavelength of light that has a higher reflectivity to collagen than to calcium.

24. The surgical controller according to claim 18, wherein when the processor identifies the plurality of surface features, the instruction causes the processor to apply the video image and the display of the camera's operation to a machine learning algorithm trained to recognize the optical texture of bone.

25. A method for registering a bone model, wherein the method is The bone model is obtained using a surgical controller, The surgical controller captures bone-referenced video images attached to the bone, The surgical controller determines the position of the distal tip of the medical device visible in the video image, wherein the distal tip contacts the bone in at least part of the video image frame, and the medical device does not have a reference point. The surgical controller generates a registered model of the bone relative to the bone reference based on the position of the distal tip of the medical device, A method comprising registering the bone model for the position of the bone reference based on the registered model using the surgical controller.

26. The method according to claim 25, further comprising providing navigation information relating to a procedure involving the bone using the surgical controller.

27. Determining the position of the distal tip of the medical device is The surgical controller segments the medical device within the video image, The surgical controller estimates multiple poses of the medical device within each of multiple frames of the video image, The method according to claim 25, further comprising determining the position based on the plurality of postures.

28. The method according to claim 27, further comprising applying the video images to a machine learning algorithm trained to perform six-dimensional pose estimation in order to estimate the plurality of poses.

29. The method according to claim 28, further comprising applying the video image to a convolutional neural network trained to perform the six-dimensional pose estimation, which is a machine learning algorithm trained to perform the six-dimensional pose estimation.

30. The method according to claim 25, wherein the medical device is at least one selected from the group including a touch probe, a sighting device, and a drill guide.

31. The method according to claim 25, wherein the camera is attached to a rigid endoscope.

32. The method according to claim 31, wherein the endoscope is monocular.

33. The aforementioned bone model, Segmenting multiple non-invasive images taken preoperatively or intraoperatively, The method according to claim 25, further comprising generating the bone model based on the segmentation.

34. It is a surgical controller, Processor and A memory that is communicably coupled to the processor, wherein the memory contains an instruction that, when executed by the processor, the processor Obtaining a bone model, Capturing bone-referenced video images fixed to the bone, Determining the position of the distal tip of a medical device visible in the video image, wherein the distal tip is in contact with the bone in at least a portion of the video image frame, and the medical device has no reference point. Based on the position of the distal tip of the medical device, a registration model of the bone is generated relative to the bone reference. A surgical controller comprising: a memory for storing instructions for registering the bone model for the position of the bone reference based on the registered model; and a memory for storing instructions for performing the above.

35. The surgical controller according to claim 34, wherein the instruction causes the processor to further provide navigation information relating to a procedure involving the bone.

36. When the processor determines the position of the distal tip of the medical device, the instruction is given to the processor: Segmenting the medical device within the aforementioned video image, Estimating multiple poses of the medical device within each of multiple frames of the video image, A surgical controller according to claim 34, which determines the position based on the aforementioned multiple postures.

37. The surgical controller according to claim 36, wherein, when the processor estimates the plurality of poses, the instruction causes the processor to apply the video image to a machine learning model trained to perform six-dimensional pose estimation.

38. The surgical controller according to claim 37, wherein when the processor applies the video image to the machine learning model, the instruction causes the processor to apply the video image to a convolutional neural network trained to perform six-dimensional pose estimation.

39. The surgical controller according to claim 34, wherein the medical device in the video image is at least one selected from the group including a touch probe, a sighting device, and a drill guide.

40. A method for registering a bone model, wherein the method is The device acquires the bone model, The device generates a patient-specific device that is configured to bond to the bone represented in the bone model in only one orientation. The patient-specific device is attached to the bone, The surgical controller captures video images of the patient-specific device, and the capture is performed by a camera. A method comprising registering the bone model based on the position of the patient-specific device using the surgical controller.

41. The method according to claim 40, further comprising guiding a surgical task of a surgical procedure using the surgical controller, wherein the guiding is based on the registered bone model.

42. The method according to claim 41, further comprising guiding the surgical task at least one selected from the group including changing the drill path entry point, changing the drill path exit point, aligning the sight along the planned drill path, indicating the location to cut the bone, and highlighting the area of ​​the bone to be avoided in a version of the video image displayed on a display device.

43. The method according to claim 41, wherein the surgical procedure is an open procedure.

44. The method according to claim 41, wherein the surgical procedure is an arthroscopic surgical procedure.

45. The method according to claim 44, wherein the camera is attached to a rigid arthroscope.

46. The method according to claim 45, wherein the arthroscope is monocular.

47. The method according to claim 44, wherein the arthroscopic surgical procedure is a surgical procedure performed on a part selected from the group consisting of the knee, hip, shoulder, wrist, and ankle.

48. The method according to claim 44, wherein the arthroscopic surgical procedure is at least one selected from the group consisting of anterior cruciate ligament repair, reduction of femoroacetabular impingement, and rotator cuff repair.

49. Obtaining the aforementioned bone model Segmenting multiple non-invasive images taken preoperatively or intraoperatively, The method according to claim 40, further comprising generating the bone model based on the segmentation.

50. It is a surgical controller, Processor and A memory that is communicably coupled to the processor, wherein the memory contains an instruction that, when executed by the processor, the processor Obtaining a bone model, Capturing a patient-specific video image of a standard device, wherein the capture is performed by a camera, A surgical controller comprising a memory for storing commands to register the bone model based on the position of the patient-specific instrument, and to perform the procedure.

51. The surgical controller according to claim 50, wherein the instruction causes the processor to further guide the surgical task of a surgical procedure, and the guidance is based on the registered bone model.

52. The surgical controller according to claim 51, wherein the guiding is at least one selected from the group consisting of: changing the drill path entry point; changing the drill path exit point; aligning the sight along a planned drill path; indicating the location to cut and / or excise the bone; reaming the bone to a specific depth along a specific direction; positioning a device (suture, anchor, or other) at a specific location; positioning a suture at a specific location; positioning an anchor at a specific location; indicating areas of bone to contact and / or avoid; and identifying areas of anatomical structures and / or landmarks.

53. The surgical controller according to claim 52, wherein the guiding is provided by highlighting within a version of the video image displayed on a display device, which may be an arthroscopic or see-through display, or by communicating with a virtual reality device or robotic tool.

54. The surgical controller according to claim 51, wherein the guiding is at least one selected from the group consisting of changing the drill path entry point, changing the drill path exit point, aligning the sight along the planned drill path, indicating the location to cut the bone, and highlighting the area of ​​the bone to be avoided in a version of the video image displayed on a display device.

55. The surgical controller according to claim 52, wherein the surgical procedure is an open procedure.

56. The surgical controller according to claim 52, wherein the surgical procedure is an arthroscopic surgical procedure.

57. The surgical controller according to claim 52, wherein the surgical procedure is a surgical procedure performed on a body part selected from the group consisting of the knee, hip, shoulder, wrist, and ankle.

58. The surgical controller according to claim 52, wherein the surgical procedure is at least one selected from the group consisting of anterior cruciate ligament repair, reduction of femoroacetabular impingement, and rotator cuff repair.

Images