A system and method for providing navigation guidance and recommendations related to repositioning surgery using machine learning.

Abstract

In some examples, a computer implementation method may include receiving one or more images related to a patient's surgical site. The method may include identifying one or more hardware pieces attached to the surgical site based on one or more images and using one or more trained machine learning models. The method may include determining navigation guidance for uninstalling one or more hardware pieces during a re-implantation surgery based on one or more hardware pieces and using one or more machine learning models. The method may include transmitting navigation guidance to a computing device associated with the surgeon performing the re-implantation surgery.

Description

TECHNICAL FIELD

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 585,372, filed on September 26, 2023, entitled "System and Method for Using Machine Learning to Provide Navigation Guidance and Recommendations Related to Revision Surgery." This provisional application is incorporated herein by reference as if its full text is reproduced below.

[0002] The anterior cruciate ligament (ACL) functions as the primary mechanical restraint in the knee to resist anterior translational movement of the tibia relative to the femur. Similarly, the posterior cruciate ligament (PCL) functions as a mechanical restraint to resist posterior translational movement of the tibia relative to the femur. These cruciate ligaments contribute significantly to knee stability, and ACL injuries are extremely common. Most ACL injuries are complete tears of the ligament.

[0003] Since ACL injuries often occur in young and active patients, ACL reconstruction is performed to enable a return to activities. Its purpose is to restore knee stability and reduce the possibility of further damage to the meniscus and articular cartilage that would lead to the development of knee osteoarthritis. Reconstruction may consist of placing a substitute graft (e.g., an autograft from the central third of the patellar tendon or from the hamstring tendon). Both ends of the graft are placed in respective tunnels prepared through the femur and tibia. Both ends of the graft may be attached using interference screws or suspension fixation devices such as the ENDOBUTTON® brand fixation device manufactured by Smith & Nephew of Andover, Massachusetts, USA.

[0004] In some cases, ACL reconstruction surgery may fail, requiring a repositioning surgery. Repositioning ACL surgery can be challenging for several reasons. Firstly, the surgeon performing the repositioning may not have been the same surgeon who performed the primary ACL repair. Furthermore, the tunnel placement, tunnel width, and / or hardware manufacturer may be unknown during preoperative planning, among other things. [Overview of the project]

[0005] In some examples, a computer implementation method may include receiving one or more images related to a patient's surgical site. The method may include identifying one or more hardware pieces attached to the surgical site based on one or more images and using one or more trained machine learning models. The method may include determining navigation guidance for uninstalling one or more hardware pieces during a re-implantation surgery based on one or more hardware pieces and using one or more machine learning models. The method may include transmitting navigation guidance to a computing device associated with the surgeon performing the re-implantation surgery.

[0006] In some examples, the system includes memory for storing instructions and a processing unit communicatively coupled to the memory. The processing unit executes instructions to receive one or more images related to the surgical site of a patient; identify one or more hardware pieces attached to the surgical site based on the one or more images and using one or more trained machine learning models; determine navigation guidance for uninstalling one or more hardware pieces during a re-operation based on the one or more hardware pieces and using one or more machine learning models; and transmit the navigation guidance to a computing device associated with the surgeon performing the re-operation.

[0007] In some cases, a tangible, non-temporary, computer-readable storage medium stores instructions that, at runtime, cause a processing unit to perform the following actions: receive one or more images related to a patient's surgical site; identify one or more hardware pieces attached to the surgical site based on the one or more images and using one or more trained machine learning models; determine navigation guidance for uninstalling one or more hardware pieces during repositioning surgery based on one or more hardware pieces and using one or more machine learning models; and transmit the navigation guidance to a computing device associated with the surgeon performing the repositioning surgery. [Brief explanation of the drawing]

[0008] For a detailed description of exemplary embodiments, please refer to the accompanying drawings.

[0009] [Figure 1] Figure 1 shows an anterior view, or frontal cross-section, of the right knee with the patella removed. [Figure 2] Figure 2 shows a posterior view, or dorsal cross-section, of the right knee. [Figure 3] Figure 3 shows a view of the intercondylar notch of the femur, seen from below. [Figure 4] Figure 4 shows a surgical system according to at least some embodiments. [Figure 5] Figure 5 shows a conceptual diagram of a surgical site according to at least some embodiments, in which various objects within the surgical site are tracked. [Figure 6] Figure 6 is an exemplary video display according to at least some embodiments, showing a portion of the femur and having a visible bone reference point inside. [Figure 7] Figure 7 is an exemplary video display, according to at least some embodiments, showing intraoperative changes related to the tunnel route. [Figure 8] Figure 8 shows a computing device that runs an artificial intelligence engine, according to at least some embodiments. [Figure 9] Figure 9 is an illustrative flowchart of a method for providing navigation guidance for repositioning surgery using a machine learning model, according to at least some embodiments. [Figure 10] Figure 10 shows a computer system according to at least some embodiments.

[0010] definition Various terms are used to refer to specific system components. Different companies may use different names to refer to the same component, but this document is not intended to distinguish between components that have different names but no difference in function. In the following discussion and claims, the terms “includes” and “equipped with” are used in an open-ended manner and should therefore be interpreted as “includes, but not limited to.” Also, the terms “couple” or “couples” are intended to mean either an indirect or direct connection. Thus, when a first device is coupled to a second device, that connection may be a direct connection or an indirect connection via other devices or other connections.

[0011] An endoscope having a "single optical path" means that the endoscope is not a stereoscopic endoscope having two separate optical paths separated by the interpupillary distance at the focusing end of the endoscope. The fact that an endoscope has two or more optical components (e.g., glass rods, optical fibers) that form a single optical path does not negate the state of having a single optical path.

[0012] "Through-hole" refers to an opening or passage through a base device. However, the term "through-hole" should not be interpreted as implying any method of creation. Thus, through-holes can be created by any preferred method, such as drilling, piercing, laser drilling, or casting.

[0013] "Counterbore" refers to an opening or passage into the base of the device. If a counterbore intersects with another opening (e.g., a through hole), it may thereby define an internal shoulder. However, the term "counterbore" should not be interpreted as implying any method of creation. Counterbores can be created by any preferred method, such as drilling, drilling, laser drilling, or casting.

[0014] The terms "input" and "output," when used as nouns, refer to connections (e.g., electrical, software) and are not to be interpreted as verbs requiring action. For example, a timer circuit may define a clock output. An exemplary timer circuit may create or drive a clock signal on the clock output. In systems implemented directly in hardware (e.g., on a semiconductor substrate), these "inputs" and "outputs" define electrical connections. In systems implemented in software, these "inputs" and "outputs" define parameters that are read or written by instructions implementing a function, respectively.

[0015] "Controller" means an individual circuit component, application-specific integrated circuit (ASIC), microcontroller with control software, reduced instruction set computing (RISC) with control software, digital signal processor (DSP), processor with control software, programmable logic device (PLD), or field-programmable gate array (FPGA) configured to read inputs and drive outputs in response to inputs, either alone or in combination.

[0016] "Real-time" can refer to actions, operations, methods, functions, or commands that are executed within 2 seconds. "Near real-time" can refer to actions, operations, methods, functions, or commands that are executed within 2 seconds to 1 minute.

[0017] When used in the context of machine learning, the term "supervised learning" can refer to the technique of training an algorithm using labeled datasets to classify data or accurately predict outcomes. Labeled input data can be provided to a machine learning model, which then adjusts its weights and / or other parameters until it is trained to properly identify labeled outputs. The algorithm measures its accuracy through a loss function by adjusting its weights and / or parameters until the error meets a threshold level.

[0018] When used in a machine learning context, the term “unsupervised learning” can refer to techniques for analyzing and clustering unlabeled datasets by identifying patterns or groups of data within a dataset based on similarities and / or differences between datasets. An example of unsupervised learning is clustering, which can refer to a data mining technique that groups unlabeled data based on similarities or differences within different parts of the unlabeled data. Another example of unsupervised learning is association rules, which can refer to a rule-based method for finding relationships between variables in a given dataset. Yet another example of unsupervised learning is dimensionality reduction, which can refer to a technique used when the number or dimensions of features in a dataset are too large. Dimensionality reduction reduces the number of data inputs to a manageable size while maintaining the completeness of the dataset.

[0019] When used in the context of machine learning, the term "reinforcement learning" can refer to a technique that enables an agent to learn in an interactive environment through trial and error by using feedback from its own actions and experiences. Reinforcement learning uses rewards and punishments as signals to indicate positive and negative actions of the agent during the training phase of a machine learning model. The goal of reinforcement learning is to discover a suitable machine learning model that maximizes the cumulative reward of the agent or the sum of the cumulative rewards associated with the agent. Detailed description of the invention

[0020] The following discussion is directed to various embodiments of the present disclosure. Although one or more of these embodiments may be preferred, the disclosed embodiments should not be construed as limiting the scope of the present disclosure, including the claims, nor should they be used in other manners. In addition, those skilled in the art will appreciate that the following description has a wide range of applications, and that the discussion of any embodiment is intended to be merely illustrative of that embodiment, and is not intended to imply that the scope of the present disclosure, including the claims, is limited to that embodiment.

[0021] Various examples are directed to methods and systems for providing navigation guidance for performing revision surgery based on identified hardware attached to a portion of a patient's body using a machine learning model. Identifying existing hardware inside a patient is a complex and time-consuming aspect of pre-operative surgical planning. Conventionally, if the hardware is not correctly identified pre-operatively, this may result in a rescheduled or delayed revision surgery. Consequently, there is a clinical need for a digital solution that can accurately identify existing hardware and allow a surgeon to obtain the relevant drivers, instruments and tools prior to revision surgery. The disclosed embodiments provide the desired solution. There are numerous commercially available metallic and non-metallic absorbable and non-absorbable screws, each requiring the correct driver / tool to remove it. A sample list of custom drivers includes hex, cannulated, square, trilobular, and torso types, all in various metric and imperial sizes. The disclosed embodiments use a trained machine learning model to analyze images (e.g., computed tomography images) to identify installed hardware (e.g., screws), and output tools and navigation guidance used to uninstall the hardware for performing revision surgery. Further, the machine learning model may output the success rate of performing the revision surgery, and / or one or more probabilities related to collision with hardware installed during the revision surgery.

[0022] Various examples have been developed in connection with ACL repair, and thus the following discussion is based on the circumstances up to the development. However, this technique is applicable to many types of ligament repair, such as medial collateral ligament repair, lateral collateral ligament repair, and posterior cruciate ligament repair. Furthermore, various exemplary methods and systems can also be used to plan and place anchors for reattaching soft tissue, such as reattachment of the hip labrum, shoulder, or meniscal root. Therefore, the present specification and the circumstances up to the development shall not be construed as limiting the applicability of the present teachings. To orient the reader, the present specification begins with a description of the knee.

[0023] Figure 1 shows an anterior view, i.e., a front cross-sectional view, of the right knee with the patella removed. In particular, what is visible in Figure 1 is the lower portion of the femur 100, which includes the lateral condyle, or lateral condyle 102, and the medial condyle, or medial condyle 104. The femur 100 and the condyles 102, 104 are in operative relation to the tibia 106, which includes the tibial tuberosity 108 and Gerdy's tubercle 110. A lateral meniscus 112 and a medial meniscus 114 are disposed between the femoral condyles 102, 104 and the tibia 106. Several ligaments, such as the ACL 116 extending from the lateral side of the femoral notch to the medial side of the tibia 106, are visible in the view of Figure 1. Conversely, the posterior cruciate ligament 118 extends from the medial side of the femoral notch to the tibia 106. The fibula 120 and several additional ligaments that are not specifically labeled are also visible.

[0024] Figure 2 shows a posterior view, or dorsal cross-section, of the right knee. In particular, what is visible in Figure 2 is the lower portion of the femur 100, including the lateral condyle 102 and the medial condyle 104. The femur 100 and the femoral condyles 102 and 104 are in a working relationship with the tibia 106, and the lateral meniscus 112 and the medial meniscus 114 are located between the femoral condyles 102 and 104 and the tibia 106. Figure 2 further shows the ACL 116 extending from the lateral side of the femoral notch to the medial side of the tibia 106, but its attachment point to the tibia 106 is not visible. The posterior cruciate ligament 118 extends from the medial side of the femoral notch to the tibia 106, but its attachment point to the femur 100 is not visible. In this case as well, several additional ligaments, which are not specifically labeled, are shown.

[0025] The most common type of ACL injury is a complete rupture of the ligament. Treatment involves reconstructing the ACL by placing a replacement graft (e.g., an autograft from either the patellar tendon, quadriceps femoris tendon, or hamstring tendon). The graft is placed inside a tunnel prepared within the femur 100 and tibia 106. The current standard of treatment for ACL repair is to position the tunnel so that the tunnel entry point for the graft is at the anatomical attachment site of the natural ACL. Such tunnel positioning to the attachment site of the natural ACL attempts to reproduce the original kinematic characteristics of the knee. In arthroscopic surgery, the location of the tunnel through the tibia 106 is relatively easy to reach, especially when bending or flexing the knee. However, the tunnel through the femur 100 is located within the intercondylar notch. Depending on the patient's physique and how the surgeon selects the location of the port, which is a skin-penetrating port through which various instruments are inserted into the knee, it may be difficult to reach the attachment site of the natural ACL to the femur 100.

[0026] Figure 3 shows a view of the intercondylar notch of the femur from below. In particular, the lateral condyle 102 and medial condyle 104 are visible in Figure 3. The femoral notch 200 is defined between the femoral condyles 102 and 104. The femoral tunnel may define a medial opening 202 within the femoral notch 200, the medial opening 202 being close to the lateral condyle 102 and displaced in the posterior portion of the femoral notch 200. The femoral tunnel extends through the femur 100 and forms a lateral opening on the outer or lateral surface of the femur 100 (the lateral opening is not visible in Figure 3). Figure 3 shows an exemplary drill wire 204 that may be used to create the initial tunnel, or pilot hole. Once the surgeon has confirmed that the drill wire 204 is precisely aligned with the planned tunnel path, the femoral tunnel is created by drilling or reaming with another instrument (e.g., a reamer) that can use the drill wire 204 as a guide. In some cases, a socket or counterbore is created on the intercondylar notch side to accommodate the width of the graft extending into the bone, and that counterbore may also be created using another instrument (e.g., a reamer) that can use the drill wire 204 as a guide.

[0027] The tunnel may be drilled from either direction. Considering the femoral tunnel again as an example, the tunnel may be drilled from the lateral portion of the femur 100 toward the femoral notch 200 and further into the femoral notch 200; this is called the “outside-in” procedure. Conversely, the exemplary femoral tunnel may be drilled from inside the femoral notch 200 toward the lateral portion of the femur 100 and further into the lateral portion; this is called the “inside-out” procedure. The various examples described later are equally applicable to both outside-in and inside-out procedures. In the outside-in procedure, an additional device may be used to hold the drill wire over the lateral portion and to physically indicate the expected tunnel position at the medial opening inside the knee. However, since the device for the outside-in procedure is difficult to use in arthroscopic procedures, the inside-out procedure is used in many arthroscopic repairs. Therefore, although the further examples described later are based on the inside-out procedure, this should not be interpreted as limiting. This specification now moves on to a description of an exemplary surgical system.

[0028] Figure 4 shows a surgical system (not to exact scale) according to at least some embodiments. In particular, exemplary surgical system 400 includes a tower or instrument cart 402, exemplary mechanical resection instruments 404, exemplary plasma-based cauterization instruments (hereinafter simply referred to as cauterization instruments 406), and an exemplary form of endoscope comprising an arthroscope 408 and an attached camera head 410. The instrument cart 402 may include a camera 412 (exemplarily shown as a stereoscopic camera), a display device 414, a resection controller 416, and a camera control unit (CCU), together with an endoscope light source and a video controller 418. In exemplary cases, the CCU and video controller 418 not only provide light to the arthroscope 408 and display images received from the camera head 410, but also implement various additional embodiments such as displaying a single aimer indicator that modifies depending on when the tool is coaxially aligned, and displaying an iso-angle overlay representing the knee angle to achieve a target exit angle. Therefore, hereafter, the CCU and video controller will be referred to as the surgical controller 418. However, in other cases, the CCU and video controller may be separate systems distinct from the controller that handles the nature of intraoperative changes, but even then, the separate devices will be operablely coupled.

[0029] The exemplary apparatus cart 402 further includes a pump controller 422 (e.g., a single or pair of peristaltic pumps). Fluid connections between the mechanical excision instrument 404 and the cauterization instrument 406 are not shown to avoid overcomplicating the drawings. Similarly, fluid connections between the pump controller 422 and the patient are not shown to avoid overcomplicating the drawings. In the exemplary system, both the mechanical excision instrument 404 and the cauterization instrument 406 are coupled to an excision controller 416, which is a dual-function controller. However, in other examples, a mechanical excision controller may be provided that is separate and distinct from the cauterization controller. The exemplary apparatus and controllers associated with the apparatus cart 402 are for illustrative purposes only, and other examples include vacuum pumps, patient positioning systems, robotic arms for holding various instruments, ultrasonic cutting devices and associated controllers, patient positioning controllers, and robotic surgical systems.

[0030] Figure 4 further illustrates additional instruments that may be present during an exemplary ACL repair. In particular, Figure 4 shows an exemplary guidewire or drill wire 424 and a sighting device 426. The drill wire 424 may be used to create an initial tunnel, or pilot tunnel, through the bone. In some cases, the diameter of the drill wire may be about 2.4 millimeters (mm), although larger or smaller diameters may be used with respect to the drill wire 424. The exemplary drill wire 424 is shown with enlarged portions at both ends, one enlarged portion indicating a cutting element on the distal end of the drill wire 424, and the other enlarged portion indicating a connector for coupling to the drill chuck. Once the surgeon has drilled the pilot tunnel, the surgeon and / or surgical controller 418 (described further below) may then evaluate whether the pilot tunnel matches or closely matches the planned tunnel path. If the pilot tunnel is deemed sufficient, the drill wire 424 may be used as a guide to create a full-diameter through-hole for the tunnel, or alternatively, as a guide to create a counterbore resulting from the intercondylar notch to accommodate the graft. In some cases, the drill wire may be used alone when creating the pilot tunnel, but in other cases, the surgeon may use a sight 426 to assist in guiding and positioning the drill wire 424 at the designed tunnel entrance location.

[0031] Figure 4 also shows that the exemplary system may include a calibration assembly 428. As will be described in more detail below, the calibration assembly 428 may be used to detect optical distortion in the image received by the surgical controller 418 via the arthroscope 408 and the accompanying camera head 410.

[0032] Additional tools and equipment will be present, such as a drill 429 for drilling with a drill wire 424, various reamers 431 for creating tunnel through-holes and counterbore holes, and various tools for suturing and anchoring grafts in place. Some of these additional tools and equipment are not shown in the drawings to avoid further complicating them.

[0033] This specification now moves on to describing an exemplary ACL repair workflow. The workflow can be conceptually divided into preoperative planning and intraoperative repair. The intraoperative repair workflow can be further conceptually divided into optical system calibration, model registration, intraoperative tunnel path planning, and intraoperative and tunnel construction. Each will be addressed in turn.

[0034] Preoperative planning According to various examples, ACL repair begins with preoperative imaging of the patient's knee (e.g., X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI)) including the inferior portion of the femur, the superior portion of the tibia, and related anatomical structures such as articular cartilage. The following description assumes MRI imaging, but even in this case, many different types of preoperative imaging may be used. MRI imaging can be segmented from image slices, thereby creating a volumetric or three-dimensional model of the anatomical structure. Any suitable, currently available, or later developed segmentation technique may be used to create the three-dimensional model. More specifically, in the example of ACL repair, a three-dimensional bone model of the inferior portion of the femur, including the femoral condyles, is created by specifically selecting a tunneling pathway through the femur.

[0035] Using a three-dimensional bone model, a surgical plan is developed that includes selecting a planned tunnel path through the femur, including the location of the bone opening that defines the ends of the tunnel. For an exemplary inside-out repair, the opening within the femoral notch is the entry point for the perforation, and the opening on the lateral surface of the femur is the exit point. For an outside-in repair, the entry and exit points for the perforation are reversed. Nevertheless, assuming an inside-out repair, the entry point may be selected so that it is identical to or close to the position of the natural ACL attachment to the femur within the femoral notch. In some cases, the selection of the entry point within the femoral notch may include using a Bernard & Hertel quadrant or grid placed on a fluoroscopic image, or placing a Bernard & Hertel quadrant on a simulated fluoroscopic image created from a three-dimensional bone model. Based on the use of the Bernard & Hertel quadrant, the tunnel entry point is selected. In inside-out repairs, the selection of the exit site is not restricted not only because the portion of the tunnel close to the exit site is used for graft anchor placement, but also because the exit site is located approximately in the center of the femur (considered to be from anterior to posterior), thus eliminating concerns about the thickness of the bone wall at the exit site. In some cases, a three-dimensional bone model relating to the proximal end of the tibia may also be created, allowing the surgeon to select one or more planned tunnel pathways that similarly penetrate the tibia.

[0036] The results of the preoperative planning may include a three-dimensional bone model of the distal end of the femur, a three-dimensional bone model of the proximal end of the tibia, the entry and exit points through the femur, and consequently the planned tunnel pathway through the femur, and the entry and exit points through the tibia, and consequently the planned tunnel pathway through the tibia.

[0037] In some embodiments, a three-dimensional bone model may be used to select planned values ​​for the socket or counterbore depth of the reamer into the bone. The surgeon may select the socket or counterbore depth so as not to create a socket that is too long, thereby resulting in a thin bone wall that is too close to the cortex. A bone wall that is thin and too close to the cortex can lead to ACL reconstruction failure. Furthermore, using a three-dimensional bone model, the surgeon may select the tunnel penetration diameter, counterbore diameter, etc. One or more reamers may be selected based on the diameter of the tunnel penetration diameter and / or counterbore diameter.

[0038] In some embodiments, the three-dimensional bone model may include a virtual representation of the patient's imaged bone mapped into a three-dimensional coordinate space (e.g., a virtual space). The dimensions, size, density, length, width, depth, height, shape, position, composition, orientation, etc., of the bone can be accurately represented in the three-dimensional coordinate space.

[0039] Desired post-repair flexion, as well as other surgical parameters such as homogeneity, may also be selected during preoperative planning; however, these additional surgical parameters are omitted herein to avoid overcomplicating this specification.

[0040] Intraoperative repair This specification now moves on to a description of intraoperative procedures. Intraoperative procedures include steps and procedures for setting up a surgical system to perform various repairs. However, it should be noted that some parts of intraoperative procedures (e.g., calibration of optical systems) may be performed before any port or incision is formed through the patient's skin, in fact, before the patient is brought into the operating room. Nevertheless, such steps and procedures can be considered intraoperative because they are performed in the surgical setting using the surgical instruments and equipment that will be used to perform the actual repairs.

[0041] The exemplary ACL repair is performed arthroscopically and is computer-assisted in the sense that a surgical controller 418 is used for arthroscopic navigation within the surgical site. More specifically, in the exemplary system, the surgical controller 418 provides computer assistance during ligament repair by tracking the positions of various objects within the surgical site, such as the position of the bone in the three-dimensional coordinate space of the arthroscopic field of view, the position of the reamer in the three-dimensional coordinate space of the arthroscopic field of view, and the positions of various instruments (e.g., drill wire 424, sight 426) in the three-dimensional coordinate space of the arthroscopic field of view. This specification will move on to a brief description of such tracking techniques.

[0042] Figure 5 shows a conceptual diagram of the surgical site, with various objects present within it. In particular, the visible elements in Figure 5 are the distal end of the arthroscope 408, a portion of bone 500 (e.g., femur), the bone reference point 502 within the surgical site, the touch probe 504, and the probe reference point 506. Each of these will be discussed in turn.

[0043] The distal end of the arthroscope 408 is designed and constructed to illuminate the surgical site with visible light. In the example in Figure 5, the illumination is indicated by arrow 508. The illumination supplied to the surgical site is reflected by various objects and tissues within the surgical site, and the reflected light returning to the distal end enters the arthroscope 408, propagates along the optical channels within the arthroscope 408, and finally enters the capture array in the camera head 410 (Figure 4). The image detected by the capture array in the camera head 410 is electronically transmitted to the surgical controller 418 (Figure 4) and displayed on the display device 414 (Figure 4). According to the exemplary system, the arthroscope 408 has a single optical path through the arthroscope for capturing images of the surgical site, even though a single optical path could be constructed from two or more optical components (e.g., glass rods, optical fibers). In other words, in the exemplary system and method, the computer-assisted navigation provided by the arthroscope 408, camera head 410 and surgical controller 418 is provided with an arthroscope 408 that is not a stereoscopic endoscope having two separate optical paths separated by the interocular distance at the distal end endoscope.

[0044] During surgical procedures, surgeons select arthroscopes with a field of view orientation that is beneficial for the planned surgical procedure. The field of view orientation refers to the straight line located at the center of the angle formed by the outer edge, or peripheral edge, of the endoscope's field of view. For some arthroscopes, the field of view orientation is aligned with the longitudinal central axis of the arthroscope, and such arthroscopes are referred to as "0-degree" arthroscopes (for example, the angle between the field of view orientation and the longitudinal central axis of the arthroscope is 0 degrees). For other arthroscopes, the field of view orientation forms a non-zero angle with respect to the longitudinal central axis of the arthroscope. For example, in the case of a 30° arthroscope, the field of view orientation forms a 30° angle with respect to the longitudinal central axis of the arthroscope, and this angle is measured as an obtuse angle beyond the distal end of the arthroscope. In many cases in ACL repair, surgeons select either a 30° or 45° arthroscope based on the location of the port to be created through the patient's skin. In the example shown in Figure 5, the field of view angle 510 of the arthroscope 408 forms a non-zero angle with respect to the longitudinal central axis 512 of the arthroscope 408.

[0045] Referring again to Figure 5, within the field of view of the arthroscope 408, there is a portion of bone 500, along with bone reference points 502, a touch probe 504, and a probe reference point 506. Although bone reference points 502 are shown as planar elements with patterns placed on them, other shapes may be used for bone reference points 502 (e.g., square blocks with patterns on each face of the block). Bone reference points 502 can be attached to bone 500 in any preferred form (e.g., fasteners such as screws). The pattern of the bone reference points is designed to provide information about the orientation of bone reference points 502 in the three-dimensional coordinate space of the field of view of the arthroscope 408. More specifically, the pattern is selected so that the orientation of bone reference points 502, and consequently the orientation of the base bone 500, can be determined from the images captured by the arthroscope 408 and the accompanying camera head 410 (Figure 4).

[0046] The probe reference point 506 is shown as a planar element attached to the touch probe 504. The touch probe 504 may be used to "paint" the faces of bone 500 as part of the registration of bone 500 to a three-dimensional bone model, as will be further described below, and the touch probe 504 may also be used to indicate the modified tunnel entrance position if the tunnel path is changed during surgery. Although the probe reference point 506 is shown as a planar element with a pattern placed on it, other shapes may be used for the probe reference point 506 (e.g., a square block surrounding the touch probe 504 with a pattern on each face of the block). The pattern of the probe reference point 506 is designed to provide information about the orientation of the probe reference point 506 in the three-dimensional coordinate space of the arthroscopy 408's field of view. More specifically, the pattern is selected so that the orientation of the probe reference point 506, and thus the position of the point on the touch probe 504, can be determined from the images captured by the arthroscopy 408 and the accompanying camera head 410 (Figure 4).

[0047] Other instruments within the field of view of the arthroscope 408, such as the drill wire 424 (Figure 4) and the sight 426 (Figure 4), may also have reference points, but additional instruments are not shown to avoid overcomplicating the drawings. Furthermore, in addition to, or instead of, tracking the position based on the field of view through the arthroscope 408, the position of one or more distal ends of instruments may be tracked by other methods and systems. For example, in the case of a device that extends rigidly outward from the surgical site (e.g., the sight 426 (Figure 4)), the position may be tracked by an optical array coupled to the sight, as well as by a camera 412 (Figure 4), such as a stereoscopic camera. The position of the camera 412 in three-dimensional coordinate space is then translated into the three-dimensional coordinate space of the exemplary arthroscopic field of view to determine the position of the distal end within the surgical site.

[0048] The images captured by the arthroscope 408 and its accompanying camera head 410 are subject to optical distortion in many forms. For example, the field of view between the distal end of the arthroscope 408 and the bone 500 in the surgical site is filled with fluids such as body fluid and saline used to inflate the joint. Many arthroscopes have one or more lenses at the distal end to widen the field of view, and widening the field of view creates a "fisheye" effect in the captured image. Furthermore, optical elements within the arthroscope (e.g., rod lenses) may have optical aberrations inherent in the manufacturing and / or assembly processes. Moreover, the camera head 410 may have various optical elements for focusing the received image onto a capture array, and these various optical elements may have aberrations inherent in the manufacturing and / or assembly processes.

[0049] Calibration of optical systems In the exemplary system, the endoscopic optics system is calibrated to account for various optical distortions before use in each surgical procedure. In particular, various embodiments include a system for calibrating the endoscopic optics system. Referring again to Figure 4, the exemplary system includes a surgical controller 418, an arthroscope 408, a camera head 410, and a calibration assembly 428. In particular, calibration may include placing the arthroscope 408 within the calibration assembly 428. The calibration assembly 428 holds the arthroscope 408 in a fixed relationship with respect to a calibration target on the inner surface of the calibration assembly 428. Once the distal end of the arthroscope 408 is placed within the calibration assembly 428, the exemplary method includes capturing multiple images of the calibration target, each image captured in a specific rotational relationship between the camera head 410 and the arthroscope 408, where the specific rotational relationship is with respect to the longitudinal central axis of the arthroscope 408. Using multiple images, the exemplary surgical controller 418 creates a characterization function that characterizes the optical distortion between the calibration target and the capture array in the camera head 410. The characterization function may include calibration to determine the orientation of reference point markers (e.g., bone reference point 502, probe reference point 506) visible within the surgical site by the arthroscope 408 and the attached camera head 410.

[0050] Model registration and human-participatory registration verification. The next exemplary step in the intraoperative procedure is the registration of the bone model created during preoperative planning. That is, during the preoperative planning stage, preoperative imaging (e.g., MRI) of the knee is performed, including the lower portion of the femur, the upper portion of the tibia, and relevant anatomical structures such as the articular cartilage. The preoperative imaging may be segmented so that a volumetric or three-dimensional model of the anatomical structure is created. More specifically, in the example of ACL repair, a three-dimensional bone model of the lower portion of the femur is created at the preoperative planning stage, by specifically selecting a tunneling pathway through the femur. Furthermore, the depth of the tunneling pathway and the diameter of the tunnel may be selected. The depth and diameter may be represented in a volumetric or three-dimensional model (e.g., a virtual space including axes and coordinates (e.g., X, Y, Z)).

[0051] During intraoperative repair, a three-dimensional bone model is provided to the surgical controller 418. Again, in the example of ACL repair, using specific computer-aided navigation for a tunnel pathway through the femur, a three-dimensional bone model of the lower femur is provided to the surgical controller 418. Thus, the surgical controller 418 receives the three-dimensional bone model, and assuming that the arthroscope 408 is inserted into the knee through a port through the patient's skin, the surgical controller 418 also receives video images of the femur. For the purpose of associating the three-dimensional bone model with the images received via the arthroscope 408 and camera head 410, the surgical controller 418 registers the three-dimensional bone model with the images of the femur received via the arthroscope 408 and camera head 410.

[0052] In the exemplary method, a reference point marker or bone reference point (e.g., bone reference point 502 in Figure 5) is attached to the femur. The placement of the bone reference point is such that it is located within the field of view of the arthroscope 408, but at a distance from the expected tunnel entrance / exit point through the lateral condyle. More specifically, in the exemplary case, the bone reference point is located within the intercondylar notch, above or superior to the expected location of the tunnel through the lateral condyle.

[0053] Figure 6 is an exemplary video display showing a portion of the femur and a bone reference point. The display may be shown, for example, on a display device 414 (Figure 4) associated with the apparatus cart 402 (Figure 4), or at any other preferred location. In particular, visible in Figure 6 are the femoral notch, i.e., the intercondylar notch 600, a portion of the lateral condyle 602, a portion of the medial condyle 604, and an exemplary bone reference point 606. The bone reference point 606 is a reference point comprising a cubic member. Of the six outer surfaces of the cubic member, the bottom surface is associated with a mounting mechanism (e.g., a screw). The bottom surface is close to or in contact with the bone when the bone reference point 606 is fixed in place, and is therefore not visible in the field of view of the arthroscope 408 (Figure 4). The outer surface located opposite the bottom surface includes a placement mechanism used to hold the bone reference point 606 before placement and to attach the bone reference point 606 to the base bone. Of the remaining four outer surfaces of the cubic member (only two of the remaining surfaces can be seen), each of the four outer surfaces has a machine-readable pattern on it, and in some cases, each machine-readable pattern is unique. Once positioned, the bone reference point 606 represents a fixed position on the outer surface of the bone within the field of view of the arthroscope 408, even if the position of the arthroscope 408 is moved and changed relative to the bone reference point 606. Initially, the position of the bone reference point 606 relative to the three-dimensional bone model is unknown to the surgical controller 418, so registration of the three-dimensional bone model is required.

[0054] For the purpose of associating or registering visible bones in the video image with a three-dimensional bone model, a surgical controller 418 (Figure 4) is provided, and therefore receives multiple locations on the outer surface of the bone. For example, a surgeon may touch multiple locations using a touch probe 504 (Figure 5). As described above, the touch probe 504 includes a probe reference point 506 (Figure 5) that is visible in the video image captured by the arthroscope 408 (Figure 4) and camera head 410 (Figure 4). Since the physical relationship between the distal end of the touch probe 504 and the probe reference point 506 is known by the surgical controller 418, when the surgeon touches each of the multiple locations on the outer surface of the bone, the surgical controller 418 acquires an additional “known” location on the outer surface of the bone relative to the bone reference point 606. Given that the touch probe 504 is a relatively rigid and inflexible instrument, in other examples, tracking of the touch probe 504 is done by optical tracking of an optically reflective array outside the surgical site (e.g., tracking by camera 412 (Figure 4)), but a portion of the touch probe 504 may be attached inside the surgical site.

[0055] In some cases, receiving multiple positions on the outer surface of a bone may involve the surgeon “painting” the outer surface of the bone, especially if a portion of the outer surface of the bone is exposed to view. “Painting” is a technical term that does not involve the application of color or pigment, but instead refers to the movement of the touch probe 504 as its distal end touches the bone. In some embodiments, other modes of capturing the shape of the bone relative to the bone reference point 606 may include using computed tomography (CT) and / or fluoroscopy with or without contrast agents (e.g., substances injected into an intravenous line or taken orally to make the body part being examined (e.g., bone tissue and structure) more clearly visible). A CT scan may use an X-ray beam that traverses the body part outward in a circular motion to obtain many different views of the same body part, and these images may be used to generate a two-dimensional morphology of the body part on a video display. Fluoroscopy may refer to an imaging technique that uses fluoroscopy to obtain real-time moving images of the patient’s body part. CT and / or fluoroscopy may allow for obtaining updated images of the outer surface of the bone relative to the bone reference point 606. In some embodiments, another mode for capturing the bone shape relative to the bone reference point 606 may include using an externally tracked ultrasound probe. An external tracker may be used to identify the position and orientation of images acquired by the ultrasound probe, which can be merged to identify the outer surface of the bone relative to the bone reference point 606.

[0056] The video display may be shown during the registration procedure and shows a portion of the femur, bone reference points, and an overlaid display of the three-dimensional bone model. The display may be shown, for example, on a display device 414 (Figure 4) associated with the device cart 402 (Figure 4), or at any other preferred location. To address potentially invalid registrations between the three-dimensional bone model and the bone visible in the image, in the exemplary method and system, the initial registration between the bone visible in the video image and the three-dimensional bone model is verified using a human-participatory process. More specifically, in the exemplary case, after receiving multiple positions of the outer surface of the bone visible in the image and performing an initial registration to the three-dimensional bone model, the surgical controller 418 displays a representation of the three-dimensional bone model overlaid on the portion of the bone visible in the video image, and the display and overlay are performed on the display device 414, or at any other preferred location. In other words, the exemplary surgical controller 418 overlays a representation of a three-dimensional bone model (e.g., by a polygon mesh or mesh model) onto the display of images captured by the arthroscope 408 and camera head 410, and the rotational and translational alignment of the three-dimensional bone model correlates with the bone visible in the video image when anchored by bone reference points 606.

[0057] Next, the surgeon visually examines the superimposed display of the three-dimensional bone model against the base bone visible in the video image to determine whether the registration process is appropriate. More specifically, the surgeon visually compares the superimposed display of the three-dimensional bone model against the bone portion visible in the video image to determine whether the three-dimensional bone model adequately matches the bone visible in the video image. Much like the registration process itself, the human-participatory verification of registration is an operation of nondeterminism. Slight differences between the three-dimensional bone model and the bone visible in the video image may be acceptable, but the registration process may still be considered appropriate in the sense that the three-dimensional bone model can be reliably used to assist in guiding the placement of the tunnel pathway (e.g., the femoral tunnel pathway in this case), or in the sense that the three-dimensional bone model can be reliably used to assist the surgeon in modifying the planned tunnel pathway during surgery. In such cases, the surgeon may provide a surgical controller 418, which may receive instructions indicating that the three-dimensional bone model is properly registered against the bone visible in the video image.

[0058] On the other hand, if the overlay of the three-dimensional bone model shows a misalignment with the bone visible in the video image, the surgeon may choose to restart the registration process by providing the surgical controller 418 with multiple locations on the outer surface of the bone, and the surgical controller 418 receiving these locations again. The surgical controller 418 may then perform the registration procedure anew. In other cases, the surgeon may choose to provide additional locations on the outer surface of the bone, and the surgical controller 418 may then perform the registration procedure using both the original locations received and the additional locations received after the overlay process. This process is repeated until the surgeon approves the registration. This specification now moves on to a description of intraoperative tunnel path planning.

[0059] Tunnel route plan Using a three-dimensional bone model, a surgical plan is created that includes the planned tunnel path through the bone, including the location of openings into the bone that define both ends of the tunnel. Furthermore, the depth of the planned tunnel path and the diameter of the tunnel path are selected.

[0060] In some embodiments, when planning a surgical solution, the surgeon may be presented with visual feedback (e.g., on a display device 414) of the area of ​​achievable exit point placement when positioning the tunnel. For example, an overlay may be presented on an image of the bone and may include determined or calculated knee flexion angles, shown as iso-knee flexion angle lines, to enable the surgeon to better approximate the patient's positioning.

[0061] However, in some cases, the surgeon may choose not to use the planned tunnel route, and thus may choose not to use the preoperatively planned entry point, exit point, or both. Such a choice may be based on one of several reasons. For example, during surgery, the surgeon may not be able to access the entry point for the planned tunnel route, and therefore may need to move the entry point to ensure adequate access. Another example is that during the intraoperative procedure, the surgeon may determine that the planned tunnel entry point is misaligned with the attachment point of the natural ACL to the femur. Furthermore, during the intraoperative procedure, the surgeon may determine that the tunnel entry point is too close to the posterior wall of the femur, increasing the likelihood of bone fracture, sometimes referred to as "posterior wall rupture." Regardless of the reason for choosing to change the tunnel route, in the exemplary system, the surgical controller 418 allows the surgeon to select a modified tunnel entry point, a modified tunnel exit point (if necessary), and thus a modified tunnel route through the bone during surgery.

[0062] Figure 7 is an exemplary video display showing an intraoperative modification of the tunnel pathway according to at least some embodiments. The display may be shown, for example, on a display device 414 (Figure 4) associated with the apparatus cart 402 (Figure 4), or at any other preferred location. In particular, Figure 7 shows a portion of the bone in a video image captured by the arthroscope 408 (Figure 4) superimposed on a mesh model of a three-dimensional bone model and on a preoperatively selected planned tunnel pathway 700, including a planned tunnel entrance 702 and a planned tunnel exit 704. However, for various reasons, the surgeon may choose to modify the tunnel entrance position and / or tunnel exit position, and thus the planned tunnel pathway. Thus, the surgeon may provide the modified tunnel entrance position or simply the modified tunnel entrance 706 to the surgical controller 418 (Figure 4), and the surgical controller 418 may receive it. Providing a modified tunnel entrance 706 may involve the surgeon touching a proposed location on the bone shown in the video image using a tracking tool such as a touch probe 504 (Figure 5) or a sight 426 (Figure 4). In some cases, the surgeon may select a modified tunnel entrance 706 based solely on what the surgeon sees with respect to the bone shown in the video image. In a more specific example, the surgeon may select and provide a modified tunnel entrance 706 based on a location that the sight 426 can reach. In yet another case, the surgical controller 418 may generate a simulated fluoroscopic image from a three-dimensional bone model and project a Bernard & Hertel quadrant or grid onto the simulated fluoroscopic image. The surgeon may then select a modified tunnel entrance 706 using the additional guidance provided by the Bernard & Hertel quadrant.

[0063] Tunnel construction Once the modified tunnel path 710 is selected, the next step in the exemplary method is to actually create the tunnel. In most cases, tunnel creation is a multi-step process, involving drilling an initial tunnel, or pilot tunnel, using a drill wire (e.g., drill wire 424 (Figure 4)), and then using the drill wire as a guide wire for one or more reamers to increase the diameter of the pilot tunnel and form an actual tunnel of full diameter through the bone. In some cases, the actual tunnel has a counterbore associated with the intercondylar notch to accommodate the width of the autograft, in which case an additional reamer may be used to create the counterbore.

[0064] Software and hardware Figure 8 shows a high-level component diagram of an exemplary system architecture 800 according to a particular embodiment of the present disclosure. In some embodiments, the system architecture 800 may include a computing system 802 that is communicably coupled via a network 812. The computing system 802 may be a real-time software platform, may include privacy software or protocols, or may include security software or protocols. The computing system 802 may include one or more processing units, memory devices, and / or network interface cards. The network interface cards may enable communication via wireless protocols for transmitting data over short distances, such as Bluetooth, ZigBee, and NFC. Furthermore, the network interface cards may also enable communication of data via wired protocols over short or long distances, and in one example, the computing system 802 may communicate with the network 812. The network 812 may be a public network (e.g., connected to the Internet via wired (Ethernet) or wireless (Wi-Fi)), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof. In some embodiments, the network 812 may also include nodes on the Internet of Things (IoT).

[0065] The computing system 802 may be any suitable computing device, such as a display, laptop, tablet, smartphone, or embedded computer device having a computer. The computing system 802 may include a display capable of presenting a user interface for an application. The application may be stored in one or more memory devices of the computing system 802 and implemented as computer instructions that can be executed by one or more processing units of the computing system 802.

[0066] The user interface may present the user with various screens that present various views, including recommendations for performing a repositioning surgery, based on one or more probabilities of success or failure detected based on one or more images using one or more machine learning models. The user interface may also present navigation guidance, including approach angles for drilling a tunnel and / or inserting tools to remove hardware included in previous ACL reconstruction surgery and / or any appropriate surgical procedure. The user interface may also present overlays that include graphical elements indicating insertion points for inserting tools and / or insertion points for drilling a tunnel, approach angles for tools and / or drills, information related to the set of tools used to perform the repositioning surgery, and step-by-step instructions on how to remove hardware attached to a person's body. The computing system 802 may also include instructions stored on one or more memory or memory devices that, when executed by one or more processors or processing units of the computing system 802, perform any of the operations described herein.

[0067] In some embodiments, the computing system 802 may include one or more servers forming a distributed computing system, which may include a cloud computing system. The servers may be rack-mount servers, routers, personal computers, portable digital assistants, mobile phones, laptop computers, tablet computers, cameras, video cameras, netbooks, desktop computers, media centers, any other devices capable of functioning as servers, or any combination thereof. Each server may include one or more processing units, memory devices, data storage, or network interface cards. The servers may communicate with each other via any suitable communication protocol.

[0068] The computing system 802 may run an artificial intelligence engine 804 and one or more machine learning models 806, as further described herein. That is, the computing system 802 may run an artificial intelligence (AI) engine 804 using one or more machine learning models 806 to implement at least one of the embodiments disclosed herein. The computing system 802 may also include a database 810 for storing data, knowledge, and data structures used to implement various embodiments. For example, the database 810 may store anatomical images, information about hardware used in reconstructive surgery, information about tools used to remove hardware, information about hardware repositioning and removal, instructions related to repositioning surgery, and so on.

[0069] In some embodiments, the computing system 802 may include a training engine 808 capable of generating one or more machine learning models 806. Although illustrated separately from the AI ​​engine 804, the training engine 808 may, in some embodiments, be included in the AI ​​engine 804 running on the computing system 802. In some embodiments, the AI ​​engine 804 may use the training engine 808 to generate machine learning models 806 trained to perform inference and / or predictive actions. The machine learning models 806 may, among other things, be trained to predict the probability of success in performing a repositioning surgery, the probability of collisions being detected if a repositioning surgery is performed, and navigation guidance for performing a repositioning surgery. One or more machine learning models 806 may be generated by the training engine 808 and may be implemented with computer instructions executable by one or more processing units of the training engine 808. To generate one or more machine learning models 806, the training engine 808 may train one or more machine learning models 806. One or more machine learning models 806 may be used by any of the methods described herein.

[0070] In some embodiments, the training engine 808 may be implemented as computer instructions stored in one or more memory devices and executed by one or more processing units of the computing system 802. In some embodiments, the training engine 808 may be isolated from the computing system 802 and may be a rack-mount server, router, personal computer, portable digital assistant, smartphone, laptop computer, tablet computer, netbook, desktop computer, Internet of Things (IoT) device, any other desired computing device, or any combination thereof. The training engine 808 may be cloud-based, may be a real-time software platform, may include privacy software or protocols, or may include security software or protocols.

[0071] To generate one or more machine learning models 806, the training engine 808 may train one or more machine learning models 806. The training engine 808 may use, among other things, a base training dataset, which includes, as inputs, labeled hardware data (e.g., images of hardware (bolts, screws, etc.) associated with surgical procedures on parts of the human body). One or more combinations of inputs may be mapped to outputs associated with one or more tools used to remove the hardware (e.g., including information related to the product, diameter, length, guidewire, driver type, head shape, part number, configuration, etc.). In addition, the training data may include labeled inputs of images of parts of the user's body, which include one or more outputs of drilled tunnels for installed hardware and / or reconstructive surgery performed, as well as navigation guidance for performing repositioning surgery. The navigation guidance may be based on the type of hardware identified in one or more images of the patient's body part, and / or one or more tunnels identified in one or more images. Navigation guidance may provide information related to one or more tools used to remove one or more identified hardware pieces within a patient's body, information related to approach angles for inserting one or more tools to remove one or more identified hardware pieces, graphics for tool placement / insertion, graphics for entry points, and step-by-step instructions for performing repositioning surgery. Furthermore, machine learning model 806 may be trained with training data that includes input of one or more images (CT images, X-ray images, etc.) of parts of the body of patients who have undergone reconstructive surgery, and output of the probability of success in performing repositioning surgery (e.g., hardware removal, drilling a new tunnel, etc.). Furthermore, machine learning model 806 may be trained with training data that includes input of one or more images of parts of the body of patients who have undergone reconstructive surgery, and output of the probability of collision with hardware (e.g., screws) that has been placed if repositioning surgery is performed. In some examples, machine learning model 806 may be trained to segment bone and hardware in images.The machine learning model 806 can identify bones present in the image and identify hardware present in the image by overlaying the identity / name of the bones and / or hardware.

[0072] In some embodiments, the training engine 808 may use supervised learning to train one or more machine learning models 806. Labeled data may be used by the training engine 808 to train one or more machine learning models 806. In some examples, unsupervised learning may be used by the training engine 808 to train one or more machine learning models 806. In some examples, reinforcement learning may be used by the training engine 808 to train one or more machine learning models 806.

[0073] Training may involve optimizing one or more parameters of one or more machine learning models. For example, parameters may include several hidden layers, the number of nodes used by each hidden layer, weights, the type of activation function used, the type of loss function used, batch size, pooling size, and the number of training iterations.

[0074] One or more machine learning models 806 may refer to model artifacts generated by the training engine 808 using training data, which includes training inputs and corresponding target outputs. The training engine 808 may find patterns within the training data, and these patterns generate machine learning models 806 that map the training inputs to target outputs and capture these patterns. Although illustrated as being included in the computing system 802, in some embodiments the artificial intelligence engine 804, the database 810, and / or the training engine 808 may reside on separate computing systems and / or servers.

[0075] One or more machine learning models 806 may, for example, include a single level of linear or nonlinear operation (e.g., a support vector machine (SVM)), or the machine learning model 806 may be a deep network, i.e., a machine learning model that includes multiple levels of nonlinear operation. Examples of deep networks include generative adversarial networks, convolutional neural networks, iterative neural networks with one or more hidden layers, and fully connected neural networks (e.g., each artificial neuron may transmit its output signal to the inputs of the remaining neurons, as well as to itself). For example, a machine learning model may include a number of layers or hidden layers that perform computations (e.g., dot products) using various neurons. In some embodiments, one or more machine learning models 806 may be trained through supervised learning, unsupervised learning, and / or reinforcement learning.

[0076] Figure 9 is an illustrative flowchart of Method 900 for providing navigation guidance for repositioning surgery using a machine learning model, according to at least some embodiments. Method 900 may be carried out by one or more processors or processing units, such as those included in the surgical controller 418 and / or computing system 802. The method may be implemented in computer instructions stored in one or more memories, executed by one or more processors.

[0077] In block 902, the processor may receive one or more images related to the surgical site of the patient. One or more images may be received from an atroscope. One or more images may be computed tomography images containing details of a part of the patient's body. One or more images may include the patient's bone, tissue, and / or hardware (e.g., screws, bolts, nails, plates, etc.) placed during reconstructive surgery.

[0078] In block 904, the processor may, based on one or more images, use one or more trained machine learning models to identify one or more hardware pieces attached to a surgical site. One or more machine learning models may be trained to identify hardware placed at a surgical site and may output the type of hardware, the manufacturer of the hardware, the length of the hardware, the diameter of the hardware, the type of tool used to install / uninstall the hardware, guide wires, the head shape of the hardware, the composition of the hardware, or some combination thereof. The machine learning models may be trained to identify the installation angle of one or more hardware pieces, the tunnel orientation of a part of the patient's body, the entry point for repositioning surgery, the thickness of the bone wall, etc.

[0079] In block 906, the processor may use one or more machine learning models, based on one or more hardware pieces, to determine navigation guidance for uninstalling one or more hardware pieces during a repositioning surgery. In some examples, the navigation guidance may include at least an approach angle for uninstalling one or more hardware pieces. In some examples, the navigation guidance may include an overlay present on the display. The overlay may include one or more graphical elements related to the navigation guidance. The overlay may be presented simultaneously on one or more images of parts of the patient's body and may be presented in real time along with step-by-step instructions on how to perform the repositioning surgery as the surgeon operates on the patient. Furthermore, the display may present an alert if the surgeon's tool deviates by a threshold amount from the recommended navigation guidance. The alert may indicate that the tool is colliding with an anatomical feature of the patient and / or a hardware piece attached to part of the patient's body. The navigation guidance may be presented in real time to enable the surgeon to correctly perform the repositioning surgery.

[0080] In block 908, the processor may transmit navigation guidance to a computing device associated with the surgeon performing the repositioning surgery. The navigation guidance may be presented via a display, such as the user interface of the display device 414 and / or computing system 802. The navigation guidance may indicate which tools to use at each step of the surgical procedure, how to insert the tools, how to remove specific hardware fragments (identified by a machine learning model), and so on.

[0081] In some cases, the processor may use one or more machine learning models to determine the probability of successfully uninstalling one or more hardware pieces during a repositioning surgery. In some cases, the processor may use that probability to determine navigation guidance for uninstalling one or more hardware pieces during a repositioning surgery. Furthermore, in some cases, the processor may determine the probability of collision with previously installed hardware and / or previously drilled tunnels when the surgeon performs a repositioning surgery.

[0082] In some embodiments, the processor may send one or more recommendations regarding repositioning surgery based on its probability. For example, if the probability of success meets a threshold probability (e.g., a success probability greater than 95%), the recommendation may suggest performing repositioning surgery. The probability of success may be determined based on comparing a data corpus of similar images of reconstructive surgeries having similar anatomical features and similarly used hardware. The data corpus may indicate whether or not repositioning surgery was successful in patients in similar positions. Furthermore, the recommendation may include instructions regarding a specific approach angle for uninstalling one or more identified hardware pieces. The approach angle may be based on the angle at which one or more hardware pieces are attached to one or more anatomical features of the patient, one or more types of hardware, one or more manufacturers of the hardware, the orientation of the drilled tunnel, etc. Furthermore, the recommendation may include instructions specifying one or more tools to be used to uninstall one or more identified hardware pieces.

[0083] In some cases, if the probability of success does not meet a threshold probability (e.g., less than 95% probability of success), recommendations may not advise proceeding with repositioning surgery.

[0084] In some cases, the processor may use one or more trained machine learning models to determine one or more tools to be used to uninstall one or more hardware pieces during a re-operation. For example, the machine learning model may be trained on hardware pieces that match hardware pieces associated with the tools used to install / uninstall those hardware pieces.

[0085] Figure 10 shows an exemplary computer system 1000. In one example, computer system 1000 may correspond to the surgical controller 418, the computing system 802, a tablet device in the operating room, or any other system implementing any or all of the various methods described herein. Computer system 1000 may be connected to other computer systems (e.g., network connections) in a local area network (LAN), intranet, and / or extranet (e.g., the equipment cart 402 network), or, at specific times, in the internet (e.g., when not being used in surgical procedures). Computer system 1000 may be a server, a personal computer (PC), a tablet computer, or any device capable of executing a set of instructions (sequential or otherwise) specifying the actions to be performed by that device. Furthermore, although only a single computer system is illustrated, the term “computer” shall also be interpreted to include any collection of multiple computers individually or collectively executing a set (or more sets) of instructions to perform any one or more of the methods described herein.

[0086] The computer system 1000 includes a processing unit 1002, main memory 1004 (e.g., read-only memory (ROM), flash memory, synchronous DRAM (SDRAM), and other dynamic random access memory (DRAM)), static memory 1006 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 1008, which communicate with each other via a bus 1010.

[0087] The processing unit 1002 represents one or more general-purpose processing units, such as a microprocessor, a central processing unit, or the same. More specifically, the processing unit 1002 may be a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set, or a processor implementing a combination of instruction sets. The processing unit 1002 may also be one or more application-specific processing units, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the same. The processing unit 1002 is configured to execute instructions to perform any operations and steps described herein. After being programmed with specific instructions, the processing unit 1002, and by extension the entire computer system 1000, becomes an application-specific device, such as the surgical controller 418 and / or computing system 802.

[0088] The computer system 1000 may further include a network interface device 1012 for communicating with any suitable network 812 (e.g., the device cart 402 network). The computer system 1000 may also include a video display 1014 (e.g., a display device 414), one or more input devices 1016 (e.g., a microphone, keyboard, and / or mouse), and one or more speakers 1018. In one exemplary example, the video display 1014 and the input devices 1016(or more) may be combined into a single component or device (e.g., an LCD touchscreen).

[0089] The data storage device 1008 may include a computer-readable storage medium 1020 containing instructions 1022 that embody any one or more of the methodologies or functions described herein (for example, instructions 1022 that implement any method and any function performed by any device and / or component illustrated and described herein). The instructions 1022 may also reside, all or at least partially, in the main memory 1004 and / or the processing unit 1002 when executed by the computer system 1000. Thus, the main memory 1004 and the processing unit 1002 also constitute a computer-readable medium. In certain cases, the instructions 1022 may be further transmitted or received over a network via the network interface device 1012.

[0090] Although computer-readable storage medium 1020 is shown in exemplary examples as a single medium, the term “computer-readable storage medium” shall be interpreted to include one or more mediums (e.g., centralized or distributed databases, and / or associated caches and servers) storing one or more instruction sets. The term “computer-readable storage medium” shall also be interpreted to include any medium capable of storing, encoding, or carrying instruction sets for machine execution, or instruction sets that cause a machine to execute any one or more methodologies in this disclosure. Accordingly, the term “computer-readable storage medium” shall be interpreted to include, but not be limited to, solid-state memory, optical media, and magnetic media.

[0091] The above discussion is intended to illustrate the principles and various embodiments of the present invention. Many variations and modifications will be apparent to those skilled in the art once the above disclosure is fully understood. The following claims are intended to be construed as encompassing all such variations and modifications.

[0092] Clauses: 1. A computer implementation method, Receiving one or more images related to the surgical site of the patient, Based on the aforementioned one or more images and using one or more trained machine learning models, identify one or more hardware pieces attached to the surgical site, Based on the one or more hardware pieces, the one or more machine learning models are used to determine navigation guidance for uninstalling the one or more hardware pieces during repositioning surgery. A computer implementation method comprising transmitting the navigation guidance to a computing device associated with the surgeon performing the repositioning surgery.

[0093] 2. A computer implementation method described in any provision of this disclosure, A computer implementation method further comprising using one or more machine learning models to determine the probability of success in uninstalling one or more hardware pieces during the repositioning surgery.

[0094] 3. A computer implementation method described in any provision of this disclosure, A computer implementation method further comprising determining the navigation guidance for uninstalling one or more hardware pieces during the repositioning surgery based on the aforementioned probability.

[0095] 4. A computer implementation method described in any provision of this disclosure, A computer implementation method further comprising transmitting one or more recommendations regarding the repositioning surgery based on the aforementioned probabilities.

[0096] 5. A computer implementation method described in any provision of this disclosure, A computer implementation method further comprising using one or more trained machine learning models to determine one or more tools used to uninstall the one or more hardware pieces during the repositioning surgery.

[0097] 6. A computer implementation method described in any provision of this disclosure, wherein the navigation guidance includes at least an approach angle for uninstalling the one or more hardware pieces.

[0098] 7. A computer implementation method described in any provision of this disclosure, wherein the navigation guidance includes an overlay present on the display of the computing device, and the overlay includes one or more graphical elements related to the navigation guidance.

[0099] 8. A system, Memory for storing instructions, A processing unit that is communicatively coupled to the memory, wherein the processing unit executes the instruction, Receiving one or more images related to the surgical site of the patient, Based on the aforementioned one or more images, one or more trained machine learning models are used to identify one or more hardware pieces attached to the surgical site, Based on the one or more hardware pieces, the one or more machine learning models are used to determine navigation guidance for uninstalling the one or more hardware pieces during repositioning surgery. A system comprising a processing unit that transmits the navigation guidance to a computing device associated with the surgeon performing the repositioning surgery.

[0100] 9. A system described in any provision of this disclosure, wherein the processing device is A system that further uses one or more machine learning models to determine the probability of success in uninstalling one or more hardware pieces during the repositioning surgery.

[0101] 10. A system described in any provision of this disclosure, wherein the processing device is A system that further determines the navigation guidance for uninstalling one or more hardware pieces during the repositioning surgery based on the aforementioned probability.

[0102] 11. A system described in any provision of this disclosure, wherein the processing device is A system that further transmits one or more recommendations regarding the repositioning surgery based on the aforementioned probabilities.

[0103] 12. A system described in any provision of this disclosure, wherein the processing device is A system that further uses one or more machine learning models to determine one or more tools to use to uninstall the one or more hardware pieces during the repositioning surgery.

[0104] 13. A system described in any provision of this disclosure, wherein the navigation guidance includes at least an approach angle for uninstalling the one or more hardware pieces.

[0105] 14. A system described in any provision of this disclosure, wherein the navigation guidance includes an overlay present on the display of the computing device, and the overlay includes one or more graphical elements related to the navigation guidance.

[0106] 15. A tangible, non-temporary computer-readable medium for storing instructions, which, at execution time, can be used by a processing unit. Receiving one or more images related to the surgical site of the patient, Based on the aforementioned one or more images, one or more trained machine learning models are used to identify one or more hardware pieces attached to the surgical site, Based on the one or more hardware pieces, the one or more machine learning models are used to determine navigation guidance for uninstalling the one or more hardware pieces during repositioning surgery. A computer-readable medium that transmits the aforementioned navigation guidance to a computing device associated with the surgeon performing the repositioning surgery.

[0107] 16. A computer-readable medium described in any provision of this disclosure, wherein the processing device is A computer-readable medium further using one or more machine learning models to determine the probability of success in uninstalling one or more hardware pieces during the repositioning surgery.

[0108] 17. A computer-readable medium described in any provision of this disclosure, wherein the processing device is A computer-readable medium further determines, based on the aforementioned probability, the navigation guidance for uninstalling one or more hardware pieces during the repositioning surgery.

[0109] 18. A computer-readable medium described in any provision of this disclosure, wherein the processing device is A computer-readable medium further transmits one or more recommendations regarding the repositioning surgery based on the aforementioned probabilities.

[0110] 19. A computer-readable medium described in any provision of this disclosure, wherein the processing device is A computer-readable medium that further uses one or more machine learning models to determine one or more tools used to uninstall the one or more hardware pieces during the repositioning surgery.

[0111] 20. A computer-readable medium as described in any provision of this disclosure, wherein the navigation guidance includes an overlay present on the display of the computing device, and the overlay includes one or more graphical elements related to the navigation guidance.

Claims

1. A computer implementation method, Receiving one or more images related to the surgical site of the patient, Based on the aforementioned one or more images and using one or more trained machine learning models, identify one or more hardware pieces attached to the surgical site, Based on the one or more hardware pieces, the one or more machine learning models are used to determine navigation guidance for uninstalling the one or more hardware pieces during repositioning surgery. A computer implementation method comprising transmitting the navigation guidance to a computing device associated with the surgeon performing the repositioning surgery.

2. A computer implementation method according to claim 1, A computer implementation method further comprising using one or more machine learning models to determine the probability of success in uninstalling one or more hardware pieces during the re-installation surgery.

3. A computer implementation method according to claim 2, A computer implementation method further comprising determining the navigation guidance for uninstalling one or more hardware pieces during the repositioning surgery based on the aforementioned probability.

4. A computer implementation method according to claim 2, A computer implementation method further comprising transmitting one or more recommendations regarding the repositioning surgery based on the aforementioned probabilities.

5. A computer implementation method according to claim 1, A computer implementation method further comprising using one or more trained machine learning models to determine one or more tools used to uninstall the one or more hardware pieces during the repositioning surgery.

6. The computer implementation method according to claim 1, wherein the navigation guidance includes at least an approach angle for uninstalling one or more hardware pieces.

7. The computer implementation method according to claim 1, wherein the navigation guidance includes an overlay present on the display of the computing device, and the overlay includes one or more graphical elements related to the navigation guidance.

8. It is a system, Memory for storing instructions, A processing unit that is communicatively coupled to the memory, wherein the processing unit executes the instruction, Receiving one or more images related to the surgical site of the patient, Based on the aforementioned one or more images, one or more trained machine learning models are used to identify one or more hardware pieces attached to the surgical site, Based on the one or more hardware pieces, the one or more machine learning models are used to determine navigation guidance for uninstalling the one or more hardware pieces during repositioning surgery. A system comprising a processing unit that transmits the navigation guidance to a computing device associated with the surgeon performing the repositioning surgery.

9. The system according to claim 8, wherein the processing apparatus is A system that further uses one or more machine learning models to determine the probability of success in uninstalling one or more hardware pieces during the repositioning surgery.

10. The system according to claim 9, wherein the processing apparatus is A system that further determines the navigation guidance for uninstalling one or more hardware pieces during the repositioning surgery based on the aforementioned probability.

11. The system according to claim 9, wherein the processing apparatus is A system that further transmits one or more recommendations regarding the repositioning surgery based on the aforementioned probabilities.

12. The system according to claim 8, wherein the processing apparatus is A system that further uses one or more machine learning models to determine one or more tools used to uninstall the one or more hardware pieces during the repositioning surgery.

13. The system according to claim 8, wherein the navigation guidance includes at least an approach angle for uninstalling one or more hardware pieces.

14. The system according to claim 8, wherein the navigation guidance includes an overlay present on the display of the computing device, and the overlay includes one or more graphical elements related to the navigation guidance.

15. A tangible, non-temporary, computer-readable medium for storing instructions, which, at execution time, the processing unit, Receiving one or more images related to the surgical site of the patient, Based on the aforementioned one or more images, one or more trained machine learning models are used to identify one or more hardware pieces attached to the surgical site, Based on the one or more hardware pieces, the one or more machine learning models are used to determine navigation guidance for uninstalling the one or more hardware pieces during repositioning surgery. A non-temporary computer-readable medium that causes the navigation guidance to be transmitted to a computing device associated with the surgeon performing the repositioning surgery.

16. A computer-readable medium according to claim 15, wherein the processing apparatus is The computer-readable medium according to claim 15, further comprising using one or more machine learning models to determine the probability of success in uninstalling one or more hardware pieces during the re-installation surgery.

17. A computer-readable medium according to claim 16, wherein the processing device is A computer-readable medium further determines, based on the aforementioned probability, the navigation guidance for uninstalling one or more hardware pieces during the repositioning surgery.

18. A computer-readable medium according to claim 16, wherein the processing device is A computer-readable medium further transmits one or more recommendations regarding the repositioning surgery based on the aforementioned probabilities.

19. A computer-readable medium according to claim 15, wherein the processing apparatus is A computer-readable medium that further uses one or more machine learning models to determine one or more tools used to uninstall the one or more hardware pieces during the repositioning surgery.

20. The computer-readable medium according to claim 15, wherein the navigation guidance includes an overlay displayed on the display of the computing device, and the overlay includes one or more graphical elements related to the navigation guidance.

Images