Fast Dynamic Registration Using Augmented Reality

Abstract

A method for fast dynamic registration using augmented reality includes aligning a model point cloud to an object point cloud, the method including obtaining a selection of an origin for the model point cloud as a sampled surface point on the object in an established collection of sampled points of the object, the collection forming the object point cloud, obtaining one or more other sampled surface points of the object and including them in the collection, determining an initial pose of the model point cloud based on the collection of sampled points, obtaining and updating the collection to include additional sampled surface points, determining a fitting of the model point cloud to the object point cloud based on the updated collection, determining a registration accuracy of the fitting of the model point cloud to the object point cloud, and performing processing based on the determined registration accuracy.

Description

[Technical field]

[0001] The present disclosure relates to fast dynamic registration using augmented reality. [Background technology]

[0002] Registration in medical imaging refers to the process of finding the relationship of one coordinate frame / system to another. This relationship is called a "transformation." In many applications of registration, two point clouds represent the same object, and registration is about aligning the points in one coordinate frame to the points in the other coordinate frame. In medical imaging applications, there may be an anatomical model, e.g., a model of a patient's bones, that is presented in one coordinate frame and registered to the patient's actual anatomy in another coordinate frame. The anatomical model of the patient's anatomy is often generated by a computed tomography (CT) scan or other imaging technique, and presents the features of the patient's anatomy (e.g., bones) for surgery / surgical planning on the model. During surgery involving the patient's anatomy, the model is registered to the image / view of the patient's anatomy that the model represents. The "array" may be rigidly fixed to the patient, for example to the patient's bones, to act as trackable markers for the imaging system, which may then be used to verify the transformation and know the exact location of the patient's anatomy and features in space. A registration probe may be used to contact the surface of the anatomy (e.g. bone) and assign position coordinates to the probe tip at each such registered point to generate a surface point cloud of the patient's anatomy. A model may then be registered to those features. Summary of the Invention

[0003] The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a computer-implemented method. The method includes registering the model point cloud to the object point cloud. The registering includes obtaining a user selection of an origin for the model point cloud; the origin is a sampled surface point on the object and is the first point included in the established collection of sample points of the object; The collection forms a point cloud of objects, To align, obtaining one or more other sampled surface points of the object and including the obtained one or more other sampled surface points in a collection; determining an initial pose of a model point cloud based on a collection of sample points of the object; obtaining additional sampled surface points at the object and updating the collection of sample points to include the additional sampled surface points, thereby providing an updated collection of sample points; determining a fitting of the model point cloud to the object point cloud based on the updated collection of sample points of the object; determining a registration accuracy of a fitting of the model point cloud to the object point cloud; and performing processing based on the determined registration accuracy.

[0004] Additionally, a computer system is provided that includes a memory and a processor in communication with the memory. The computer system is configured to carry out the method. The method includes registering the model point cloud to the object point cloud. The registering includes obtaining a user selection of an origin for the model point cloud; the origin is a sampled surface point on the object and is the first point included in the established collection of sample points of the object; The collection forms a point cloud of objects, To align, obtaining one or more other sampled surface points of the object and including the obtained one or more other sampled surface points in a collection; determining an initial pose of a model point cloud based on a collection of sample points of the object; obtaining additional sampled surface points at the object and updating the collection of sample points to include the additional sampled surface points, thereby providing an updated collection of sample points; determining a fitting of the model point cloud to the object point cloud based on the updated collection of sample points of the object; determining a registration accuracy of a fitting of the model point cloud to the object point cloud; and performing processing based on the determined registration accuracy.

[0005] Still further, a computer program product is provided that includes a computer readable storage medium readable by a processing circuit, the computer readable storage medium storing instructions for performing a method by the processing circuit. The method includes registering the model point cloud to the object point cloud. The registering includes obtaining a user selection of an origin for the model point cloud; the origin is a sampled surface point on the object and is the first point included in the established collection of sample points of the object; The collection forms a point cloud of objects, To align, obtaining one or more other sampled surface points of the object and including the obtained one or more other sampled surface points in a collection; determining an initial pose of a model point cloud based on a collection of sample points of the object; obtaining additional sampled surface points at the object and updating the collection of sample points to include the additional sampled surface points, thereby providing an updated collection of sample points; determining a fitting of the model point cloud to the object point cloud based on the updated collection of sample points of the object; determining a registration accuracy of a fitting of the model point cloud to the object point cloud; and performing processing based on the determined registration accuracy.

[0006] In an embodiment, the model point cloud comprises an anatomical structure model point cloud and the object comprises a patient anatomy.

[0007] In an embodiment, performing the process includes repeating obtaining additional sampled surface points, determining the fitting, and determining the registration accuracy one or more times based on the determined registration accuracy being less than a preconfigured threshold level of accuracy. In an embodiment, the iterating stops based on the determined registration accuracy being at least a preconfigured threshold level of accuracy. In an embodiment, based on stopping the iterating, the determined fitting of the bone model point cloud to the point cloud of the patient anatomy provides a registration of the bone model point cloud to the point cloud of the patient anatomy, and the method further includes determining and digitally presenting one or more instructions of surgical guidance to the surgeon.

[0008] In some embodiments, obtaining the user selection of the origin includes providing a bone model augmented reality (AR) element to the patient anatomy that overlays a portion of the view. The view may show a registration probe, and the bone model AR element may be provided at a fixed position relative to a probe tip of the probe. User movement of the probe may change the position of the bone model AR element, and the user selection of the origin may include a user positioning and orienting the bone model AR element in the view to overlay the patient anatomy by contacting the probe tip with the patient anatomy and then providing an input (e.g., mouse click, button press, verbal confirmation, etc.) that selects the origin as the position of the probe tip that contacts the patient anatomy. Additionally, in some embodiments, obtaining the user selection of the origin further includes providing a probe axis AR element to the patient anatomy that overlays another portion of the view. The probe axis AR element may comprise an axis line extending from the probe at a first position (e.g., tip) and away from the probe tip towards a second position. , the axis line represents the axis of the probe / probe tip.

[0009] Additionally or alternatively, determining a fitting of the bone model point cloud to the point cloud of the patient anatomy based on the updated collection of sample points of the patient anatomy may include: performing a coarse fitting of the bone model point cloud to the patient anatomy point cloud using the updated collection of sample points of the patient anatomy; and performing a fine fitting of the bone model point cloud to the point cloud of the patient anatomy using the updated collection of sample points of the patient anatomy based on performing the coarse fitting. In an embodiment, performing the coarse fitting comprises applying a random sample consensus (RANSAC) algorithm, and / or performing the fine fitting comprises applying an iterative closest point (ICP) algorithm.

[0010] Additionally or alternatively, determining the initial pose of the bone model point cloud may also utilize coarse fitting and / or fine fitting. For example, determining the initial pose of the bone model (e.g., after the first two or three sampled points) may involve: performing a coarse fitting of the bone model point cloud to the patient anatomical structure point cloud by applying a random sample consensus (RANSAC) algorithm; Based on performing the coarse fitting, performing a fine fitting of the bone model point cloud to the patient anatomy point cloud by applying an iterative closest point (ICP) algorithm.

[0011] Additional features and advantages are realized through the concepts described herein. [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates an example environment for point sampling on a patient anatomy, according to aspects described herein. [Diagram 2] 1 illustrates an example of AR-assisted user bone model origin selection and model positioning relative to patient anatomy, according to aspects described herein. [Diagram 3] 1 illustrates an example of AR-assisted user bone model origin selection and model positioning relative to patient anatomy, according to aspects described herein. [Figure 4] An example is given of solving inferring an impossible solution by solving for a local minimum when fitting a model to a patient anatomy. [Diagram 5] 1 illustrates an example process for fitting a bone model point cloud to a surface point cloud of a patient anatomy, according to aspects described herein. [Figure 6]13 illustrates an example visualization of updating a registration transformation for a bone model point cloud based on additional sampled points according to aspects described herein. [Figure 7] 1 illustrates an example of AR-assisted sample point identification according to an embodiment described herein. [Figure 8] 1 illustrates an example of a computer system and associated devices for incorporating and / or using aspects described herein. [Figure 9] 1 illustrates an example of a smart eyewear device. [Figure 10] 13 illustrates an exemplary restriction of not displaying the tracking array as a virtual object. [Figure 11] Indicates to what extent the orientation of the registration probe should not affect the coordinates of the sampled points. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The aspects described in the present application may be particularly pointed out and distinctly claimed, and the objects, features, and advantages of the present disclosure will be apparent from the detailed description given in this application when taken in conjunction with the accompanying drawings.

[0014] Existing approaches for registration in surgical planning and other applications have drawbacks. For example, they often require a relatively large number of sampled points (e.g., 40 or more) per bone using a probe. Furthermore, they often require points to be sampled in a specified order, requiring the surgeon to follow a guidance application with on-screen prompts to sample specific points indicated by the system. Furthermore, existing registration workflows require all of the points to be sampled, regardless of whether they are needed or would actually improve registration accuracy in order to proceed to the next step in the workflow. According to a specific example where surgery is being performed on a knee, certain systems require the sampling of 40 points on each of the femur and tibia (80 points total) in order to proceed to the next step in the workflow. Furthermore, some require the sampling of points that are not in close proximity to many of the other collected points, which is disruptive to the workflow. For example, some systems require point samples at the distal end of the femur, the proximal point of the femur, the medial and lateral malleoli, which are different anatomical structures.

[0015] Many current registration algorithms rely on "Iterative Closest Point" calculations to determine the transformation (which may be an approximation, e.g., in image-less systems) of the tracked object to its pre-operative reference frame. The ICP algorithm attempts to minimize the difference between two point clouds. The ICP algorithm may output results that do not represent the smallest difference (i.e., a local minimum) between the point clouds.

[0016] In addition to those mentioned above (and using the example of knee surgery), some approaches require leg manipulation, where the leg must be removed from the knee positioner, manipulated, and then re-constrained to align the hip center.

[0017] Many robotics and navigation companies use infrared (IR) based tracking cameras that require 3D model renderings of the surgical theater, including patient anatomy and surgical instruments. These renderings of objects can contain glitches and often do not include a contextual background (i.e., objects appear to be floating in space on a blank screen). These issues with existing digital representations can create frustration and distract the surgeon from the procedure, as they focus on a screen that contains an incomplete rendering of the procedure.

[0018] Rendered objects often have artifacts, latency, and other imperfections that result in an inaccurate depiction of reality. When performing registration, for example, if a surgeon is required to touch the tip of a rendered probe to a specific point in a rendered bone model, such latency and errors can cause frustration during point sampling. Latency in updating the model, for example after removing a portion of bone, can cause additional frustration. The probe tip may indicate that it has penetrated the bone even though it has not (which is highly unlikely).

[0019] Many surgeons do not understand the purpose of registration, which can also lead to frustration and withdrawal. Some existing systems do not provide visual cues to help surgeons understand the purpose of their actions or to provide an error check on registration accuracy. Furthermore, current systems do not include a fail-safe against collecting points without any penalty prior to registration. For example, a surgeon may introduce an error by unintentionally collecting points in the air (not touching bone).

[0020] Thus, current approaches suffer from one or more of these and / or other drawbacks by increasing surgical time, requiring additional and frustrating steps for the user (e.g., ordered points vs. randomly sampled points), and / or sampling unreasonable points (e.g., the center of the hip, the medial malleolus, and the lateral malleolus), which may reduce accuracy and tend to generally disengage users from thoughtful participation in the registration workflow.

[0021] Described herein is an approach that allows for faster registration with minimal disruption to surgical workflow and without compromising accuracy. Aspects propose the use of reality augmentation, improved algorithms, and thoughtful point sampling that reduce sampling times and provide a user-friendly workflow. As noted, registration may be commonly used in navigation systems, e.g., robotics. Although the examples described herein are presented in the context of registration between a bone model and a real patient anatomy, i.e., respective point clouds, for use in connection with surgical guidance, aspects of the registration approach described herein are applicable more broadly outside of anatomical registration and surgical applications.

[0022] As noted, a typical registration process involves a process of point sampling with a tracked registration probe to calculate the coordinate frame of a rigidly mounted trackable array, typically one array per bone, relative to the coordinate frame of the bone. A point cloud is generated using the registration probe to contact the bone surface and assign position coordinates to its tip at each registered point. The point cloud thus represents a set of data points in space that correspond to the anatomical structures of the patient's bone surface.

[0023] Referring to Figure 1, an exemplary registration determines the location of fixed tracking arrays (102, 104) connected to the patient anatomy (femur 106 and tibia 108 - fibula not shown in Figure 1). The location of the arrays can be found through point sampling of points on the bone surfaces, sampling the medial and lateral malleoli, and inferring the center of the hips through leg manipulation. A probe 110 with a probe tip 112 samples points at the end of the femur 106 in this example.

[0024] As previously mentioned, some existing approaches require a large number of point samples (e.g., 40) at each of the distal end of the femur and the proximal end of the tibia, plus additional samples from points on the medial and lateral malleolus of the tibia 108. As noted, this can become unwieldy.

[0025] 1 also shows a line 114 as a central axis representing the axis of the probe 110, which extends from the probe 110 (from the probe tip 112 in this embodiment) to the end of the femur in this embodiment. According to aspects described herein, this is a guide line and may be presented for the surgeon as an augmented reality (AR) element. As will be explained in more detail below, the surgeon can orient the probe so that this line extends as closely as possible through the central axis of the bone (e.g., the femur 106) to represent a line at the center of the patient's hip. This can be used in place of physical manipulation of the leg to orient the bone model so that it is relatively closely coaxially aligned with the actual patient's bone.

[0026] In an embodiment, a human (such as a surgeon) is involved in performing point sampling with a probe to align the bone model with respect to the patient anatomy. The surgery performed based on the registration, e.g., cutting bones, inserting medical devices, etc., can be performed by one or more surgeons, one or more robots (with or without human involvement), or a combination of the two. In particular, pre-operative data, e.g., a CT scan, may contain more information than what is visible to the surgeon during the procedure. For example, a CT scan captures the thickness of the cortical wall. Accurate registration correlates this pre-operative data to the real-time pose of the anatomy so that the surgeon has access to additional patient information. Thus, based on the registration, the process can determine one or more pieces of information indicative of the surgical guidance, for example, determined based on the bone model, and presented to the surgeon digitally with respect to the actual patient anatomy.

[0027] The registration methods provided herein may be faster, more accurate, and easier to perform. This may be performed, for example, without the need for ordered points, leg manipulation to infer hip centers, or medial or lateral malleolar samples. They may be easier to use due to innovative reality augmentation that aids in bone model pose and point sampling accuracy as described herein. Meanwhile, registration accuracy may be checked during point sample collection rather than waiting until the end of sample collection. This can be used to determine when registration is complete (e.g., registration accuracy based on the most recent sampled points meets a desired threshold), thereby avoiding the user having to sample additional points when not necessary to achieve a desired level of registration accuracy. Additionally, aspects engage the user in the registration workflow via visual cues. "User" as used herein refers to a user using the system to navigate the registration process. In many cases, this will be a surgeon, and thus the terms "user" and "surgeon" may be used interchangeably in this application, although it is noted that the user who collects the sample points does not necessarily have to be a surgeon, but instead may be, for example, a medical paramedic.

[0028] The registration methods provided herein may also reduce the occurrence of failed registrations, i.e., registrations where a minimum accuracy threshold condition is not met. Failed registrations are problematic because they result in increased surgical time and user frustration.

[0029] These and other aspects may be useful in any navigated surgical procedure, not just those related to the knee as described or illustrated herein, but also to the spine or other anatomical structures. Additionally, aspects may be applied in other industrial and / or navigated applications to register point clouds.

[0030] According to some embodiments, one or more visible imaging sensors are used, for example, one or more red, green, blue wavelength (RGB) cameras. The RGB cameras provide a view of the environment / surgical theater for a person (e.g., a surgeon) to understand the environment. Such a camera or cameras may be used in conjunction with a tracking system that tracks the patient anatomy in space. An infrared (IR) based tracker may act as such a tracking system, but there are other exemplary equipment / algorithms that may be used.

[0031] In accordance with a particular embodiment, the Polaris Vega® VT Optical Tracker (VEGA is a registered trademark) provided by Northern Digital Inc., Waterloo, Ontario, Canada may be utilized, which includes an integrated high definition video camera and one or more IR cameras. In the VT system of interest, the IR data coordinate system may be aligned with respect to the camera stream.

[0032] As described elsewhere in this application, AR overlays (i.e., digital elements presented to overlay the image / camera presentation) may be provided.

[0033] One aspect of the approach described herein is to set the origin of the bone model scan (from a CT scan, for example) at a location relative to the patient anatomy that is easy and intuitive to sample and from which as much useful information as possible can be inferred. The model origin defines its coordinate system and may determine where objects are located in real space. Note that other objects of interest, such as rigid tracking arrays, may be rendered as virtual reality augmentations to improve the dimensionality of the image from the camera frame (e.g., to ensure that objects closer to the camera than others are not visible behind such objects, or vice versa). Various surgical instruments or objects (such as one or more trackable arrays) may be rendered as virtual objects to improve the visualization of the camera view on the screen. According to a non-limiting example, it may be interesting to render fixed rigid arrays (e.g., 104, 104 in FIG. 1) as augmented reality overlays to assist the surgeon with spatial orientation. See FIG. 10, which demonstrates the limitation of not rendering arrays as virtual objects. Because the tracker array 1002 is physically closer to the camera than the registration probe 1004 and bone 1006 and is not rendered as a virtual reality augmentation, it appears to be behind these objects in the reality augmentation (screen view). To avoid this concern, it may be interesting to render objects of interest such as tracking arrays, e.g., tracking arrays 1002 and 1008, as virtual objects (e.g., as 1010, 1012 in the camera view).

[0034] Registration is facilitated, expedited, and accuracy-assured by having the user quickly select the model origin and its location relative to the actual patient bone location using a single, easily selected, sampled starting point aligned with the assistance of on-screen reality augmentation. The bone model origin is a useful point because it aids in the initial alignment of the model to the patient anatomy. If a good initial alignment is achieved, fewer additional points are needed to accurately and properly determine the transformation that registers the bone model point cloud to the patient anatomy point cloud defined by the sampled points. With regard to the origin, it may generally be desirable for the patient anatomy corresponding to the bone model origin to be easily accessible and positioned such that the axis of the probe tip is intuitively aligned to the axis of the bone.

[0035] According to a non-limiting example, the bone model origin may be a proximal surface point within a cylinder approximated by the bone shaft, generally aligned with the bone tubercles. By approximating the bone as a cylinder, it may be beneficial to set the bone model origin to a surface point within the cylinder. For example, the bone origin may be selected to be a distal (femur) or proximal (tibia) point through the approximated axis of the bone. In the example of FIG. 1, the origin may be set to a point on the surface of the femur 106 at the tip 112 of the probe 110. Note that the origin may be any point such that the initial placement and orientation of the probe with respect to the anatomical structures and AR overlay is intuitive. According to a non-limiting example, the origin may be a point on the distal surface, the femur 106, or the proximal surface of the tibia 108. Note that the probe may sample the bone surface through the skin, although generally it is most preferred that the probe tip contacts the bone and therefore the sample point is within the incision.

[0036] A system according to aspects described herein can automatically assist the user in selecting the best initial point / alignment. For example, the bone model can be presented to the user as an AR overlay that displays the patient anatomy in a fixed position relative to the probe tip. The user can manipulate the probe to orient and position the bone model to match the patient anatomy, i.e., visually fitting the model in the appropriate position.

[0037] Referring to FIG. 2, a bone model 202 is shown presented as an AR element placed over a view of an environment 200 including a patient's bone 204. For example, the view may be provided by a camera offering, and a computer system may place the AR element over the view and display the view including the AR element on a screen. Additionally or alternatively, a user may wear smart glasses or other wearable devices to view the environment through one or more transparent displays (such as transparent lenses with an active display built thereon), and one or more AR elements may be presented on the transparent displays to provide the user with an augmented view. Also shown in FIG. 2 is a user's arm / hand 206 holding a probe 208, specifically the shaft of the probe. At the end of this shaft is the probe tip (directly under the user's thumb in FIG. 2). Since the exact location of the probe tip is known by the probe tracking provided by the probe, the system may place the AR bone model origin at the tip of the probe 208, as shown in FIG. 2. When the user lowers the probe in this view, the bone model 202 moves with the probe, remaining in a fixed position and orientation relative to the probe tip. The model "floats" and moves around with the probe tip. If the user reorients the probe to change the axis of the probe tip (indicated by line 210), the axis of the bone model changes accordingly. Here, the shaft of the bone 204 is approximated as a cylinder, and line 210 (also an AR element) is provided to represent the probe axis, which may be visually aligned to correspond to the bone axis. The user may align line 210 to the axis of the patient's bone 204 as visually estimated by the user.

[0038] Thus, the user holds the probe and moves and pivots it to orient (in position and rotation) the bone model 202 relative to a particular object of interest, in this example the top of the femur 204. Since the bone model 202 starts at the tip of the probe 208, it is expected that the probe tip will contact the surface of the patient's bone when the model is in approximately the correct position and orientation. The user may then provide some input (keystroke, mouse click, button press on the probe, etc.) to select an origin point and temporarily fix the position of the bone model starting from that point. This user selection selects the initial alignment of the bone model 202, and the model is placed in that position (i.e., as reflected in AR) selected by the user. From there, the user may move the probe 208 to collect other sample points on the patient anatomy, as described below. When the user samples additional points, this provides the system with additional actual bone surface points, taken as true values ​​of the bone surface locations. Each additional true value may result in the system slightly adjusting the position of the model to fit the model to the points collected for that point in the process. It is expected that the registration of the bone model will become more accurate with each additional point sampled. Note that all of the captured data points may be processed simultaneously or in parallel by algorithms that aid in pose determination. According to non-limiting examples, outlier detection algorithms (e.g., RANSAC algorithm) and iterative closest point algorithms (e.g., ICP algorithm) may take all sampled points of interest as data inputs and process such data points in parallel or in series to determine the associated registration transformation.

[0039] In conventional systems, the orientation of the probe when a point is registered is generally not considered to be relevant data, and the goal is simply to capture the coordinates of the probe tip. Registering a surface point requires knowing the orientation of the probe, but when a point is sampled, the probe position itself is generally considered arbitrary and irrelevant (see FIG. 11, which shows how little the orientation of the registration probe 1102 in the four scenarios described needs to affect the coordinates of the sampled point - the position of the probe tip relative to the bone surface is generally the only relevant data input). That is, the surgeon determines the orientation of the point as is practical.

[0040] In contrast, the embodiments described herein assign an association to the probe orientation with respect to at least the first sampled point, e.g., the origin, to provide an initial starting point for global coarse fitting of the model and fine fitting of the model. The global coarse fitting may generally be performed using the sampled points by random sampling of measurements, applying thereto an algorithm to estimate the parameters of the model, e.g., Random sample consensus (RANSAC), maximum likelihood estimator sample consensus, maximum a posteriori sample consensus, causal inference of the state of the dynamical system, resampling, HOP diffusion Monte Carlo, Hough transform, or similar algorithms. According to a non-limiting example, the "Random sample consensus (RANSAC)" algorithm and the fine fitting may be performed by applying thereto an "Iterative Closest Point (ICP)" algorithm from points to a plane. Rather than simply taking the coordinates of the origin surface point, the embodiments establish the coordinates of this point based on the orientation (i.e., "pose") of the probe. This first point may be treated as the bone model origin, so that the process properly aligns the origin coordinate frame containing the first sampled point. Locating the model origin coordinate frame containing the first sampled point may significantly reduce error metrics and the likelihood of iterating toward a local minimum rather than an absolute minimum. In other words, the initial pose provided by the user-selected orientation, as described above, allows the system to initially filter some of the infinite possibilities provided by fine fitting (e.g., ICP) and instead establish a starting point that provides the most useful information from which an initial guess can be made. One or more fitting algorithms are provided with respect to the probe orientation, which is known, so that the overall orientation of the model is provided. The fine fitting algorithm (e.g., ICP) may otherwise assume that the bone could be anywhere.By providing this initial orientation, we potentially eliminate some "local minima" that the fine fitting algorithm might otherwise consider to be orientation candidates. In fact, the initial orientation incorporates some intelligence into the fitting algorithm that uses this initial pose; instead of simply generating a surface map and letting an algorithm (e.g., ICP) iteratively solve for the minimum error between the two point clouds (model and patient anatomy), the user defines an approximated initial orientation of the model to eliminate what might otherwise be a possible (erroneous) outcome of the fitting.

[0041] To improve the usefulness of the probe orientation as a relevant input, aspects use augmented reality to help a user properly orient the bone model point cloud with respect to the patient anatomy and make this process intuitive, for example as shown in FIG. 2. In some examples, a user views a live video stream from one or more cameras capturing images of the environment in which the patient anatomy is located, and one or more AR elements are displayed along with the video stream on one or more display devices. In other examples, the user's view of the environment is through AR glasses worn by the user, the AR glasses having one or more transparent displays, for example provided as lenses for glasses. The AR elements may be displayed on one or more transparent displays to position the elements in the user's line of sight through the lenses with respect to the environment.

[0042] An example of reality augmentation is an AR element of a bone model from a CT scan, although it should be appreciated that aspects work for image-less systems that do not use advanced imaging. One or more IR cameras or other tracking systems track the real-time position of the registration stylus / probe to determine the corresponding movement of the AR overlays, so that they move with the probe to enable the positioning shown in Figure 2. The AR overlays do not need to be patient-specific, but can be a generalized shape of interest.

[0043] As shown in FIG. 2 and with further reference to FIG. 3, the bone model 302 may be a patient anatomical feature, in this example a section (or optionally the entirety) of the bone, displayed at the tip 312 of the probe 308, such that the origin of the bone model is a point at the probe tip 312. For usability, the transparency of the bone model overlay 302 may be adjusted. When the probe tip 312 is placed on the patient's bone surface anatomy to define the origin, the bone model overlay 302 may be rendered such that the augmented reality overlay is generally aligned and overlaid with the patient anatomy. In practice, this is immediately intuitive, and the user positions the probe to at least visually match the AR overlay 302 and the patient anatomy.

[0044] FIG. 3 shows an AR extension that improves the usefulness of the registration probe to allow for more accurate initial positioning of the bone model. To aid the user in understanding the orientation of the probe, the probe tip 312 corresponds to the origin of the bone model and the virtual line 310 corresponds to the central axis of the probe. Since in this example the model 302 represents only a portion of a bone, the user may visually align the line 310 to the axis of the patient's bone to aid the user in aligning the bone model to the patient anatomy beyond what the bone model itself visually provides. By way of a non-limiting example, the line 310 through the axis of the probe tip may be a length that extends from the origin to a distal (tibia) or proximal (femur) point and is generally parallel to the bone axis. In particular, this line may be a length from the origin to the center of the hip for the femur, the exact length of which is determined from the initial imaging on which the bone model is based. The line may aid in proper orientation of the probe with respect to the initial sampled points. Other AR overlays are also possible and may be provided to assist the user in positioning the model relative to the initial sample point / origin.

[0045] The AR bone model 302 is placed at the probe tip in these examples, but it may be placed anywhere, allowing the user to intuitively and easily sample points on the surface of the anatomical structure. In general, initial pose selection by the user may be desired, though intuitive enough, so that the user manipulates the probe to approximately correctly orient the model at the bone. As noted above, the origin may be a root point to which other sampled points may be referenced, and while this origin may be anywhere, it is typically on an exposed surface of an exposed anatomical structure (e.g., the top of a bone exposed during surgery) to allow the user to directly contact the probe tip to a surface point on the patient anatomy.

[0046] One approach for registration uses, at least in part, one or more iterative closest point (ICP) algorithms for registration. The ICP algorithms attempt to minimize the difference between the point clouds. In the embodiment described herein, one point cloud is generated by capturing actual bone surface points with a registration probe, and the other point cloud corresponds to a bone model generated, for example, by a CT scan. Through a trial and error process, the algorithm iteratively tries to orient one point cloud to the other. The registration accuracy can be mathematically inferred, which describes how well the positions of the (transformed) bone model points describe the positions of the actual patient anatomy. According to a non-limiting embodiment, the registration accuracy may be calculated as the root mean square of the differences between matched pairs. The ICP algorithm iteratively modifies the transformation (bone model point cloud) to minimize this error metric.

[0047] Some conventional anatomical model registrations use an ICP algorithm, but in particular it lacks the "intelligence" to iteratively check for errors in the transformation, which may be random. Since there are infinite possible transformations of the point clouds and the algorithm can only check a finite number of options, a limitation of the ICP algorithm is that it can be iteratively solved for a local minimum that is not an absolute minimum. Solving for a local minimum may infer an impossible solution. Referring to FIG. 4, 402a shows the actual patient anatomy (femur 404 resting on fibula 408 and tibia 406). The ICP algorithm may minimize the difference in the point cloud with an inverted model (including femur 404', fibula 408', and tibia 406') shown in 402b in the iterative solution set. This shows the limitation of the ICP algorithm. In practice, it is common to find a local minimum that is not the actual minimum, which is practically impossible or at least results in an incorrect orientation. Until a sufficient number of points are sampled, the results of the ICP algorithm can be very poor and / or meaningless. Traditional approaches overcome this by increasing the number and variety of points in the sampled point cloud, but this has the drawback of taking increased time, as mentioned above.

[0048] Furthermore, current approaches do not incorporate registration accuracy as a real-time variable in the registration workflow. Existing systems have a registration protocol that must be fully compliant. Only after a sufficiently complete run does the system calculate the registration accuracy and determine whether it falls above or below some defined threshold (e.g., 0.5 mm). According to a non-limiting example of a registration protocol with 4 sampled points, after only 10 sampled points have been collected, the registration error may actually be less than the acceptable registration error threshold, but this will not be known and the user will still be required to unnecessarily sample the remaining 30 points before registration and accuracy determination can be performed. Furthermore, the user has no notion of registration error when a point is sampled, the user often samples points without understanding why, and the user has no intuitive way to evaluate the registration accuracy as they proceed with sampling. After the model is fitted to the collected points, the user is often presented with data that is not intuitive in a particular application, and the surgeon typically has no knowledge of the significance and acceptability of an RMS error of, for example, 0.5 mm.

[0049] According to the registration approach described herein, reality augmentation is used to facilitate the user's proper orientation of the bone model point cloud relative to the patient anatomy originating from the first sampled point, and multiple fitting algorithms are used. As an example, one fitting algorithm is applied for coarse fitting of the model orientation, and another (different) fitting algorithm is applied for fine fitting of the point cloud based on the additional sample points. As an example coarse fitting, a RANSAC algorithm may be used in parallel or in series with an ICP algorithm for outlier detection and to help find an initial pose for the preliminary transformation, while RANSAC (or generally a similar algorithm that estimates the model's parameters by random sampling of measured data) and / or ICP may be used for the refinement of the transformation. In particular, the two algorithms may be run simultaneously or consecutively.

[0050] As described above with reference to Figures 2 and 3, the user selects the origin and signals the initial transformation by moving the probe tip into the field of view and selecting an initial placement of the bone model point cloud. User identification of the origin in this manner provides the first sampled point of the patient anatomy point cloud. The user then uses the probe to sample one or more other points on the patient anatomy. These one or more points may be selected arbitrarily by the user or may be based on one or more points suggested by the system. At a point after the initial placement and the additional one or more points are sampled, a coarse (or "global") fitting (e.g., RANSAC) algorithm is applied. Generally, the global fitting provides a coarse alignment / fitting by searching in a relatively large area around the sampled points. If the initial placement of the bone model is relatively far away from the patient anatomy, the global fitting provides a better initial alignment for the bone model. The global fitting at this point may provide the user with adjustments to the initial fitting. After this coarse fitting, a fine fitting is performed, such as applying ICP, to provide a more focused fitting of the bone model to the sampled points. Note that a global fitting algorithm may be performed simultaneously with ICP. The ICP fitting may be performed on the output of the coarse fitting. After this fine fitting, the registration accuracy (e.g., an error metric) is determined. Assuming the registration accuracy is below some configurable threshold, the point sampling may continue with global and / or fine fitting being performed after each one or more additional sample points are collected.

[0051] According to a particular embodiment, the process obtains the origin of the initial pose selected by the user, then obtains one or two additional user samples of the patient anatomy for a total of two or three points that make up the patient anatomy point cloud. At that point, the RANSAC algorithm is applied to generate a coarse fitting, and then the ICP algorithm is applied for a finer fitting. It is determined whether the registration is sufficiently accurate. Depending on how the accuracy is measured, the threshold can be a maximum or minimum threshold. If the accuracy is represented by an error measurement (e.g., RMS method), the threshold may be the maximum allowable error, for example, 0.5 mm or "less than 0.5 mm". Assuming that the registration is not at the desired accuracy at that point, the process obtains another (i.e., one additional) point sample of the patient anatomy. The user samples the anatomy surface with the probe, and the fitting is performed again, this time with the additional sampled points. Again, the fitting may include a coarse fitting using all collected points followed by a fine fitting using all collected points, or may include only one such fitting (e.g., the fine fitting). Again, the registration accuracy may be determined, and the process may proceed by iterating (if the accuracy is less than the desired value) or by stopping when the desired accuracy is achieved. In this manner, the process may iterate through point collection, fitting, and accuracy determination until the registration accuracy is sufficient.

[0052] In some embodiments, after each additional sampled point, coarse and fine fitting is performed until a threshold accuracy of the fitting is reached, in other embodiments, more than one additional point is collected in an iteration before performing a refit for that iteration.

[0053] In this manner, global fitting (e.g., RANSAC) and fine fitting (e.g., ICP) may be performed using sampled points of the patient anatomy, applied as the point sampling progresses, e.g., during sampling of the points.

[0054] In some embodiments, a global fit is performed once after the first n points are collected (n>=3), and then only fine fitting is applied thereafter, e.g., after each additional point is sampled. In other embodiments, a global fit and a fine fit are used as described above after each additional point is sampled. In still other embodiments, a global fit may be applied periodically or aperiodically during sampling, e.g., after every k additional samples are collected, and a fine fit may be optionally performed after each sample is collected. Once the accuracy determination determines that a desired accuracy of the point cloud registration has been achieved, the iterations of sample collection, fitting, and accuracy determination may stop and terminate.

[0055] In some embodiments, after each additional point is sampled, the registration accuracy is determined. In embodiments, the registration accuracy may be a composite of two sets of measurements: (i) how far each sampled point is from the bone model, and (ii) a covariance that indicates the uncertainty that exists in all six degrees of freedom. The error metric at any time may be a function of each sampled point, i.e., a composite / aggregation of the errors for each of those points. The RMS error uses the distance from the point to the surface. The accuracy may be determined after each additional point is sampled so that the registration process can be terminated as soon as the desired accuracy is achieved, i.e., without wasting time and effort sampling more points than are needed to provide the desired accuracy. If the error after the most recently sampled point is less than a predefined threshold, the system may inform the user that the registration is complete and may proceed to the next phase of the workflow.

[0056] By way of non-limiting example, the registration error threshold may be 0.5 mm RMS (i.e., the desired accuracy is any error less than 0.5 mm). Using the described process, registration with an error of less than 0.5 mm has been achieved in as few as 8-10 samples in some experiments.

[0057] It is of interest to determine when the user has sampled enough points to accurately align the bones with the pre-operative plan. It is not always clear when the user has achieved accurate registration. The algorithm may only perform a mathematical inference of the accuracy of the registration. A direct measurement of the registration accuracy is not possible due to practical clinical limitations (although the visual cues claimed in this application facilitate surgeon input). We may wish to capture the minimum number of points required to actually achieve a sufficiently accurate registration. To determine when the user has sampled a sufficient number of points, and therefore the achieved accurate registration, is of commercial interest.

[0058] In particular, the ICP error metric may not be robust enough to determine when a user has achieved a reasonably accurate registration. According to a non-limiting example, we may also investigate the impact of the selected transformation on the sampled data points. The variance of spatial positions between the sampled points before and after each of the transformations is applied may be used to infer the accuracy of the registration. A smaller variance would correspond to a more accurate registration. According to a non-limiting example, the selected transformation may correspond to the transformation that has the smallest ICP error metric for each sampled point after the fourth sampled point. According to a non-limiting example, the distribution of transformations based on the combination of the four points may be evaluated for each sampled point and used as a means to select an appropriate transformation.

[0059] FIG. 5 illustrates an exemplary process for fitting / registering a bone model point cloud to a surface point cloud of a patient anatomy according to embodiments described herein. The process may be performed by a computer system executing software performing embodiments described herein. This computer system may be the same or different computer system as: (i) one that stores / maintains the bone model point cloud; (ii) one that acquires sampled points of the patient anatomy from the probe; and / or (iii) one that presents a live view of the sampling / surgical environment augmented with AR elements as described herein on one or more displays. Thus, there may be one or more computer systems participating in data collection and / or processing to perform embodiments described herein. In embodiments involving more than one computer system, such computer systems may be in wired and / or wireless data communication with each other, e.g., via one or more networks.

[0060] The process of FIG. 5 obtains (502) an origin point as the first sampled point of the patient anatomy. This point is provided as part of a collection that is expanded as additional points are sampled. The process proceeds by obtaining (504) one or more additional sampled points for inclusion in the collection. At 504, one or more additional sample points are collected. Points determined to be outliers may be automatically rejected and, optionally, replaced by resampling with another point. In a particular embodiment, the first iteration of 504 collects and adds two additional sample points to the collection so that the collection contains three points before proceeding.

[0061] The process then proceeds to attempt to fit the bone model point cloud to a surface point cloud defined by the points present in the current collection. The process performs (506) a coarse fitting (e.g., by applying a RANSAC algorithm) on the collection points. In certain embodiments, all points present in the collection at the current time in the process are used in this fitting. The process then performs (508) a fine fitting (e.g., by applying an ICP algorithm) on the collection points. In certain embodiments, all points present in the collection at the current time in the process are used in this fitting. The process then determines (510) the registration accuracy and queries (512) whether a desired accuracy has been achieved (e.g., based on one or more thresholds that define the desired registration accuracy). If so (512,Y), the point clouds are registered to each other with sufficient accuracy and the process ends. The points of the bone model point cloud, once registered to the patient anatomy, may then be made an accurate reflection of the surface points of the patient anatomy for use in surgical activities.

[0062] Alternatively, if it is determined that the desired accuracy has not yet been achieved (512,N), the process loops back to 504 to obtain and include one or more additional sampled points in the collection, and then proceeds again through coarse and fine fitting (506,508) using the points now in the collection (including the one or more additional sampled points). In certain embodiments, only one additional sampled point is collected when repeating 512-504 before repeating the coarse and fine fitting. Thus, in such embodiments, after each additional sample point is collected, a determination of the registration accuracy and whether further sampling is required is performed.

[0063] Presenting the bone model in AR can provide visualization of the real-time transformation of the bone model point cloud overlaid on the actual patient anatomy, giving the user an intuitive understanding of how the registration process works and providing the user with a visual representation of the updating registration accuracy. These visual cues make the registration intuitive and promote additional safety. For example, in conventional systems that require sampling of, for example, 40 points, the user's attention may be directed away from the surgical area to the display monitor. The provided AR overlay according to the embodiments described herein allows the user to pay direct attention to the surgical area and the patient anatomy while taking a relatively small number of required samples to achieve the desired accuracy. The visualization of how the transformation of the bone model point cloud changes with each sampled point allows the user to intuitively evaluate the registration accuracy.

[0064] 6 shows an example of a bone model fitting after sampling two points (at 602). As the user samples additional points, the registration transformation on the bone model point cloud is updated via an augmented reality overlay 606, allowing the user to see the registration improvement with each sampled point until the two point clouds are aligned (at 604).

[0065] An additional limitation of existing methods is that, as noted above, point sampling is often done in a specific order, where the user captures a diverse and comprehensive point cloud, albeit in a very inefficient manner. The goal is to solve a transformation of the pre-operative bone model (generated from a CT scan) that generates a point cloud representing the patient bone surface anatomy and minimizes an error metric between these point clouds. Current systems guide the user to sample ordered bone surface points of the patient anatomy via a screen prompt represented by a circle in a virtual bone model rendering of the bone model from the CT scan. The next point to be sampled may be a different color or a different diameter, as the user prompts. The bone model rendering is not oriented to the actual patient position, but is free to move and be positioned arbitrarily. While it can be rotated by the surgeon, it is the user's responsibility to orient the bone model in the appropriate position. This process is very inefficient, unintuitive, and cumbersome for the user.

[0066] The aspects described herein do not constrain the user to ordered points, and the user may sample any point of interest until the process determines that the registration accuracy is below an acceptable threshold. In particular, the user may be prompted to capture various sets of points, but the location and order of those points is not a system constraint. According to a non-limiting example, the system may display points of interest for the user to align with the probe tip, which may be captured in any order. According to a non-limiting example, the system may show a visual representation of points / regions that have already been sampled by the surgeon, allowing the surgeon to visualize areas that have not yet been sampled.

[0067] Referring to FIG. 7, a view 700 of the environment displays an AR bone model 702 as a bone overlay (i.e., superimposed on the actual patient anatomy in the user's view). Points 701 on the bone model 702 indicate sampled points of the patient anatomy (bones), and a window 720 presents a computer-generated bone model 722 showing the system determining where those sampled points 701 should be on the bone model 722. A visual representation of regions already sampled by the surgeon may provide a visual cue for areas that have not yet been sampled. Thus, additionally or alternatively, the system may show "points of interest" in the view 700 as suggested points for the user to sample in any order.

[0068] Exemplary processing may include an outlier rejection approach to overcome the limitations of collecting erroneous samples, e.g., air samples, or other locations not facing the patient anatomy or object of interest. This increases the robustness of the system. The processing may incorporate an auto-reject feature that rejects sampled points at run-time (i.e., before sampling is terminated) if they deviate significantly from the rest of the point cloud. With the current approach of sampling 40 or more points, finding an outlier point would require sampling to be restarted from the first point.

[0069] In some embodiments, a single tracking camera is used. This constrains the view orientation to one angle, but note that one or more additional tracking cameras may be added to the system, for example to help determine the orientation more accurately in three dimensions. For example, more than one tracking camera may be used to facilitate three-dimensional alignment of the initial bone model pose.

[0070] Using the techniques described herein in a cadaveric lab, it has been demonstrated that the RMS error for real-time femur to femur registration was 0.40 mm with fewer than 10 sampled points.

[0071] The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a computer-implemented method, a computer system configured to perform the method, and a computer program product including a computer-readable storage medium storing instructions for performing the methods described herein. Additional features and advantages are realized through the concepts described herein.

[0072] In one embodiment of the computer-implemented method, the method includes registering the bone model point cloud to a point cloud of a patient anatomy. The registration includes obtaining a user selection of an origin for the bone model point cloud. The origin may be a sampled surface point on the patient anatomy, or may be the first point included in an established collection of sample points on the patient anatomy. The collection forms a point cloud of the patient anatomy. The registering further includes acquiring one or more other sampled surface points on the patient anatomy and including the acquired one or more other sampled surface points in the collection. The alignment further includes: determining an initial pose of a bone model point cloud based on a collection of sample points of the patient anatomy; obtaining additional sampled surface points on the patient anatomy and updating the collection of sample points to include the additional sampled surface points, thereby providing an updated collection of sample points; determining a fitting of the bone model point cloud to the point cloud of the patient anatomy based on the updated collection of sample points of the patient anatomy; determining a registration accuracy of a fitting of the bone model point cloud to the patient anatomy point cloud; and performing processing based on the determined registration accuracy.

[0073] In an embodiment, performing the process includes repeating obtaining additional sampled surface points, determining the fitting, and determining the registration accuracy one or more times based on the determined registration accuracy being less than a preconfigured threshold level of accuracy. In an embodiment, the iteration stops based on the determined registration accuracy being at least a preconfigured threshold level of accuracy. In an embodiment, based on stopping the iteration, the determined fitting of the bone model point cloud to the patient anatomy point cloud provides a registration of the bone model point cloud to the patient anatomy point cloud; The method further includes determining and digitally presenting one or more instructions for surgical guidance to the surgeon.

[0074] In some embodiments, obtaining the user selection of the origin includes providing a bone model augmented reality (AR) element that overlays a portion of the view on the patient anatomy. The view may show a registration probe, and the bone model AR element may be provided in a fixed position relative to the probe tip of the probe. User movement of the probe may change the position of the bone model AR elements; User selection of the origin may include the user positioning and orienting the bone model AR element in the view to overlay the patient anatomy by touching the probe tip to the patient anatomy and then providing input (e.g., mouse click, button press, verbal confirmation, etc.) that selects the origin as the location of the probe tip in contact with the patient anatomy. Additionally, in some embodiments, obtaining a user selection of an origin further includes providing a probe axis AR element to the patient anatomy that overlays other portions of the view. The probe axis AR element may comprise an axis extending from the probe at a first location (eg, the tip) and away from the probe tip towards a second location. The axis represents the axis of the probe / probe tip.

[0075] Additionally or alternatively, determining a fitting of the bone model point cloud to the point cloud of the patient anatomy based on the updated collection of sample points of the patient anatomy may include: performing a coarse fitting of the bone model point cloud to the patient anatomy point cloud using the updated collection of sample points of the patient anatomy; and performing a fine fitting of the bone model point cloud to the point cloud of the patient anatomy using the updated collection of sample points of the patient anatomy based on performing the coarse fitting. In an embodiment, performing the coarse fitting comprises applying a random sample consensus (RANSAC) algorithm and / or performing the fine fitting comprises applying an iterative closest point (ICP) algorithm.

[0076] Additionally or alternatively, determining the initial pose of the bone model point cloud may also utilize coarse fitting and / or fine fitting. For example, determining the initial pose of the bone model (e.g., after the first two or three sampled points) may involve: performing a coarse fitting of the bone model point cloud to the patient anatomical structure point cloud by applying a random sample consensus (RANSAC) algorithm; Based on performing the coarse fitting, performing a fine fitting of the bone model point cloud to the patient anatomy point cloud by applying an iterative closest point (ICP) algorithm.

[0077] It is noted that it may be impossible, much less practical, for a person to perform the registration of two point clouds by mental activity. For example, point clouds consist of digital representations of points in space, and applying an algorithm to align two point clouds, each consisting of just two points or more points, may not be practical or possible by a person's mental activity, much less at the speed required for surgical and other applications. Furthermore, it is not possible to sample points in the patient anatomy purely by mental activity and obtain point data that can be used in the calculation to align the point clouds. The bone model point cloud according to the embodiments described herein is a digital construct and does not exist mentally. Furthermore, it is not possible to provide augmented reality, for example, by overlaying digital graphical elements as AR elements on a view of the environment purely by a person's mental activity. Furthermore, point cloud registration is crucial for the planning and execution of surgical procedures and the corresponding safety and success of the surgical procedures. The embodiments described herein at least improve the technical field of registration, surgical procedures, and other technologies.

[0078] The processes described herein may be performed by one or more computer systems, such as, by way of example, one or more systems that are or communicate with a registration probe, a camera system, a tracking system, and / or an AR system, either alone or collectively. FIG. 8 illustrates an example of such a computer system and associated devices for incorporating and / or using aspects described herein. The computer system may be referred to herein as a data processing device / system, a computing device / system / node, or simply a computer. The computer system may be based on one or more of a variety of system architectures and / or instruction set architectures, such as those provided by Intel Corporation (Santa Clara, California, USA) or ARM Holdings plc (Cambridge, England, United Kingdom), by way of example.

[0079] FIG. 8 illustrates a computer system 800 in communication with one or more external devices 812. The computer system 800 includes one or more processors 802, e.g., a central processing unit (CPU). The processor may include functional components used in the execution of instructions, such as functional components that retrieve program instructions from a location such as a cache or main memory, decode the program instructions, execute the program instructions, access memory for instruction execution, and write results of executed instructions. The processor 802 may also include one or more registers used by one or more of the functional components. The computer system 800 also includes memory 804, input / output (I / O) devices 808, and I / O interfaces 810, which may be connected to the one or more processors 802 and to each other via one or more buses and / or other connections. The bus connections represent one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), Enhanced ISA (EISA), Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI).

[0080] Memory 804 may be or include, by way of example, a main or system memory (e.g., random access memory), one or more storage devices, such as one or more hard drives, flash media, optical media by way of example, and / or cache memory used to execute program instructions. Memory 804 may include, by way of example, a cache, such as a shared cache, which may be connected to a local cache (e.g., L1 cache, L2 cache, etc.) of one or more processors 802. Additionally, memory 804 may be or include at least one computer program product having a collection (e.g., at least one) of program modules, instructions, code, etc., configured to perform the functionality of embodiments described herein when executed by one or more processors.

[0081] The memory 804 may store an operating system 805 and other computer programs 806, such as one or more computer programs / applications that perform aspects described herein. In particular, the programs / applications may include computer readable program instructions that may be configured to perform the functions of embodiments of aspects described herein.

[0082] Examples of I / O devices 808 include, but are not limited to, microphones, speakers, Global Positioning System (GPS) devices, RGB and / or IR cameras, lights, accelerometers, gyroscopes, magnetometers, sensor devices configured to detect light, proximity, heart rate, body and / or air temperature, blood pressure, and / or skin resistance, registration probes, and activity monitors. The I / O devices may be integrated into the computer system as shown, although in some embodiments the I / O devices may be considered external devices (812) connected to the computer system via one or more I / O interfaces 810.

[0083] Computer system 800 may communicate with one or more external devices 812 via one or more I / O interfaces 810. Exemplary external devices include a keyboard, a pointing device, a display, and / or any other device that allows a user to interact with computer system 800. Other exemplary external devices include any device that allows computer system 800 to communicate with one or more other computing systems or peripheral devices, such as a printer. Network interfaces / adapters are exemplary I / O interfaces that allow computer system 800 to communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), providing communication with other computing devices or systems, storage devices, and the like. Ethernet-based (e.g., Wi-Fi) interfaces and Bluetooth adapters are just examples of currently available types of network adapters used in computer systems (BLUETOOTH is a registered trademark of Bluetooth SIG, Inc., Kirkland, Washington, USA).

[0084] Communications between I / O interfaces 810. External devices 812 may also occur over wired and / or wireless communication links 811 (Ethernet based wired or wireless connections). Exemplary wireless connections include cellular, Wi-Fi, Bluetooth, proximity based, near field, or other types of wireless connections. More generally, communication link(s) 811 may be any suitable wireless and / or wired communication link(s) for communicating data.

[0085] The specific external device(s) 812 may include one or more data storage devices, which may store one or more programs, one or more computer readable program instructions, and / or data, etc. The computer system 800 may include and / or be connected to and communicate with (e.g., as a computer system external device) removable / non-removable, volatile / non-volatile computer system storage media. For example, it may include and / or be connected to a non-removable, non-volatile magnetic medium (typically called a "hard drive"), a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and / or an optical disk drive for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium.

[0086] The computer system 800 is operational with numerous other general purpose or special purpose computing system environments or configurations. The computer system 800 may take any of a variety of forms, known examples of which include, but are not limited to, one or more personal computer (PC) systems, one or more server computer systems such as one or more messaging servers, multiprocessor systems, microprocessor-based systems, telephony devices, network equipment (such as edge device(s)) (one or more mobile devices / computers such as one or more thin clients, one or more thick clients, one or more workstations, one or more laptops, one or more handheld devices, one or more smartphones, one or more tablets, and one or more wearable devices, one or more virtualization devices, one or more storage controllers, one or more set-top boxes), one or more programmable consumer electronic devices, one or more network PCs, one or more minicomputer systems, one or more mainframe computer systems, and one or more distributed cloud computing environments that include any of the above systems or devices, and the like.

[0087] FIG. 9 illustrates another embodiment of a computer system for incorporating and using aspects described herein. FIG. 9 illustrates an exemplary eyewear based on a wearable device, for example a wearable smartglasses device that facilitates the presentation of AR elements to a wearer of the device. The device 900 may include many of the same types of components included in the computer system 800 described above. In the embodiment of FIG. 9, the device 900 is configured to be wearable on the head of a device user. The device includes a display 902 that is located in the line of sight of the user's peripheral vision when the device is in an operational position on the user's head. A suitable display may utilize LCD, CRT, or OLED display technology, as examples. The lens 914 may optionally include an active translucent display, where the inner and / or outer surfaces of the lens can display images and other content. This provides the ability to overlay this content directly on the user's line of sight and overlay at least a portion of the user's view of the environment through the lens. In the particular embodiment described herein, the content presented on the lenticular display is an AR element that overlays streams from one or more cameras showing the surgical environment / theater.

[0088] The device 900 also includes a touch input 904 that allows a user to input touch gestures to control functions of the device. Such gestures may be interpreted as commands, such as a command to take a picture or a command to launch a particular service. The device 900 also includes buttons 909 for controlling one or more functions of the device. Exemplary functions include locking, shutting down, or putting the device into a standby or sleep mode.

[0089] Various other input devices are provided, such as a camera 608 that can be used to capture images or video. The camera may be used by the device to obtain one or more images / videos of a view of the wearer's environment, for example for use in capturing an image / video of the scene. Additionally, the camera or cameras may be used to track the direction of the user's eyesight, to ascertain where the user is looking, and to track other eye activity of the user, such as blinking or movement.

[0090] Further, for example, within the housing 910, one or more microphones, proximity sensors, light sensors, accelerometers, speakers, GPS devices, and / or other input devices (not numbered) may be provided. The housing 910 may also include other electronic components, such as electronic circuitry, including one or more processors, memory, and / or communication devices, such as cellular, short-range wireless (e.g., Bluetooth), or Wi-Fi circuitry, for connection to remote devices. The housing 910 may further include a power source, such as a battery, for powering the components of the device 900. Additionally or alternatively, any such circuitry or battery may be included in the enlarged end 912, which may be enlarged to accommodate such components. The enlarged end 912, or any other portion of the device 900, may also include one or more physical ports (not shown) used to connect the device 900 to a power source (for recharging the battery), any other external device, such as a computer. Such physical ports may be of any standardized or proprietary type, such as Universal Serial Bus (USB).

[0091] Aspects of the present invention may be systems, methods, and / or computer program products, any of which may be configured to perform or facilitate the aspects described herein.

[0092] In some embodiments, aspects of the invention may take the form of a computer program product, which may be embodied as one or more computer readable media. The computer readable medium may be a tangible storage device / media having computer readable program code / instructions stored thereon. The one or more exemplary computer readable media include, but are not limited to, electronic, magnetic, optical, or semiconductor storage devices or systems, or any combination of the foregoing. Exemplary embodiments of computer readable media include hard drives or other mass storage devices, electrical connections with wires, random access memory (RAM), read-only memory (ROM), erasable-programmable read-only memory (EPROM) or flash memory, optical fiber, portable computer disks / diskettes such as compact disc read-only memory (CD-ROM) or Digital Versatile Disc (DVD), optical storage devices, magnetic storage devices, or any combination of the foregoing. The computer readable medium may be readable by a processor, processing unit, etc., to retrieve data (e.g., instructions) from the medium for execution. In certain embodiments, a computer program product is or includes one or more computer-readable media that contain / store computer-readable program code for providing and facilitating one or more aspects described herein.

[0093] As noted, the program instructions contained or stored on the computer-readable medium may be retrieved and executed by any of a variety of suitable components, such as a processor of a computer system, to cause the computer system to behave and function in a particular manner. Such program instructions for performing operations to implement, achieve, or facilitate aspects described herein may be written in any desired programming language or compiled from code written in such a language. In some embodiments, such programming languages ​​include object-oriented and / or procedural programming languages, such as C, C++, C#, Java, etc.

[0094] Program code may include one or more program instructions acquired for execution by one or more processors. The computer program instructions may be provided to one or more processors, for example, of one or more computer systems, such that the program instructions, when executed by the one or more processors, produce a machine that performs, accomplishes, or facilitates aspects of the present invention, such as the actions or functions illustrated in the flowcharts and / or block diagrams described herein. Thus, each block, or combinations of blocks, of the flowchart illustrations and / or block diagrams illustrated or described herein may, in some embodiments, be implemented by computer program instructions.

[0095] Although various embodiments have been described above, these are merely examples.

[0096] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used herein, the singular forms "a", "an", and "the" are intended to include the plural, unless the context clearly indicates otherwise. It will be further understood that the term "comprising", as used herein, specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0097] Corresponding structures, materials, acts, and equivalents of all means or steps and functional components in the appended claims are intended to include any structures, materials, or acts for performing the functions in combination with other claimed components, if any, as specifically recited in the claims. The description of one or more embodiments has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosed forms. Numerous modifications and changes will become apparent to those skilled in the art. The embodiments have been selected and described to best explain various aspects and practical applications, and to enable those skilled in the art to understand various embodiments with various modifications as suited to the particular use contemplated.

Claims

1. 1. A computer-implemented method comprising registering a model point cloud to an object point cloud, the method comprising: The registering includes obtaining a user selection of an origin for the model point cloud; the origin is a sampled surface point on the object and is the first point included in an established collection of sample points of the object; said collection forming a point cloud of said object; The above alignment is obtaining one or more other sampled surface points of the object and including the obtained one or more other sampled surface points in the collection; determining an initial pose of the model point cloud based on a collection of sample points of the object; obtaining additional sampled surface points on the object and updating the collection of sample points to include the additional sampled surface points, thereby providing an updated collection of sample points; determining a fitting of the model point cloud to the object point cloud based on the updated collection of sample points of the object; determining a registration accuracy of a fitting of the model point cloud to the object point cloud; and performing processing based on the determined registration accuracy. method.

2. the model point cloud comprises an anatomical structure model point cloud; the object comprises a patient anatomy; The method of claim 1.

3. performing the process includes repeating obtaining additional sampled surface points, determining a fitting, and determining the registration accuracy one or more times based on the determined registration accuracy being less than a preconfigured accuracy threshold level. The method of claim 2.

4. the iterating is stopped based on the determined registration accuracy being at least the preconfigured accuracy threshold level. The method of claim 3.

5. based on stopping the iterating, the determined fitting of the anatomical structure model point cloud to the patient anatomy point cloud provides a registration of the anatomical structure model point cloud to the patient anatomy point cloud; The method further includes determining and digitally presenting to the surgeon one or more instructions for surgical guidance. The method of claim 4.

6. obtaining the user selection of the origin includes providing an anatomy model augmented reality (AR) element to the patient anatomy that overlays a portion of a view showing a registration probe; the anatomical structure model AR element is provided in a fixed position relative to a probe tip of the probe; User movement of the probe changes the position of the anatomical structure model AR element; the user selection includes a user positioning and orienting the anatomical structure model AR element in the view to overlay the patient anatomy by contacting the probe tip with the patient anatomy and providing an input selecting the origin as the location of the probe tip contacting the patient anatomy. The method of claim 2.

7. obtaining the user selection of the origin further includes providing a probe axis AR element to the patient anatomy that overlays other portions of the view; The probe axis AR element comprises an axis extending from the probe at a first location and away from the probe tip towards a second location; The axis represents the axis of the probe. The method of claim 6.

8. Determining a fitting of the anatomical structure model point cloud to the patient anatomy point cloud based on the updated collection of sample points of the patient anatomy includes: performing a coarse fitting of the anatomical structure model point cloud to the patient anatomy point cloud using the updated collection of sample points of the patient anatomy; performing a fine fitting of the anatomical structure model point cloud to the patient anatomy point cloud using an updated collection of sample points of the patient anatomy based on performing the coarse fitting. The method of claim 2.

9. performing the coarse fitting includes applying a random sample consensus (RANSAC) algorithm and / or performing the fine fitting includes applying an iterative closest point (ICP) algorithm; The method of claim 8.

10. Determining an initial pose of the anatomical structure model point cloud includes: performing a coarse fitting of the anatomical structure model point cloud to the patient anatomy point cloud by applying a random sample consensus (RANSAC) algorithm; and performing a fine fitting of the anatomical structure model point cloud to the patient anatomical structure point cloud by applying an iterative closest point (ICP) algorithm based on performing the coarse fitting. The method of claim 2.

11. 1. A computer system comprising: a memory; and a processor in communication with the memory, The computer system is configured to execute a method including registering a model point cloud to an object point cloud; The registering includes obtaining a user selection of an origin for the model point cloud; the origin is a sampled surface point on the object and is the first point included in an established collection of sample points of the object; said collection forming a point cloud of said object; The above alignment is obtaining one or more other sampled surface points of the object and including the obtained one or more other sampled surface points in the collection; determining an initial pose of the model point cloud based on a collection of sample points of the object; obtaining additional sampled surface points on the object and updating the collection of sample points to include the additional sampled surface points, thereby providing an updated collection of sample points; determining a fitting of the model point cloud to the object point cloud based on the updated collection of sample points of the object; determining a registration accuracy of a fitting of the model point cloud to the object point cloud; and performing processing based on the determined registration accuracy. Computer system.

12. the model point cloud comprises an anatomical structure model point cloud; the object comprises a patient anatomy; 12. The computer system of claim 11.

13. performing the process includes repeating obtaining additional sampled surface points, determining a fitting, and determining the registration accuracy one or more times based on the determined registration accuracy being less than a preconfigured accuracy threshold level.

13. The computer system of claim 12.

14. the iterating is stopped based on the determined registration accuracy being at least the preconfigured accuracy threshold level.

14. The computer system of claim 13.

15. based on stopping the iterating, the determined fitting of the anatomical structure model point cloud to the patient anatomy point cloud provides a registration of the anatomical structure model point cloud to the patient anatomy point cloud; The method further includes determining and digitally presenting to the surgeon one or more instructions for surgical guidance.

15. The computer system of claim 14.

16. obtaining the user selection of the origin includes providing an anatomy model augmented reality (AR) element to the patient anatomy that overlays a portion of a view showing a registration probe; the anatomical structure model AR element is provided in a fixed position relative to a probe tip of the probe; User movement of the probe changes the position of the anatomical structure model AR element; the user selection includes a user positioning and orienting the anatomical structure model AR element in the view to overlay the patient anatomy by contacting the probe tip with the patient anatomy and providing an input selecting the origin as the location of the probe tip contacting the patient anatomy.

13. The computer system of claim 12.

17. obtaining the user selection of the origin further includes providing a probe axis AR element to the patient anatomy that overlays other portions of the view; The probe axis AR element comprises an axis extending from the probe at a first location and away from the probe tip towards a second location; The axis represents the axis of the probe.

17. The computer system of claim 16.

18. Determining a fitting of the anatomical structure model point cloud to the patient anatomy point cloud based on the updated collection of sample points of the patient anatomy includes: performing a coarse fitting of the anatomical structure model point cloud to the patient anatomy point cloud using the updated collection of sample points of the patient anatomy; performing a fine fitting of the anatomical structure model point cloud to the patient anatomy point cloud using an updated collection of sample points of the patient anatomy based on performing the coarse fitting.

13. The computer system of claim 12.

19. performing the coarse fitting includes applying a random sample consensus (RANSAC) algorithm and / or performing the fine fitting includes applying an iterative closest point (ICP) algorithm; 20. The computer system of claim 18.

20. 13. The computer system of claim 12. Determining an initial pose of the anatomical structure model point cloud includes: performing a coarse fitting of the anatomical structure model point cloud to the patient anatomy point cloud by applying a random sample consensus (RANSAC) algorithm; and performing a fine fitting of the anatomical structure model point cloud to the patient anatomical structure point cloud by applying an iterative closest point (ICP) algorithm based on performing the coarse fitting.

13. The computer system of claim 12.

21. 1. A computer program product comprising: a computer-readable storage medium readable by a processing circuit, the computer-readable storage medium storing instructions for causing the processing circuit to execute a method comprising aligning a model point cloud to an object point cloud, the computer-readable storage medium comprising: The registering includes obtaining a user selection of an origin for the model point cloud; the origin is a sampled surface point on the object and is the first point included in an established collection of sample points of the object; said collection forming a point cloud of said object; obtaining one or more other sampled surface points of the object and including the obtained one or more other sampled surface points in the collection; determining an initial pose of the model point cloud based on a collection of sample points of the object; obtaining additional sampled surface points on the object and updating the collection of sample points to include the additional sampled surface points, thereby providing an updated collection of sample points; determining a fitting of the model point cloud to the object point cloud based on the updated collection of sample points of the object; determining a registration accuracy of a fitting of the model point cloud to the object point cloud; and performing processing based on the determined registration accuracy. Computer program products.

22. the model point cloud comprises an anatomical structure model point cloud; the object comprises a patient anatomy; 22. The computer program product of claim 21.

23. performing the process includes repeating obtaining additional sampled surface points, determining a fitting, and determining the registration accuracy one or more times based on the determined registration accuracy being less than a preconfigured accuracy threshold level.

23. A computer program product as claimed in claim 22.

24. the iterating is stopped based on the determined registration accuracy being at least the preconfigured accuracy threshold level.

24. A computer program product as claimed in claim 23.

25. based on stopping the iterating, the determined fitting of the anatomical structure model point cloud to the patient anatomy point cloud provides a registration of the anatomical structure model point cloud to the patient anatomy point cloud; The method further includes determining and digitally presenting to the surgeon one or more instructions for surgical guidance.

25. A computer program product as claimed in claim 24.

26. obtaining the user selection of the origin includes providing an anatomy model augmented reality (AR) element to the patient anatomy that overlays a portion of a view showing a registration probe; the anatomical structure model AR element is provided in a fixed position relative to a probe tip of the probe; User movement of the probe changes the position of the anatomical structure model AR element; the user selection includes a user positioning and orienting the anatomical structure model AR element in the view to overlay the patient anatomy by contacting the probe tip with the patient anatomy and providing an input selecting the origin as the location of the probe tip contacting the patient anatomy.

23. A computer program product as claimed in claim 22.

27. obtaining the user selection of the origin further includes providing a probe axis AR element to the patient anatomy that overlays other portions of the view; The probe axis AR element comprises an axis extending from the probe at a first location and away from the probe tip towards a second location; The axis represents the axis of the probe.

27. A computer program product as claimed in claim 26.

28. Determining a fitting of the anatomical structure model point cloud to the patient anatomy point cloud based on the updated collection of sample points of the patient anatomy includes: performing a coarse fitting of the anatomical structure model point cloud to the patient anatomy point cloud using the updated collection of sample points of the patient anatomy; performing a fine fitting of the anatomical structure model point cloud to the patient anatomy point cloud using an updated collection of sample points of the patient anatomy based on performing the coarse fitting.

23. A computer program product as claimed in claim 22.

29. performing the coarse fitting includes applying a random sample consensus (RANSAC) algorithm and / or performing the fine fitting includes applying an iterative closest point (ICP) algorithm; 30. The computer program product of claim 28.

30. Determining an initial pose of the anatomical structure model point cloud includes: performing a coarse fitting of the anatomical structure model point cloud to the patient anatomy point cloud by applying a random sample consensus (RANSAC) algorithm; and performing a fine fitting of the anatomical structure model point cloud to the patient anatomical structure point cloud by applying an iterative closest point (ICP) algorithm based on performing the coarse fitting.

23. A computer program product as claimed in claim 22.

Images